E4M3 / E5M2)Weight-only compensation cannot eliminate extreme activation spikes when quantizing both weights and activations (W4A4 / W8A8), and uniform FP8 E4M3 overflows (max=448.0) on heavy-tailed backward pass gradients.
We implement an $O(d \log d)$ orthonormal Sylvester Fast Walsh-Hadamard Transform (fast_walsh_hadamard) following QuaRot / SpinQuant alongside bit-accurate OCP FP8 E4M3 (forward) and FP8 E5M2 (backward, max=57344.0) codecs.
7.49x (24.56 -> 3.28), cutting INT4 output MSE by 3.34x while preserving exact bilinear invariance (max |Δ| = 6.87e-5).E5M2 for backward gradients (26.26 dB SQNR, 0.00% clipping vs E4M3 5.79 dB due to saturation above 448.0).