quant-forge

ADR 0002: Fast Walsh-Hadamard Incoherence Rotation & Hybrid OCP FP8 (E4M3 / E5M2)

Context

Weight-only compensation cannot eliminate extreme activation spikes when quantizing both weights and activations (W4A4 / W8A8), and uniform FP8 E4M3 overflows (max=448.0) on heavy-tailed backward pass gradients.

Decision

We implement an $O(d \log d)$ orthonormal Sylvester Fast Walsh-Hadamard Transform (fast_walsh_hadamard) following QuaRot / SpinQuant alongside bit-accurate OCP FP8 E4M3 (forward) and FP8 E5M2 (backward, max=57344.0) codecs.

Consequences