quant-forge

ADR 0003: Native C++20 Bit-Packed INT4 (2 weights/byte) Register-Fused GEMM

Context

Simulated “fake quantization” in PyTorch stores 4-bit weights inside float32 tensors (32 bits/weight), saving zero physical RAM and hiding bit-packing/nibble-unpacking bugs.

Decision

We implement physical 2-weights-per-byte uint8_t bit-packing and algebraically fused register Dequant-GEMM in csrc/quant_kernels.cpp (clang++ -O3 -std=c++20), cached by SHA-256 hash in .quant_cache/ with an exact NumPy fallback.

Consequences