Variable Bit-width Quantization learns per-group precision for bigger-but-smaller language models

Researchers introduce Variable Bit-width Quantization (VBQ), a training-time method where each contiguous group of 64 weights learns its own resolution from {1,2,4,8} bits via Gumbel-Softmax relaxation. VBQ discovers a consistent, strongly heterogeneous allocation within individual projection types, allowing for a "bigger-but-smaller" regime that improves quality-per-byte.

• 69% of weight groups collapse to 1 bit, while the first MLP block retains ~2.5 bits.
• A 131M model at 1.82 mean bits beats a 55M FP16 model on TinyStories with 3.8x less storage.
• VBQ is 3.9-8.4x more efficient than FP16 in quality-per-byte metrics.
• Custom fused dequantize-and-multiply kernels accelerate inference, achieving up to 4.7x speedup at 9B parameters on Apple silicon.

VBQ reframes precision as a learnable, non-uniform resource, demonstrating that spending a fixed bit budget unevenly beats uniform allocation.