Researchers introduce Log_bQuant, a novel logarithmic quantization approach featuring adjustable bases designed to adapt to common parameter distributions in language models.
The method addresses the suboptimal representations caused by low-frequency high-magnitude weights found in previous uniform quantization codebooks. It demonstrates superior performance at 4-bit precision on several benchmarks compared to asymmetric linear quantization at tensor-wise granularity, while achieving moderate speedup and high memory savings.
This approach makes language models suitable for private use on consumer-grade GPUs by reducing memory requirements and improving inference speed.