A new study challenges the reliance on accuracy and perplexity for evaluating post-training quantization in large language models, demonstrating that these metrics fail to capture significant behavioral changes. The authors introduce "correctness agreement," a decision-level metric measuring prediction overlap between base and quantized models, revealing that behavioral divergence emerges under moderate quantization even when task performance appears preserved.
- Analysis of quantization as a structural operator on attention weights identifies non-linear breakpoints at low bit-widths.
- Query and key projections are found to be consistently more sensitive to distortion than value and output projections.
- The study covers multiple models and quantization schemes ranging from 8-bit down to 2-bit.
These findings expose an illusion of equivalence between base and quantized models, motivating the adoption of behavioral evaluation metrics beyond conventional performance measures.