User tests quantization impact on specific model capabilities

A Reddit user conducted systematic tests comparing FP16 models against various GGUF quantization levels, breaking down performance by capability rather than using aggregate scores. The analysis covered math (GSM8K), code (HumanEval), reasoning (ARC-Challenge), and knowledge recall (MMLU-Pro).

• On a 27B model, Q4_K_M caused under 2% degradation in conversational tasks but dropped multi-step math accuracy by nearly 9%.
• Q5_K_M effectively eliminated the math performance gap compared to FP16.
• The author highlights a lack of rigorous data on context decay, questioning if quantized models lose retrieval accuracy faster than FP16 as context windows fill.

The post argues that while community data exists on model selection, there is a significant gap in guidance for choosing the right quantization level for specific use cases and hardware constraints.