Multiverse Computing's QAH makes 4-bit GPT-OSS outperform its full-precision original

Multiverse Computing introduces Quantization-Aware Healing (QAH), a method that distills compressed, quantized large language models directly from their original pre-compression teachers. Applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, the approach produces a model that beats its own full-precision bfloat16 version on 7 of 9 benchmarks.

The QAH model surpasses the original 120B teacher on LiveCodeBench (66.5 vs. 66.0) and achieves large gains in long-context reasoning (+7.4 on AA-LCR) and math (+5.6 on AIME 2025).
Compared to quantization-aware training (QAT), QAH reaches peak performance in roughly 100 steps, about 7 times faster, while avoiding the sharp accuracy collapse seen in QAT after step 700.
The method uses memory-efficient chunked KL-divergence loss to handle long-context healing up to 32k tokens without materializing the full vocabulary-by-sequence grid.

This technique allows compressed models to be smaller, cheaper to serve, and more accurate than their uncompressed counterparts, effectively inverting the usual trade-off between efficiency and capability.

Benchmarks

Benchmark	Model	Score
GPQA Diamond	QAH model (GPT-OSS 120B compressed to 60B, MXFP4)	67.4%
LiveCodeBench	QAH model (GPT-OSS 120B compressed to 60B, MXFP4)	66.5%

Benchmark

Model

Score

GPQA Diamond

QAH model (GPT-OSS 120B compressed to 60B, MXFP4)

67.4%

LiveCodeBench

QAH model (GPT-OSS 120B compressed to 60B, MXFP4)

66.5%

Benchmarks

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

DSTP mitigates visual token pruning failures in complex MLLM reasoning

Open-source Decay-Gated O(N) Causal Linear Attention with fused Triton kernel

Jason Van Pham documents Gemma 4 multi-turn greedy degeneration under letter-level constraints

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention