Auditing LLM-as-Judge Reliability: Judge Upgrades Are Not Interchangeable

The article treats evaluator-replacement ambiguity as a measurement-validity problem, demonstrating that LLM-as-judge scores can shift even when candidate responses remain fixed. Across four judgment datasets, the authors compare scaling Qwen3 dense judges from 1.7B to 32B parameters and switching between MiniMax M2-M2.7 released APIs.

• Only the upgrade from Qwen3 1.7B to 4B provides a robust adjacent gain; other upgrades are not interchangeable.
• MiniMax adjacent releases do not yield consistent improvements in measurement stability.
• Stronger judges reduce but do not eliminate position and verbosity bias.
• Repeated-sample juries add little value when errors are correlated.
• Structured debate can move decisions substantially, but attribution requires parser and fallback logs.

The authors argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails to ensure reliability.