Auditing LLM-as-Judge Reliability Shows Judge Upgrades Are Not Interchangeable

A study treats evaluator-replacement ambiguity as a measurement-validity problem, demonstrating that LLM-as-judge scores can shift even when candidate responses remain fixed. The authors compare two upgrade paths across four judgment datasets: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs.

• Only the upgrade from Qwen3 1.7B to 4B provides a robust adjacent gain, while MiniMax adjacent releases do not.
• Stronger judges reduce but do not remove position and verbosity bias.
• Repeated-sample juries add little value when errors are correlated.
• Structured debate can move decisions substantially, but shifts cannot be attributed to deliberation without parser and fallback logs.

The authors argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails to ensure reliability.