The article highlights a structural failure in building synthetic corpora for LLM-as-judge bias studies, where shared decoding-budget parameters can truncate generated hallucinated answers. This truncation caused a 32-point cross-lingual collapse in judge selection accuracy, which was only detectable through manual inspection of raw generations rather than aggregate statistical checks.
- A shared parameter between judging and generation calls truncated one producer's hallucinated answers to a few words.
- The resulting items produced a statistically robust effect: a 32-point cross-lingual collapse in one judge's selection accuracy, replicated from N=50 to N=500.
- A second measured bias (Markdown-formatting preference) was distorted by the same fault, with its magnitude and sign shifting with stimulus length.
- The authors frame this as a test oracle problem, noting that corpora built by deterministic perturbation of a gold answer allow for 100% accurate detection of such faults via string comparison.
The authors propose a validation protocol for analysts working in the oracle-less regime that describes most contemporary multilingual LLM-as-judge corpora.