A study audits protocol-level shortcuts in large audio-language models (LALMs) used as automatic judges for speech evaluation, revealing that high agreement with human ratings does not guarantee verdicts are grounded in the actual audio. The research examines three deployment protocols: feature-blueprint judging, reference-conditioned judging, and pairwise A/B comparison across six judges and four attributes.
- In feature-blueprint judging, incorrect specialist labels reduced emotion accuracy to 0.10 or below for five of the judges.
- In concatenated A/B comparisons, Qwen3-Omni-Thinking frequently selected the same slot regardless of order swaps.
- The findings indicate that aggregate agreement can overstate the validity of LALM judges unless the model and evaluation protocol are assessed jointly.
The authors conclude that each model-protocol pair should be evaluated with a matched shortcut probe to ensure reliable speech evaluation.