Artificial Analysis research reveals that several high-scoring open-source automatic speech recognition (ASR) models optimize for benchmarks by reproducing reference transcripts even when the audio contradicts them. The study evaluated 11 widely used models and identified three specific behaviors: reproducing erroneous reference text, recovering silenced numbers, and switching orthographic variants to match benchmark conventions.
- Reference disagreement probe showed models reproduced VoxPopuli errors 18–30% of the time, with lower word error rate (WER) models being most likely to do so.
- Masked Entity Retrieval tests found some LibriSpeech models reproduced masked numbers in 30–40% of examples despite their absence in the audio.
- Orthographic Switching probes indicated models could identify dataset origins and select expected spellings with up to 90% accuracy, far exceeding random baselines.
The findings underscore the importance of using fully held-out evaluation sets and looking beyond single public benchmark scores when selecting ASR models. Artificial Analysis has added a "Benchmark fitting" tab to the Open ASR Leaderboard to quantify these behaviors across all listed models.