Best-of-N TTS evaluation confounded by ASR family alignment

Researchers identify that Best-of-N inference for zero-shot text-to-speech is confounded by the specific Automatic Speech Recognition (ASR) family used as a verifier, causing ranking reversals across Whisper, wav2vec 2.0, and HuBERT evaluators.

• Same-family verifier-evaluator pairs recover 2-3 times more oracle headroom than cross-family pairs despite near-identical representations.
• The authors propose two cross-family rank ensembles that attain the lowest mean Word Error Rate (WER) of 1.61% at N=10, a 12% improvement over F5-TTS.
• Using the best single verifier under the official F5-TTS evaluator drives WER from 2.06% to 1.72%, a 16.5% reduction.

The authors recommend cross-evaluator triangulation as the default reporting practice to mitigate these alignment issues.