Hume has introduced Real World VoiceEQ, a benchmark designed to evaluate the human quality of voice interaction by assessing how well systems recognize and produce acoustic information beyond transcripts. The benchmark evaluates over 40 proprietary and open-source models across 15+ dimensions using more than 60 metrics derived from 785,000 TTS and 48,000 STS human ratings.
- Evaluation covers ASR, TTS, S2S, and Speech Understanding capabilities.
- Data originates from Kairos platform with over 1 million individual human ratings across diverse demographics and environments.
- Findings show no single "best" voice model, as systems optimize for different strengths like technical accuracy versus emotional understanding.
- Traditional benchmarks are insufficient for real-world conditions, with performance varying significantly on accents, noise, and emotion.
- Automated speech-language models (SLMs) showed lower agreement with humans on subjective tasks compared to verifiable ones.
The benchmark aims to extend the paradigm of quantitative metrics by providing a human-grounded evaluation method for synthetic voice interactions in complex real-world conversations.