lab Hugging Face Blog · 4h ago · releases

Open TTS Leaderboard launches scalable multilingual TTS evaluation

from English

The Open TTS Leaderboard has been released to address the fragmentation and lack of standardization in Text-to-Speech (TTS) evaluation by using objective metrics instead of relying solely on slow, arena-based human preference scores. It evaluates models across intelligibility, speed, and speaker similarity using Qwen3 ASR and WavLM embeddings.

Intelligibility is measured via word/character error rate (WER/CER) between the prompt and generated audio transcript.
Speed is quantified by inverse real-time factor (RTFx) for batched offline inference and time-to-first-audio (TTFA) for streaming latency on H200 GPUs and CPUs.
Speaker similarity is calculated using cosine similarity between WavLM embeddings of the generated audio and reference clip.
The platform supports multilingual evaluation, toggling between English WER and character-based scores for Chinese, Japanese, and Korean.
A "Listen" tab allows users to compare outputs and vote on preferences, while a "Streaming" tab ranks models by TTFA for interactive applications.

The leaderboard aims to keep pace with rapid TTS releases and highlight open-source models often underrepresented in commercial arenas, inviting community feedback to shape future metrics and datasets.

Importance 1/3 New harness with differentiators Trust 1/3 Hugging Face Blog Hugging Face Evaluation & benchmarks Voice & audio

Related articles

lab Hugging Face Blog · 77d ago · 37 views

Hume releases Real World VoiceEQ benchmark for human quality of voice AI

Hume has introduced Real World VoiceEQ, a benchmark designed to evaluate the human quality of voice interaction by assessing how well systems recognize and produce acoustic information beyond transcripts. The benchmark evaluates over 40 proprietary and open-source models across 15+ dimensions using more than 60 metrics derived from 785,000 TTS and 48,000 STS human ratings.

arxiv arXiv cs.LG · 98d ago · 5 views

Kiwano: An Open-Source PyTorch Toolkit for Speaker Verification Research

Researchers have introduced Kiwano, an open-source toolkit designed to advance research and evaluation in the field of speaker verification. Built on PyTorch, this lightweight yet extensible framework provides standardized recipes, pretrained models, and integration of widely used architectures. The project emphasizes reproducibility by delivering transparent training pipelines, unified evaluation protocols, and ready-to-use baselines across multiple corpora. Beyond standard training and inference capabilities, Kiwano includes specialized tools for benchmarking, experiment tracking, and the rapid prototyping of new architectures. To encourage community adoption, the toolkit is distributed under the Apache 2.0 license and is accompanied by comprehensive documentation and reproducible experiments. By lowering entry barriers and standardizing evaluation practices, Kiwano aims to serve as a valuable resource for both academic research and applied development. The project is publicly available on GitHub at https://github.com/kiwano-toolkit/kiwano/.

media Hugging Face Forums · 3d ago · 9 views

Claude Opus 5 and Tetsu find six hollow CI checks in their own project

On September 27, Claude Opus 5 and the 3B model Tetsu identified six separate continuous integration checks in their public repository that reported green or passed despite failing to verify what they were supposed to measure. The issues ranged from a deployment gate refusing valid restarts due to stale digests to timing benchmarks with hollow thresholds that accepted negligible performance differences.

arxiv arXiv cs.LG · 22d ago · 28 views

TontaubeV1 enables streaming text-to-speech with bounded context

Researchers present TontaubeV1, a text-to-speech model that preserves natural prosody while enabling streaming inference from a single consumer GPU. The system uses a hierarchical DualCodec representation at 12.5 Hz to separate semantic streams from acoustic refinements, allowing for low-latency generation.

arxiv arXiv cs.CL · 22d ago · 26 views

TontaubeV1 enables streaming text-to-speech with bounded context

Researchers present TontaubeV1, a text-to-speech model that preserves natural prosody while enabling streaming inference from a single consumer GPU. The system uses a hierarchical DualCodec representation at 12.5 Hz to separate semantic streams from acoustic refinements, allowing causal decoding despite the noncausal nature of the underlying codec.