The Open ASR Leaderboard has introduced its first Global South languages with two new evaluation sets: Monsoon en-IN (Indian English) and Monsoon hi-IN (Hindi). These additions address the lack of demographic diversity in current benchmarks, which have historically focused on European languages.
- The datasets comprise 4,888 speaker-disjoint speakers across four public and private splits.
- Data was collected via the Voice Arena platform from hundreds of districts to ensure broad geographic and device coverage.
- Each segment includes 12 metadata fields such as age, gender, occupation, and handset model.
- Hindi sets utilize lattices to handle orthographic variation rather than standard string references.
- The collection varies along nine axes including geography, age, gender, vocabulary, and acoustic environments.
This expansion allows for disaggregated analysis of ASR performance across diverse populations, revealing error rate disparities that aggregate WER scores hide.