The authors introduce PluraMath, a new dataset that extends the PolyMath benchmark to 18 additional underrepresented languages spanning six language families. This work addresses the heavy bias toward high-resource languages like English and Chinese in existing mathematical reasoning evaluations for Large Language Models.
- The dataset covers mid-resource to extreme low-resource settings, filling gaps left by PolyMath's 18-language limit.
- Data was constructed via a human-curated pipeline where native speakers validated pre-computed translations.
- The authors benchmarked 27 reasoning LLMs across small, mid-size, large, and closed-source ensemble scales.
- Analysis reveals a persistent performance gap between high-resource and underrepresented languages, linked to instruction-following ability.
The dataset, acquisition pipeline, and evaluation framework are fully open-sourced to lower barriers for multilingual benchmark development.