arxiv Moonshot AI (arXiv) · 797d ago

MMLU-Pro introduces a more robust benchmark with 10 options and reasoning-focused questions

from English

Researchers introduce MMLU-Pro, an enhanced version of the Massive Multitask Language Understanding benchmark designed to address performance saturation and instability in current large language models. The new dataset expands the choice set from four to ten options, integrates challenging college-level reasoning questions, and eliminates noisy items through expert review.

Accuracy drops by 16% to 33% compared to MMLU, with top model GPT-4o scoring only 72.6%.
Model score sensitivity to prompt variations decreases from 4-5% in MMLU to just 2% in MMLU-Pro across 24 styles.
Chain of Thought reasoning significantly boosts performance on MMLU-Pro, improving GPT-4o by 19%, whereas it hurts results on the original benchmark.
The benchmark spans 14 domains with over 12,000 questions, providing greater discriminative power between models like GPT-4o and GPT-4-Turbo.

MMLU-Pro serves as a more reliable and difficult metric for tracking progress in expert-level language understanding and reasoning capabilities.

Importance 1/3 Moonshot AI (arXiv) Benchmark results Evaluation & benchmarks

Benchmarks

Benchmark	Model	Score
MMLU-Pro	GPT-4o	72.6%
MMLU-Pro	GPT-4-Turbo	63.7%

Related articles

arxiv arXiv cs.AI · 3d ago

SciCode-Verified corrects benchmark defects to reveal models' true scientific-coding ability

The authors identify that plateauing scores on the SciCode benchmark stem from significant defects in the evaluation instrument rather than limitations in model capability. A domain-expert audit of 65 test problems uncovered 263 defects, with 192 causing correct solutions to be wrongly rejected due to issues like non-reproducible gold answers and over-tight tolerances.

media Hugging Face Forums · 7d ago · 8 views

Levent Bulut benchmarks LLM annotation reliability on objective projection

Levent Bulut published a three-study benchmark evaluating the reliability of machine-generated annotations for a Turkish narrative corpus, revealing significant discrepancies between automated raters and human judgment. The study tested six binary craft features across 120 and 100 scenes using rule-based detectors and models including Gemini 2.5 Flash, Grok, ChatGPT 5.5, and Claude Fable 5 High.

arxiv τ²-bench (tau2-bench) · 17d ago · 1 view

τ-bench 1.0.1 fixes banking_knowledge grading to prevent penalizing cautious agent behavior

Sierra Research released τ-bench 1.0.1, which corrects the `banking_knowledge` task grading scheme to stop penalizing agents for prudent verification reads and fixes several data inconsistencies. The update ensures that re-grading leaderboard trajectories only increases scores, with no previously passing simulations failing.

arxiv arXiv cs.CL · 24d ago · 1 view

Variance decomposition reveals query language dominates LLM brand answer noise

A study decomposes the non-determinism in large language model (LLM) brand recommendations into four separable sources: within-prompt resampling, prompt paraphrase, model identity, and query language. Using a crossed random-effects analysis on 12,933 responses across three models and eight languages, the authors identify query language as the largest source of variance at 26.5%, while brand identity accounts for only 1.5%.

arxiv arXiv cs.CL · 26d ago · 8 views

AdvancedMathBench introduces ProverBench and VerifierBench for advanced mathematical proof evaluation

Researchers introduce AdvancedMathBench, a benchmark suite designed to evaluate the advanced mathematical reasoning capabilities of large language models, addressing gaps in disciplinary scope and evaluation granularity found in existing benchmarks.