Benchmark · general
MMLU-Pro
MMLU-Pro is a harder, reasoning-heavy successor to MMLU that tests a model's knowledge and problem-solving across many academic disciplines with multiple-choice questions, each offering 10 answer options. Results are reported as percentage accuracy — the share of questions answered correctly.
Read more
- Example
- A typical item is a multiple-choice question from a field like mathematics, physics, law, or engineering that needs several reasoning steps, where the model must choose the single correct answer among 10 options (versus 4 in the original MMLU).
- Scoring
- The metric is accuracy: the percentage of questions for which the model picks the correct option, computed as correct answers divided by total questions.
- Verification
- Each response is graded automatically by exact-matching the model's chosen option against the known correct label, so no human judging is involved.
- Why it matters
- Because leading models had nearly saturated the original MMLU, MMLU-Pro's tougher questions and 10 options make it more discriminating and more sensitive to real reasoning, giving a cleaner measure of progress.
Worked example
Task
MMLU-Pro (physics, 10 options). A projectile is launched from ground level at 20 m/s, 30° above the horizontal (g = 10 m/s², no air resistance). What is its maximum height? A) 2.5 m B) 5.0 m C) 7.5 m D) 10.0 m E) 12.5 m F) 15.0 m G) 17.3 m H) 20.0 m I) 3.5 m J) 8.7 m
Solution
v_y = v·sin30° = 20 × 0.5 = 10 m/s; h_max = v_y² / (2g) = 10² / (2 × 10) = 5.0 m → B
Walkthrough
At the apex the vertical velocity is zero, so h_max = v_y²/(2g) with v_y = v·sin30°; only the vertical component sets the height. MMLU-Pro grades by exact-match of the chosen letter against the gold key (accuracy over its 10 options).