Benchmark · general
MMLU
MMLU (Massive Multitask Language Understanding) tests a model's breadth of knowledge across 57 subjects—from history and law to math and medicine—using multiple-choice questions. Performance is reported as % accuracy: the share of questions answered correctly.
Read more
- Example
- A typical item gives a question and four labeled answer choices (A–D), and the model must pick the correct one—for instance, a college-level medicine question or an elementary math problem.
- Scoring
- The metric is accuracy: the percentage of multiple-choice questions for which the model selects the correct option, averaged across all 57 subjects.
- Verification
- Grading is automatic and objective—the model's chosen letter is exact-matched against the known correct answer, so no human judgment is needed.
- Why it matters
- MMLU became a standard yardstick for general knowledge and reasoning breadth, but top models now score so high that it is largely saturated and less useful for separating the best systems.
Worked example
Task
The following is a multiple-choice question about high school chemistry. Which of the following is a strong acid? A) HF B) HCl C) CH3COOH D) H2CO3
Solution
HCl ionizes completely in water, so it is a strong acid; HF, CH3COOH, and H2CO3 ionize only partially (weak acids). Final answer: B) HCl.
Walkthrough
A strong acid fully dissociates into H+ and its conjugate base in aqueous solution, which HCl does and the others do not. MMLU grades it by exact-match accuracy: the predicted option letter must equal the gold label (B).