Benchmark · multimodal
MMMU
MMMU (Massive Multi-discipline Multimodal Understanding) tests whether a model can answer college-level questions that combine images and text across many academic disciplines. Results are reported as percent accuracy.
Read more
- Example
- A question might pair a chemistry diagram, a chart, or a medical image with text and ask the model to reason it through and choose the correct answer — the kind of item a university exam would contain.
- Scoring
- Every question is marked right or wrong, and the score is the percentage of questions answered correctly (accuracy).
- Verification
- Answers are verified automatically by exact match against the reference answer (mostly multiple-choice, some short open answers), so no human judging is required.
- Why it matters
- It measures expert, college-level reasoning over images and text across dozens of subjects, making it a demanding check of true multimodal understanding rather than text-only skill.
Worked example
Task
(Image: a circuit diagram — a 12 V battery in a single loop with two resistors in series, R1 = 4 Ω and R2 = 8 Ω.) What is the current through resistor R2? Options: (A) 1.0 A (B) 1.5 A (C) 2.0 A (D) 3.0 A.
Solution
R_total = R1 + R2 = 4 Ω + 8 Ω = 12 Ω; I = V / R_total = 12 V / 12 Ω = 1.0 A (series ⇒ same current in R2). Answer: (A).
Walkthrough
In a series circuit the same current flows through every element and the total resistance is the sum of the parts, so I = 12/12 = 1.0 A. MMMU scores it by exact match of the predicted option letter against the single gold answer.
| Date | Model | Score | Source |
|---|---|---|---|
| 2025-08-07 | GPT-5 | 84.2% | OpenAI introduces GPT-5 with unified routing and expert-level reasoning |
| 2025-08-07 | GPT-5 | 84.2% | OpenAI launches GPT-5 with adaptive reasoning and unified architecture |
| 2025-04-17 | o4-mini | 81.6% | OpenAI releases o4-mini, a faster, cheaper multimodal reasoning model |
| 2025-04-15 | Gemini 2.0 Flash | 71.7% | Google releases Gemini 2.0 Flash with enhanced quality and twice the speed of Gemini 1.5 Pro |
| 2025-03-26 | Gemini 2.5 Pro | 81.7% | Google releases experimental Gemini 2.5 Pro reasoning model with 1M token context |
| 2024-05-13 | GPT-4o | 69.1% | OpenAI |
| 2024-03-04 | Claude 3 Opus | 59.4% | Anthropic |
| 2023-11-27 | GPT-4V | 56.8% | MMMU paper |