Benchmark · multimodal
MMMU
MMMU (Massive Multi-discipline Multimodal Understanding) tests whether a model can answer college-level questions that combine images and text across many academic disciplines. Results are reported as percent accuracy.
Read more
- Example
- A question might pair a chemistry diagram, a chart, or a medical image with text and ask the model to reason it through and choose the correct answer — the kind of item a university exam would contain.
- Scoring
- Every question is marked right or wrong, and the score is the percentage of questions answered correctly (accuracy).
- Verification
- Answers are verified automatically by exact match against the reference answer (mostly multiple-choice, some short open answers), so no human judging is required.
- Why it matters
- It measures expert, college-level reasoning over images and text across dozens of subjects, making it a demanding check of true multimodal understanding rather than text-only skill.
Worked example
MMMU (Massive Multi-discipline Multimodal Understanding) poses college-level questions that pair text with an image — a diagram, chart, chemical structure, or medical scan — across six disciplines, so a model must actually read the picture to answer. A representative Science item: the accompanying figure shows a circuit with a 12 V battery connected to two resistors in series, labeled R1 = 4 Ω and R2 = 2 Ω. Question: "What is the current supplied by the battery?", with choices (A) 1 A, (B) 2 A, (C) 3 A, (D) 6 A. The correct answer is (B) 2 A. Reasoning: series resistors add, so the total resistance is R = 4 Ω + 2 Ω = 6 Ω, and Ohm's law gives I = V / R = 12 V / 6 Ω = 2 A. The item is "multimodal" because the resistor values and their series arrangement can only be obtained by reading the schematic, not the text alone.
No verified scores reported yet for this benchmark.