Benchmark · multimodal

MMMU-Pro

0 results 0 models

MMMU-Pro is a robustness-hardened version of the MMMU benchmark that measures college-level multimodal (image + text) understanding and reasoning across six disciplines; the metric is multiple-choice accuracy.

Example: A discipline question pairs an image — a diagram, chart, chemical structure, or medical scan — with a college-level question and up to 10 answer options; in the vision-only setting the whole question and its options are embedded inside a screenshot or photo, so the model must read the text out of the image itself.
Scoring: Accuracy: the model's chosen option is exact-matched against the gold key and the score is the percentage of items answered correctly. Candidate options are augmented from 4 to 10 to blunt random guessing, and an overall MMMU-Pro score averages the Standard (10-option) and Vision-input settings (Chain-of-Thought prompting is typically used).
Verification: Every item is single-answer multiple choice, so grading is automatic: the final selected option is parsed from the model's (often chain-of-thought) output and compared for exact match with the reference answer. Robustness is baked into the data — questions that text-only models can answer are filtered out and the option count is raised to 10 — so a high score reflects genuine multimodal reasoning rather than text shortcuts or lucky guesses.
Why it matters: The original MMMU inflated scores because models could exploit text-only shortcuts or guess among four options; MMMU-Pro removes both, so measured accuracy drops sharply and better separates models that truly integrate vision and text, making it a more honest gauge of expert-level multimodal ability.

Worked example

Task

Representative Science (Physics) item in MMMU-Pro format. [Image: a block on a frictionless incline of angle θ = 30°.] Question: What is the magnitude of the block's acceleration along the incline? Options (10): (A) 0 m/s² (B) 1.6 m/s² (C) 2.5 m/s² (D) 3.3 m/s² (E) 4.9 m/s² (F) 5.7 m/s² (G) 6.9 m/s² (H) 7.4 m/s² (I) 8.5 m/s² (J) 9.8 m/s². In the vision-only setting this entire prompt appears embedded inside a screenshot.

Solution

a = g·sin θ = 9.8 × sin 30° = 9.8 × 0.5 = 4.9 m/s² → (E)

Walkthrough

On a frictionless incline the only force acting along the surface is the gravity component mg·sin θ, so the acceleration is g·sin θ and is independent of the block's mass; g·sin 30° = 4.9 m/s², which matches option (E). Grading compares the parsed option letter to the gold key by exact match.

No verified scores reported yet for this benchmark.