Benchmark · multimodal

MathVista

0 results 0 models

MathVista is a multimodal benchmark that measures mathematical reasoning over visual inputs — figures, charts, geometric diagrams, and scientific plots. Performance is reported as accuracy: the share of questions answered correctly.

Example: A typical item pairs an image — a geometry diagram, a function plot, a bar chart, or a table — with a question solvable only by reading fine-grained visual detail and then reasoning mathematically, such as computing a length from a labeled figure or reading a value off a chart. Items are either multiple-choice or free-form (integer, decimal, or list).
Scoring: The metric is accuracy. Because answers are free-form or multiple-choice, an LLM first extracts the short final answer from the model's full response; it is normalized and compared to the ground truth by exact match. Accuracy is the fraction of correct items, also reported broken down by task type and reasoning skill.
Verification: Evaluation uses two splits: a 1,000-item testmini set with public answers for local scoring, and a larger test set of about 5,141 items whose answers are withheld and scored only by submitting predictions to the official evaluation leaderboard. A result counts as correct when the answer-extraction-plus-exact-match pipeline accepts it.
Why it matters: Text-only math benchmarks and generic visual-question-answering benchmarks each miss the intersection MathVista targets: mathematics that is impossible without genuine visual understanding. It exposed a wide gap between models and humans and became a standard yardstick for the visual mathematical reasoning of multimodal foundation models.

Worked example

Task

Image: a right triangle with its two perpendicular legs labeled 6 and 8. Question: 'What is the length of the hypotenuse in the figure?' Answer type: free-form integer.

Solution

By the Pythagorean theorem, the hypotenuse c = √(6² + 8²) = √(36 + 64) = √100 = 10. Final answer: 10.

Walkthrough

Solving it requires reading the labeled side lengths from the diagram (visual understanding) and then applying the Pythagorean theorem (mathematical reasoning). Grading: an LLM extracts the short answer '10' from the response, which is matched against the ground truth by normalized exact match.

No verified scores reported yet for this benchmark.