Benchmark · multimodal

CharXiv

0 results 0 models

CharXiv is a benchmark that tests how well multimodal LLMs understand real scientific charts taken from arXiv papers; it reports accuracy separately for descriptive questions (reading basic chart elements) and reasoning questions (comparing and analyzing values).

Example: Given a chart image, a descriptive task might ask you to read the title, count the legend entries, or state a tick label, while a reasoning task might ask you to compare two plotted series and say which is larger at a given point — always answerable from the chart's visuals alone.
Scoring: Every question takes a short free-form answer. GPT-4o serves as an automatic judge that extracts the model's answer and assigns a binary correct/incorrect against the human-validated ground truth; the score is accuracy (percent correct), reported separately for the descriptive and reasoning subsets.
Verification: The 2,323 charts, their questions, and ground-truth answers are all hand-curated and checked by human experts; a response is accepted as correct only when the GPT-4o grader judges it to match the reference answer, with reported human accuracy (about 80.5% on reasoning) as the reference ceiling.
Why it matters: Reading charts is central to applying MLLMs to scientific papers and financial reports, but earlier benchmarks used simple templated charts that overstated progress; CharXiv's natural, diverse arXiv charts expose a large gap between the best models and humans, making it a demanding, realistic test of visual reasoning.

Worked example

Task

[Illustrative reasoning item] A grouped bar chart titled 'Validation Accuracy by Model Size'. X-axis: model size (7B, 13B, 70B); Y-axis: accuracy (%). Each group has two bars, Dataset A and Dataset B: 7B → A=61, B=58; 13B → A=68, B=62; 70B → A=79, B=71. Question (give a short answer): For which model size is the accuracy gap between Dataset A and Dataset B the largest?

Solution

Per-group gap A − B: 7B → 61 − 58 = 3; 13B → 68 − 62 = 6; 70B → 79 − 71 = 8. Largest = 8 → answer: 70B.

Walkthrough

It is a reasoning question because you must read all six bar heights and compare their differences, not just extract one value; the GPT-4o judge maps the short answer to the ground truth '70B' and scores it binary, contributing to the reasoning-accuracy metric.

No verified scores reported yet for this benchmark.