Benchmark · multimodal
CharXiv
CharXiv is a benchmark that tests how well multimodal LLMs understand real scientific charts taken from arXiv papers; it reports accuracy separately for descriptive questions (reading basic chart elements) and reasoning questions (comparing and analyzing values).
Read more
- Example
- Given a chart image, a descriptive task might ask you to read the title, count the legend entries, or state a tick label, while a reasoning task might ask you to compare two plotted series and say which is larger at a given point — always answerable from the chart's visuals alone.
- Scoring
- Every question takes a short free-form answer. GPT-4o serves as an automatic judge that extracts the model's answer and assigns a binary correct/incorrect against the human-validated ground truth; the score is accuracy (percent correct), reported separately for the descriptive and reasoning subsets.
- Verification
- The 2,323 charts, their questions, and ground-truth answers are all hand-curated and checked by human experts; a response is accepted as correct only when the GPT-4o grader judges it to match the reference answer, with reported human accuracy (about 80.5% on reasoning) as the reference ceiling.
- Why it matters
- Reading charts is central to applying MLLMs to scientific papers and financial reports, but earlier benchmarks used simple templated charts that overstated progress; CharXiv's natural, diverse arXiv charts expose a large gap between the best models and humans, making it a demanding, realistic test of visual reasoning.
Worked example
Task
[Illustrative reasoning item] A grouped bar chart titled 'Validation Accuracy by Model Size'. X-axis: model size (7B, 13B, 70B); Y-axis: accuracy (%). Each group has two bars, Dataset A and Dataset B: 7B → A=61, B=58; 13B → A=68, B=62; 70B → A=79, B=71. Question (give a short answer): For which model size is the accuracy gap between Dataset A and Dataset B the largest?
Solution
Per-group gap A − B: 7B → 61 − 58 = 3; 13B → 68 − 62 = 6; 70B → 79 − 71 = 8. Largest = 8 → answer: 70B.
Walkthrough
It is a reasoning question because you must read all six bar heights and compare their differences, not just extract one value; the GPT-4o judge maps the short answer to the ground truth '70B' and scores it binary, contributing to the reasoning-accuracy metric.
| Date | Model | Score | Source |
|---|---|---|---|
| 2026-08-03 | Inkling-Small | 77.4% | Thinking Machines Lab releases Inkling-Small, a 276B open weights multimodal MoE model |
| 2026-07-17 | Inkling | 82.0% | Thinking Machines Lab releases Inkling, a 975B-parameter open-weights multimodal MoE model |
| 2025-12-11 | GPT-5.2 Thinking | 88.7% | OpenAI releases GPT-5.2, outperforming Gemini 3 in coding and reasoning benchmarks |
| 2025-04-17 | o4-mini | 72.0% | OpenAI releases o4-mini, a faster, cheaper multimodal reasoning model |
| 2024-06-20 | Claude 3.5 Sonnet | 95.2% | Anthropic releases Claude 3.5 Sonnet addendum with improved coding and vision benchmarks |