Benchmark · reasoning
ARC-AGI 2
ARC-AGI 2 (Abstraction & Reasoning Corpus, version 2, run as the ARC Prize 2025) measures fluid reasoning: inferring an abstract rule from a few examples and applying it to a new case. The score is the percentage of puzzles solved correctly on a held-out, semi-private evaluation set.
Read more
- Example
- A task gives a few colored-grid input→output pairs (a small grid of colored cells changed in one consistent way), and the solver must work out the hidden transformation and produce the correct output grid for a new input.
- Scoring
- Each puzzle is pass/fail: the produced output grid is compared to the exact correct grid, and the reported score is the percentage of puzzles solved on the semi-private evaluation set.
- Verification
- Results are checked automatically by exact match on the output grid — the predicted grid must match the answer cell for cell — on a semi-private test set that is not publicly released to prevent memorization.
- Why it matters
- It focuses on puzzles that are easy for people but hard for AI, so it is a closely watched measure of real abstraction and general reasoning rather than memorized knowledge.
Worked example
Task
ARC-AGI-2 grid puzzle: infer the transformation from the training pairs, then give the output grid for the test input (digits 0–9 are colors, 0 = black background). Train 1: [[4,0,0,0],[0,0,6,0],[4,0,0,0],[0,0,6,0]] → [[0,0,0,0],[0,0,0,0],[4,0,6,0],[4,0,6,0]]. Train 2: [[0,3,0,0],[0,0,0,8],[0,3,0,0],[0,0,0,0]] → [[0,0,0,0],[0,0,0,0],[0,3,0,0],[0,3,0,8]]. Test: [[0,0,2,0],[5,0,0,0],[0,0,2,0],[5,0,0,0]] → ?
Solution
[[0, 0, 0, 0],
[0, 0, 0, 0],
[5, 0, 2, 0],
[5, 0, 2, 0]]
Walkthrough
Every training pair shows one rule — 'gravity': each colored cell falls straight down to the lowest free cells of its column, keeping its color and count, while the 0-background moves to the top. On the test, column 0's two 5s and column 2's two 2s settle into the bottom two rows. ARC-AGI-2 grades by exact grid match — right dimensions and every cell value — scored pass@2 over two allowed attempts.