Benchmark · reasoning
ARC-AGI 1
ARC-AGI 1 measures abstract reasoning and generalization: each task shows a few input→output grid examples, and the solver must infer the hidden transformation rule and produce the correct output grid for a new input. The metric is the percentage of tasks solved with an exactly matching output grid.
Read more
- Example
- A task might present two or three small colored grids where each input turns into its output by one consistent rule — for instance filling every enclosed region or mirroring a shape — and you must apply that inferred rule to a new, unseen grid.
- Scoring
- A task counts as solved only when the produced grid matches the reference answer cell-for-cell; the final score is the fraction (percentage) of tasks solved across the evaluation set.
- Verification
- Checking is automatic and exact: the submitted output grid is compared cell-by-cell with the ground-truth grid, with no partial credit and no human voting.
- Why it matters
- The puzzles are built to be easy for people but hard for models that lean on memorized patterns, so the benchmark probes real abstraction and few-shot generalization rather than recalled knowledge — making it a closely watched measure of progress toward more general AI.
Worked example
Task
ARC-AGI 1 shows a few input→output grid demonstrations, then a held-out test input; you must infer the transformation and draw the exact output grid (cells are colors 0-9, 0=black). Here the demos reveal a self-fractal rule: each 3×3 grid expands to 9×9, stamping a copy of the whole grid wherever a cell is colored and a blank 3×3 block wherever it is 0. Test input (3×3):
0 8 0 / 8 8 8 / 0 8 0 (a plus shape in color 8).Solution
0 0 0 0 8 0 0 0 0
0 0 0 8 8 8 0 0 0
0 0 0 0 8 0 0 0 0
0 8 0 0 8 0 0 8 0
8 8 8 8 8 8 8 8 8
0 8 0 0 8 0 0 8 0
0 0 0 0 8 0 0 0 0
0 0 0 8 8 8 0 0 0
0 0 0 0 8 0 0 0 0
Walkthrough
The color-8 cells sit at the four edge-midpoints and the center of the plus; stamping the plus into exactly those five 3×3 blocks (and leaving the four corners blank) builds the 9×9 fractal. ARC-AGI scores an exact cell-for-cell grid match — a single wrong cell fails the task, with up to 2 attempts allowed.