lab ARC Prize · 514d ago

ARC Prize Foundation launches ARC-AGI-2 benchmark and ARC Prize 2025

from English

The ARC Prize Foundation has launched ARC-AGI-2, a new benchmark designed to measure fluid intelligence by presenting tasks that are easy for humans but difficult for current AI systems. Alongside the benchmark, the organization announced the ARC Prize 2025 competition to drive open-source progress on efficient general intelligence.

ARC-AGI-2 raises the difficulty bar significantly compared to its predecessor, with pure LLMs scoring 0% and public reasoning systems achieving only single-digit percentages. The benchmark consists of 120 tasks per eval set (up from 100) that require symbolic interpretation, compositional reasoning, and contextual rule application. Human participants solved every task in two attempts or fewer, highlighting the gap between human adaptability and current AI capabilities.

The initiative aims to serve as a North Star for AGI research by focusing on capability gaps that do not emerge from scaling up, inspiring researchers to develop novel ideas for general intelligence.

Importance 2/3 ARC Prize Evaluation & benchmarks Reasoning models

Benchmarks

Benchmark	Model	Score
ARC-AGI 2	public AI reasoning systems	9%

Related articles

media Hugging Face Forums · 17d ago · 25 views

GPT-5.6 recovers 95% of hidden messages in unseen literary cryptography benchmark

A working paper by Joseph JM Walker evaluates frontier language models on an unseen multi-channel literary cryptography benchmark embedded in the physical novel "I Wrote a Book and Made a Million Dollars (I.B. Wryten)." The study finds that GPT-5.6 independently recognized the secondary communication channel and recovered approximately 95% of the embedded material on its first pass, whereas earlier models demonstrated effectively 0% ability.

arxiv arXiv cs.CL · 35d ago · 29 views

GSM-Plus-BN introduces perturbation-based benchmark for Bangla math reasoning

The study introduces GSM-Plus-BN, a novel perturbed Bengali mathematical dataset derived from the English GSM-Plus benchmark and verified by human translators. This resource addresses the lack of systematic benchmarks for assessing model robustness in linguistically diverse regions like Bangladesh.

arxiv arXiv cs.AI · 37d ago · 20 views

WILDTRACE introduces benchmark for natural evidence trails in long-context reasoning

The authors introduce WILDTRACE, a new benchmark designed to evaluate how well models integrate evidence dispersed across distant passages in long documents. Unlike existing benchmarks that rely on planted facts or reverse-engineered chains, WILDTRACE uses 481 tasks derived from 214 naturally occurring sources, such as technical incident reports and literary narratives.

arxiv arXiv cs.CL · 38d ago · 20 views

WILDTRACE benchmark introduces 481 tasks for natural evidence trails in long-context reasoning

The authors introduce WILDTRACE, a new benchmark designed to evaluate how well models integrate evidence dispersed naturally across distant passages in long documents. Unlike existing benchmarks that use planted facts or reverse-engineered chains, WILDTRACE relies on source-internal causal, temporal, and narrative logic.

lab OpenAI News · 51d ago · 25 views

Introducing GeneBench-Pro

GeneBench-Pro is a research-level benchmark designed to measure how AI agents handle ambiguity and make consequential judgments in computational biology, expanding on the original GeneBench. It addresses the limitation of current assessments by testing higher-order capabilities such as handling data noise, revising assumptions, and determining when results are decision-ready.