lab ARC Prize · 422d ago

Testing major AI reasoning systems finds no clear winner on ARC benchmarks

from English

A review of frontier AI reasoning systems from labs including OpenAI, DeepMind, xAI, DeepSeek, and Anthropic reveals that while test-time compute scaling improves accuracy, it significantly reduces efficiency. The analysis identifies a Pareto frontier for ARC-AGI-1 scores but notes complete failure on the more challenging ARC-AGI-2 benchmark, suggesting common limitations across all major models.

Modern reasoning systems rely on long-running inference, knowledge recomposition, and parallel CoT sampling to extend "time to think."
OpenAI's o3-medium/high offers highest accuracy at high cost, while Google's Gemini 2.5 Flash provides a balance of accuracy and cost.
xAI's Grok 3 mini-low offers modest accuracy improvements over pure LLMs with lower resource requirements.
ARC-AGI-2 remains unsolved by all tested systems, indicating that current scaling strategies are insufficient for breakthrough AGI capability.

The findings suggest there is no single universal winner and that AI reasoning systems lack strong product-market fit compared to pure LLMs due to API inconsistencies and adoption friction.

Importance 1/3 ARC Prize OpenAI Anthropic Google DeepMind Benchmark results Reasoning models

Related articles

arxiv arXiv cs.CL · 27d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 43d ago · 1 view

REDACT: Multilingual PII Benchmark with Systematic Control

REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.

arxiv arXiv cs.CL · 43d ago · 2 views

Turing-RL: Learning User Simulators with Turing Rewards

Turing-RL introduces a reinforcement learning method using an LLM judge to evaluate how indistinguishable generated responses are from real user inputs. It outperforms baseline methods in both LLM and human evaluations across chat and Reddit forum domains, demonstrating that optimizing for indistinguishability improves user simulator performance.

arxiv arXiv cs.CL · 44d ago

Frustrated Synchronization Network Outperforms Transformers

The Frustrated Synchronization Network (FSN) achieves lower validation loss than a RoPE-SwiGLU transformer at every epoch on character-level text and code tasks. At one million parameters, FSN converges to a validation loss of 1.5953 ± 0.0014, outperforming the transformer's converged loss of 1.611. This advantage persists up to four million parameters, with ongoing evaluations beyond that scale.

arxiv arXiv cs.CL · 44d ago · 1 view

Decoupling Search from Reasoning in LLM Agents

Decoupled Search Grounding (DSG) separates search functionality from reasoning models, enabling vendor-agnostic, tunable, and reusable search grounding. DSG achieves near-native accuracy on SimpleQA with 91% lower search cost and 99.4% warm-cache hit rate, while reducing latency by 68% and preserving concise output contracts.