media Passionfruit · 521d ago

Benchmark comparison of Claude 3.7 Sonnet, o3-mini, R1, and Grok 3 Beta

from English

A comparative analysis evaluates the capabilities of Claude 3.7 Sonnet, Claude 3.5 Sonnet, OpenAI o3-mini, DeepSeek R1, and Grok 3 Beta across math, coding, and reasoning benchmarks.

On GPQA Diamond, Grok 3 Beta reaches 84.6% with extended reasoning, while Claude 3.7 Sonnet achieves 84.8% in the same mode.
In AIME math competitions, Grok 3 Beta leads with 93.3%, whereas Claude 3.7 Sonnet scores 61.3% but excels on Math500 at 96.2%.
OpenAI o3-mini demonstrates strong performance on AIME (up to 83.3%) and GPQA Diamond (78% in higher-compute mode) with a focus on cost-effectiveness.
DeepSeek R1 performs competitively on AIME (79.8%) but shows variability on graduate-level reasoning tasks.

The study aims to help users make informed decisions by highlighting trade-offs in cost, context window size, and reliability across these leading models.

Importance 2/3 Passionfruit Anthropic DeepSeek OpenAI Benchmark results Reasoning models

Benchmarks

Benchmark	Model	Score
AIME 2025	Grok 3 Beta	93.3%
GPQA Diamond	Claude 3.7 Sonnet	84.8%
GPQA Diamond	Grok 3 Beta	84.6%
AIME 2025	OpenAI o3-mini	83.3%
AIME 2025	DeepSeek R1	79.8%
GPQA Diamond	OpenAI o3-mini	78%
GPQA Diamond	DeepSeek R1	71.5%
GPQA Diamond	Claude 3.5 Sonnet	65%
AIME 2025	Claude 3.7 Sonnet	61.3%
AIME 2025	Claude 3.5 Sonnet	41.3%
MATH-500	Claude 3.7 Sonnet	96.2%
MATH-500	Claude 3.5 Sonnet	82.2%

Read original

media Vellum · 523d ago

Anthropic releases Claude 3.7 Sonnet with dynamic reasoning control

Anthropic has released Claude 3.7 Sonnet, a model designed for real-world business and developer use cases rather than just benchmark optimization. The release introduces the ability to dynamically switch between standard and advanced reasoning modes, allowing users to control the number of tokens allocated to "thinking time."

lab ARC Prize · 422d ago

Testing major AI reasoning systems finds no clear winner on ARC benchmarks

A review of frontier AI reasoning systems from labs including OpenAI, DeepMind, xAI, DeepSeek, and Anthropic reveals that while test-time compute scaling improves accuracy, it significantly reduces efficiency. The analysis identifies a Pareto frontier for ARC-AGI-1 scores but notes complete failure on the more challenging ARC-AGI-2 benchmark, suggesting common limitations across all major models.

media Walturn · 669d ago

Comparing OpenAI o1 to other Top Models

This article examines and compares four prominent AI models—OpenAI's o1, Meta's LLaMA 3.2, Anthropic's Claude 3.5 Sonnet, and Google's Gemini 1.5 Pro—focusing on their inference capabilities, training methodologies, and performance benchmarks.

lab ARC Prize · 687d ago

OpenAI o1 models outperform GPT-4o on ARC-AGI-Pub but require significantly more test-time compute

OpenAI released the o1-preview and o1-mini models, which are trained to emulate reasoning by generating and refining reasoning tokens before providing a final answer. Testing on the ARC-AGI public evaluation dataset shows that both o1 variants outperform GPT-4o in accuracy.

media r/LocalLLaMA · 2h ago

DeepSeek V4 Flash GA ranks same as Sonnet 5 and Grok 4.5 on DeepSWE

DeepSeek claims that its newly generally available model, DeepSeek V4 Flash, achieves performance parity with Anthropic's Sonnet 5 and xAI's Grok 4.5 on the DeepSWE benchmark.