media LessWrong · 510d ago

Epoch's evaluation of o3-mini on FrontierMath yields 11% vs OpenAI's 32%

from English

OpenAI claims its o3-mini model with high reasoning and Python tool access achieves a 32% score on the FrontierMath benchmark. However, Epoch's official independent evaluation reports a significantly lower score of only 11%.

Epoch built the benchmark and has better incentives for accurate scoring.
OpenAI reported a 28% score on the hardest problem tier, which is suspiciously close to its overall score.
Epoch has published extensive information about its testing infrastructure and data, whereas OpenAI has published little.
The discrepancy may stem from OpenAI using a more powerful internal scaffold, more test-time compute, or a different subset of problems (180 vs 290).

The difference in results highlights the importance of transparent evaluation methodologies and independent verification in benchmarking large language models.

Importance 2/3 LessWrong OpenAI Benchmark results Evaluation & benchmarks

Benchmarks

Benchmark	Model	Score
FrontierMath	o3-mini with high reasoning and a Python tool	32%

Read original

arxiv arXiv cs.CL · 35d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 51d ago · 2 views

Benchmarking Agentic Review Systems for AI-Assisted Research

A study evaluates four AI review systems across six language models, finding OpenAIReview with GPT-5.5 achieves 83.0% accuracy in matching paper quality to external signals and detects 71.6% of injected errors. Real user feedback shows positive sentiment, with a 1.44-to-1 vote ratio, though false positives and minor nitpicks remain common.

arxiv arXiv cs.CL · 51d ago · 1 view

REDACT: Multilingual PII Benchmark with Systematic Control

REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.

media r/LocalLLaMA · 51d ago · 1 view

GLM-5.2 Outperforms GPT-5.5 in AA-Briefcase Evaluation

Artificial Analysis' new agentic knowledge work evaluation, AA-Briefcase, shows GLM-5.2 surpassing GPT-5.5 in performance. The benchmark assesses real-world task execution and reasoning capabilities in knowledge work scenarios.

arxiv arXiv cs.AI · 52d ago

Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images

A new benchmark evaluates AI-generated text-rich images across six domains, including commercial posters and receipts. It reveals significant domain-dependent performance and sensitivity to JPEG compression, highlighting the need for text- and layout-aware detection methods.