arxiv arXiv cs.AI · 2h ago · research

SciCode-Verified corrects benchmark defects to reveal models' true scientific-coding ability

from English

The authors identify that plateauing scores on the SciCode benchmark stem from significant defects in the evaluation instrument rather than limitations in model capability. A domain-expert audit of 65 test problems uncovered 263 defects, with 192 causing correct solutions to be wrongly rejected due to issues like non-reproducible gold answers and over-tight tolerances.

The team corrected every confirmable defect to create SciCode-Verified, adding necessary specifications and repairing grading logic.
Re-evaluating twelve frontier model snapshots on the corrected benchmark showed subproblem accuracy rising from 45–60% to 84–98%.
Main-problem accuracy increased significantly from 9–27% to 69–92%, demonstrating that state-of-the-art models are far more proficient than previously suggested.

The authors release SciCode-Verified with its complete audit trail as the corrected public standard, establishing that the previous bottleneck was the quality of the evaluation instrument.

Importance 2/3 New harness with differentiators arXiv cs.AI Benchmark results Evaluation & benchmarks

Read original

media Hugging Face Forums · 4d ago · 8 views

Levent Bulut benchmarks LLM annotation reliability on objective projection

Levent Bulut published a three-study benchmark evaluating the reliability of machine-generated annotations for a Turkish narrative corpus, revealing significant discrepancies between automated raters and human judgment. The study tested six binary craft features across 120 and 100 scenes using rule-based detectors and models including Gemini 2.5 Flash, Grok, ChatGPT 5.5, and Claude Fable 5 High.

arxiv τ²-bench (tau2-bench) · 14d ago · 1 view

τ-bench 1.0.1 fixes banking_knowledge grading to prevent penalizing cautious agent behavior

Sierra Research released τ-bench 1.0.1, which corrects the `banking_knowledge` task grading scheme to stop penalizing agents for prudent verification reads and fixes several data inconsistencies. The update ensures that re-grading leaderboard trajectories only increases scores, with no previously passing simulations failing.

arxiv arXiv cs.CL · 21d ago · 1 view

Variance decomposition reveals query language dominates LLM brand answer noise

A study decomposes the non-determinism in large language model (LLM) brand recommendations into four separable sources: within-prompt resampling, prompt paraphrase, model identity, and query language. Using a crossed random-effects analysis on 12,933 responses across three models and eight languages, the authors identify query language as the largest source of variance at 26.5%, while brand identity accounts for only 1.5%.

arxiv arXiv cs.CL · 23d ago · 8 views

AdvancedMathBench introduces ProverBench and VerifierBench for advanced mathematical proof evaluation

Researchers introduce AdvancedMathBench, a benchmark suite designed to evaluate the advanced mathematical reasoning capabilities of large language models, addressing gaps in disciplinary scope and evaluation granularity found in existing benchmarks.

arxiv arXiv cs.CL · 29d ago · 6 views

DataGovBench benchmarks LLMs against real-world data complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings by focusing on fact retrieval from small tables while overlooking challenges like large multi-tabular datasets and exploratory insight discovery. To address this, the authors introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.