arxiv τ²-bench (tau2-bench) · 1h ago · evaluation

τ-bench 1.0.1 fixes banking_knowledge grading to prevent penalizing cautious agent behavior

from English

Sierra Research released τ-bench 1.0.1, which corrects the `banking_knowledge` task grading scheme to stop penalizing agents for prudent verification reads and fixes several data inconsistencies. The update ensures that re-grading leaderboard trajectories only increases scores, with no previously passing simulations failing.

Extra database read calls that were not in the golden trajectory no longer zero the reward by poisoning the DB-hash comparison.
Numeric tool arguments are normalized to prevent integer vs. float spelling from altering deterministic record IDs.
Gold trajectories for tasks 077–086 now include necessary reads for lost/stolen card scenarios.
Bank account transaction outputs are sorted by date descending to resolve tie-breakers correctly.
Contradictory cash-back rates in the Platinum Rewards knowledge doc and under-refunded ATM fees in task_074 have been corrected.

The re-grade shows upward score movements for all models, with gpt-5-5 gaining 9.02 pass^1 points. Users can re-grade their own results using the new `--fresh-tasks` flag or pin to the `pre-v1.0.1` tag to reproduce previous scores.

Importance 1/3 Trust 2/3 τ²-bench (tau2-bench) Benchmark results Evaluation & benchmarks

Related articles

arxiv arXiv cs.CL · 6d ago

Variance decomposition reveals query language dominates LLM brand answer noise

A study decomposes the non-determinism in large language model (LLM) brand recommendations into four separable sources: within-prompt resampling, prompt paraphrase, model identity, and query language. Using a crossed random-effects analysis on 12,933 responses across three models and eight languages, the authors identify query language as the largest source of variance at 26.5%, while brand identity accounts for only 1.5%.

arxiv arXiv cs.CL · 9d ago · 2 views

AdvancedMathBench introduces ProverBench and VerifierBench for advanced mathematical proof evaluation

Researchers introduce AdvancedMathBench, a benchmark suite designed to evaluate the advanced mathematical reasoning capabilities of large language models, addressing gaps in disciplinary scope and evaluation granularity found in existing benchmarks.

arxiv arXiv cs.CL · 15d ago · 3 views

DataGovBench benchmarks LLMs against real-world data complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings by focusing on fact retrieval from small tables while overlooking challenges like large multi-tabular datasets and exploratory insight discovery. To address this, the authors introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.

arxiv arXiv cs.CL · 15d ago

MORE: multilingual document parsing benchmark covering 149 languages

Researchers introduce MORE, a large-scale benchmark designed to evaluate the performance of Vision-Language Models on multilingual document parsing. The dataset addresses the lack of ground truth for low-resource languages by curating samples from real-world documents through a model-assisted, human-refined annotation pipeline.

arxiv arXiv cs.CL · 16d ago

MTEB-PT introduces native Brazilian Portuguese text embedding benchmark

The authors introduce MTEB-PT, a dedicated benchmark for evaluating text embeddings in Brazilian Portuguese that excludes translated data. The benchmark comprises 22 native tasks across seven categories and evaluates 93 models ranging from 23M to 27B parameters.