media Hugging Face Forums · 2h ago · open_models

Levent Bulut benchmarks LLM annotation reliability on objective projection

from English

Levent Bulut published a three-study benchmark evaluating the reliability of machine-generated annotations for a Turkish narrative corpus, revealing significant discrepancies between automated raters and human judgment. The study tested six binary craft features across 120 and 100 scenes using rule-based detectors and models including Gemini 2.5 Flash, Grok, ChatGPT 5.5, and Claude Fable 5 High.

Cohen’s kappa scores were near chance for five of the six labellers, ranging from 0.000 to 0.027.
For the rule requiring inference on abstract states, human raters identified 9 positives while ChatGPT 5.5 identified 40 and the detector identified 72.
ChatGPT 5.5 achieved the highest raw agreement at 84.5%, primarily because five of the six rules had lopsided distributions favoring negative labels.
The author notes that four of the six rules were effectively untestable due to evaluation set defects rather than model failures.

The results suggest either that the feature requires inference beyond current models or that the definition is not operational enough for consistent application. Bulut has open-sourced the dataset, prompts, and scripts to seek a second independent human rater.

Importance 1/3 Trust 1/3 Hugging Face Forums Benchmark results Evaluation & benchmarks

Related articles

arxiv τ²-bench (tau2-bench) · 10d ago

τ-bench 1.0.1 fixes banking_knowledge grading to prevent penalizing cautious agent behavior

Sierra Research released τ-bench 1.0.1, which corrects the `banking_knowledge` task grading scheme to stop penalizing agents for prudent verification reads and fixes several data inconsistencies. The update ensures that re-grading leaderboard trajectories only increases scores, with no previously passing simulations failing.

arxiv arXiv cs.CL · 17d ago

Variance decomposition reveals query language dominates LLM brand answer noise

A study decomposes the non-determinism in large language model (LLM) brand recommendations into four separable sources: within-prompt resampling, prompt paraphrase, model identity, and query language. Using a crossed random-effects analysis on 12,933 responses across three models and eight languages, the authors identify query language as the largest source of variance at 26.5%, while brand identity accounts for only 1.5%.

arxiv arXiv cs.CL · 19d ago · 2 views

AdvancedMathBench introduces ProverBench and VerifierBench for advanced mathematical proof evaluation

Researchers introduce AdvancedMathBench, a benchmark suite designed to evaluate the advanced mathematical reasoning capabilities of large language models, addressing gaps in disciplinary scope and evaluation granularity found in existing benchmarks.

arxiv arXiv cs.CL · 25d ago · 3 views

DataGovBench benchmarks LLMs against real-world data complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings by focusing on fact retrieval from small tables while overlooking challenges like large multi-tabular datasets and exploratory insight discovery. To address this, the authors introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.

arxiv arXiv cs.CL · 26d ago

MORE: multilingual document parsing benchmark covering 149 languages

Researchers introduce MORE, a large-scale benchmark designed to evaluate the performance of Vision-Language Models on multilingual document parsing. The dataset addresses the lack of ground truth for low-resource languages by curating samples from real-world documents through a model-assisted, human-refined annotation pipeline.