media Hugging Face Forums · 1h ago · open_models

Claude Fable 5 and ChatGPT 5.5 evaluated on 'show, don't tell' detection

from English

A follow-up study tested Claude Fable 5 and ChatGPT 5.5 against a human annotator on the inferential feature of materialized metaphor using 100 scenes. The analysis revealed significant discrepancies in detection thresholds among LLMs, with Claude identifying 78 instances and ChatGPT 40, compared to near-zero counts for other models.

On materialized metaphors, human annotators found 9/100 instances, while Grok found 0, Gemini ~2, ChatGPT 40, and Claude 78.
Agreement scores (κ) were approximately zero for the hardest features, suggesting the definition itself contributes to the variance.
The study corrects earlier claims that atmosphere contradiction was machine-proof, showing measurable agreement for Claude (κ 0.27) and ChatGPT (κ 0.18).

The dataset and deterministic scoring script are available on Hugging Face for recomputation without model involvement.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks

Read original

media Hugging Face Forums · 7h ago

SRRM benchmark reveals Qwen2.5-1.5B outperforms 3B in long-context memory

A researcher is seeking an arXiv cs.CL endorsement for a paper introducing Simple Retrieval-Reconstruction Memory (SRRM), a lightweight benchmark designed to evaluate long-context memory in Small Language Models.

media Hugging Face Forums · 13h ago

Jeanbosange releases LIMEN Runtime Audit prototype for inspecting layer-wise activation trajectories

Jeanbosange has published the first public prototype of LIMEN Runtime Audit, an open-source toolkit designed to inspect layer-wise activation trajectories in open-weight language models. The tool treats the sequence of hidden states across layers as a measurable trajectory, providing a reproducible observability layer for comparing internal runtime behavior.

media Hugging Face Forums · 3d ago

John6666 releases first independent reference implementation for REO v1.5

John "John6666" has developed the first independent reference implementation of the Reference Evaluation Object (REO v1.5) protocol, fulfilling a key goal of the EPE research program to enable reproducible benchmarks by external contributors.

arxiv τ²-bench (tau2-bench) · 3d ago

τ-bench 1.0.1 fixes banking_knowledge grading to prevent penalizing cautious agent behavior

Sierra Research released τ-bench 1.0.1, which corrects the `banking_knowledge` task grading scheme to stop penalizing agents for prudent verification reads and fixes several data inconsistencies. The update ensures that re-grading leaderboard trajectories only increases scores, with no previously passing simulations failing.

media Hugging Face Forums · 3d ago

Author proposes open k registry for evidence saturation in local LLMs

The author has published a pilot paper and an open registry on Hugging Face to measure the evidence-saturation point (k*) of language models. This metric determines the optimal number of evidence fragments to inject into prompts, as adding more context does not monotonically improve correctness and can increase cost or epistemic contamination.