arxiv arXiv cs.AI · 1d ago · research

API benchmark scores overestimate chatbot interface performance

from English

An audit of ChatGPT, Claude, and Gemini reveals that API evaluation metrics do not reliably transfer to deployed chatbot interfaces. The study identifies a "context-validity gap" where API-based measurements systematically differ from real-world usage.

API evaluations score 3.4 percentage points higher in accuracy than interface evaluations.
Test-retest agreement is 2.1 percentage points higher for API access compared to interface access.
For ChatGPT, the performance drop from switching to an interface exceeds the difference between GPT 5.3 and GPT 5.4.
Adjusting system prompts, sampling parameters, and reasoning settings via API controls does not reliably eliminate this performance gap.

These findings complicate the use of API evaluations as proxies for deployed systems, indicating that benchmark scores may overstate actual user-facing capabilities.

Importance 2/3 arXiv cs.AI OpenAI Anthropic Google DeepMind Evaluation & benchmarks

Read original

arxiv arXiv cs.CL · 67d ago · 9 views

GPT-5 outperforms humans in inducing belief states via planning

A new study evaluates Large Language Models' ability to induce specific belief states in other agents through actions rather than conversation, a capability termed Non-Conversational Planning ToM (NCP-ToM). Using the NCP-ExploreToM framework, researchers tested six frontier models and human participants on 600 task instances where agents had to move objects or direct characters to achieve belief goals.

arxiv arXiv cs.CL · 67d ago · 10 views

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 69d ago · 5 views

Automated grading of Linux/bash examinations using large language models

This study evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The research demonstrates that structured prompts significantly improve agreement with human graders, establishing a framework for AI-assisted assessment in computing education.

arxiv arXiv cs.AI · 73d ago · 7 views

When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs

A study reveals that large language models systematically suppress 'Causal Caution'—the tendency to refrain from causal judgment without sufficient evidence—when shifting from academic to practical advisory contexts. This suppression occurs despite the models retaining the underlying capability, as evidenced by the ability to restore cautious reasoning through specific prompts.

media r/LocalLLaMA · 79d ago · 15 views

EU AI Act mandates AI-generated text watermarking from August 2024

The EU AI Act requires all AI systems generating synthetic text to include machine-readable, detectable watermarks using robust, interoperable technical solutions with two layers. This applies to all AI models, including open-source ones, and extends to any service accessible by EU citizens, regardless of location. Non-compliance risks fines of up to 35 million euros or a percentage of annual income, with providers of 'systemic risk' AI models facing heightened liability.