arxiv arXiv cs.CL · just now · research

Corpus-specific VITA RAG matches or beats frontier LLMs on HealthBench

from English

VITA, a retrieval-augmented generation system designed for low- and middle-income settings, ranks first on the English-language subset of HealthBench, outperforming GPT-5.4, o4-mini, Gemini 3.1 Pro, and Claude Sonnet 4.6.

VITA scored 51.9% of possible rubric points on 4,023 questions, ahead of GPT-5.4 (46.1%) and Claude Sonnet 4.6 (37.3%).
When re-evaluated against newer models like GPT-5.5 using a neutral judge, VITA achieved parity with GPT-5.5 on mean per-question score while leading on points-weighted score.
The system retrieves from a curated corpus of disease-specific guidelines and India-specific antimicrobial resistance data.

The results indicate that purpose-built clinical RAG systems remain competitive with frontier LLMs, demonstrating that corpus specificity improves grounding despite lower communication polish.

Importance 2/3 Beats a top-lab benchmark arXiv cs.CL OpenAI Anthropic Google DeepMind Benchmark results Retrieval & RAG

Benchmarks

Benchmark	Model	Score
HealthBench	VITA	51.9%
HealthBench	GPT-5.4	46.1%
HealthBench	o4-mini	44.3%
HealthBench	Gemini 3.1 Pro	42.6%
HealthBench	Claude Sonnet 4.6	37.3%

Read original

arxiv arXiv cs.CL · 39d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 55d ago · 1 view

REDACT: Multilingual PII Benchmark with Systematic Control

REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.

arxiv arXiv cs.CL · 57d ago · 1 view

ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

ProvenanceGuard introduces a source-aware verifier for MCP-based LLM agents that detects cross-source conflation by routing claims to specific evidence sources and comparing stated attribution with actual source ownership. It achieves block F1 of 0.802 and source accuracy of 0.858 on 260 source-eligible claims, outperforming source-blind baselines, and detects all injected attribution swaps in 50 clinical probes.

arxiv arXiv cs.AI · 57d ago

ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

media IntuitionLabs · 119d ago · 45 views

GPT-5.2 achieves ~53% on ARC-AGI-2 benchmark in December 2025

OpenAI's GPT-5.2, released in December 2025, marked a significant milestone in AI abstract reasoning by achieving approximately a 53–54% success rate on the ARC-AGI-2 benchmark. This performance far exceeded the prior state of the art, where no closed-model system had previously cleared even 5% success on these tasks.