LMRE addresses limitations of single-reference evaluation by modeling multiple valid phrasings of speech. It outperforms traditional methods in aligning with human judgment on acceptance and scoring, demonstrating scalability and robustness for Korean speech annotations.
Answer Engineering introduces a runtime layer that applies localized rule-based corrections to a model's reasoning trajectory during generation, without retraining. In a clinical benchmark for sudden sensorineural hearing loss, it increased protocol-compliant outcomes from 54.5% to 83.5% and conductive-case adherence from 1.6% to 58.9%.
Dutch language models exhibit coherence illusions similar to human readers. Surprisal and attention entropy metrics show that models are misled by context matches, with energy from associative memory identifying discourse coherence mechanisms.
A multi-agent audit framework improves clinical mental health screening by decomposing reasoning into perception, retrieval, inference, and audit stages. Evaluated on the DAIC-WOZ dataset, it reduces PHQ-8 depression severity prediction error from 5.35 to 5.02 and offers interpretable, verifiable diagnostic rationales.
A new study reveals over 1,000 legal filings contain fabricated citations, with the number rising annually. Benchmarking five AI models shows improved performance, with GPT-5 achieving 82.8% recall and 60.5% F1 in agentic settings, though all models struggle with subtle errors and face resource constraints due to limited information access.
Dementia-Agents introduces a clinically aligned multi-agent framework for real-world dementia staging and phenotyping. It improves diagnostic performance over monolithic models and prior systems, while maintaining domain-level interpretability, using data from 1,066 patients across two cognitive neurology services.
The paper argues that reference in large language models is not a fixed link but a profile-based, context-sensitive, and numerically structured phenomenon. It proposes that LLMs ground reference through linguistic traces parameterized via optimization, with referential profiles distributed and activated via context-sensitive computations in vector spaces.
A mechanistic analysis shows retrieval heads are causally necessary for long-context recall. Higher RoPE frequencies do not reduce head counts, and zeroing low-frequency RoPE dimensions in retrieval heads degrades recall dose-dependently, with effects observed across five models and multiple architectures.
SCOPE introduces a framework that uses a readable hidden layer and conformal calibration to detect out-of-distribution inputs. It employs a supermartingale e-process to provide theoretical guarantees for service-boundary detection, outperforming standard final-layer detectors in multiple LLM backbones.
ARCO introduces a rubric framework that enables step-level credit assignment for multi-step LLM agents. It jointly updates a shared model with generation and scoring heads, allowing the rubric content and scoring function to co-evolve via on-policy data, improving performance and interpretability across benchmarks.
Large language models use non-contiguous, redundant paths to retrieve factual attributes. These paths often skip layers and involve multiple equivalent routes, indicating distributed and redundant knowledge computation, challenging current understanding of LLM knowledge storage and retrieval.
SciFactCheck evaluates 18 LLMs across five scientific domains, finding that scientifically fine-tuned models show degraded factual reliability and reduced internal confidence despite greater linguistic assertiveness. Human studies reveal limited agreement between fact-checking tools and expert judgments, highlighting challenges in defining valid scientific claims.
A reinforcement learning framework enables precise control over clinical precision and recall in radiology report generation. By integrating a clinical reward and group-relative training, the model improves clinical efficacy beyond language fluency metrics, outperforming state-of-the-art methods on the MIMIC-CXR dataset.
A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.
A two-stage alignment pipeline enhances large language models' pedagogical performance in math mistake remediation. The approach combines supervised fine-tuning with Direct Preference Optimization using synthetic data on scaffolding and factuality, outperforming base and existing tutoring models in both accuracy and teaching quality. Human evaluations show the model competes with a proprietary baseline, offering greater openness and reproducibility.
MedHal-Loc introduces a benchmark to evaluate whether medical hallucination detectors accurately localize errors. It finds that while some architectures localize well above chance, a knowledge-graph pipeline performs no better than random due to poor entity extraction, despite strong detection performance. The results show that detection capability does not guarantee faithful localization, challenging assumptions about architectural explainability.
A controlled ablation study evaluates agentic RAG components using a local 7B model on HotpotQA. Fixed hybrid retrieval outperforms adaptive routing by 1.8 EM and 1.9 F1, while two retrieval iterations capture 95% of the gains from five. Query decomposition and cross-encoder reranking show statistically significant but smaller improvements.
The paper proposes a dynamic rubric framework that adapts MQM evaluation spaces to individual translation instances. By selecting subtype spaces and granularities based on case-specific needs, it improves error coverage and localization, outperforming static rubric methods on WMT span-level benchmarks.
Researchers identify 'role confusion' as a key vulnerability in LLMs, where models misinterpret user input due to stylistic similarities with internal role tags. Destyling user prompts reduces attack success from 61% to 10%, showing that subtle text style changes can dramatically alter model behavior, even when the content appears identical to humans.
Sakana AI has launched Sakana Fugu, an orchestration model that routes tasks across a swappable pool of frontier LLMs via a single OpenAI-compatible API. Fugu Ultra outperforms individual models on key benchmarks like SWE Bench Pro and GPQA-D, and the system demonstrates superior performance on complex, multi-step tasks such as auto-research, Rubik's Cube solving, and blindfold chess.