A user reports noticing less reliable outputs from vLLM compared to llama.cpp, including formatting errors, context forgetting, and lower code quality. They ask whether such differences stem from quantization, chat templates, parser issues, or configuration errors, and seek confirmation if others have observed similar quality discrepancies between inference backends.
SAFER is a training-free framework that enhances robustness of test-time adaptation by using reliability-guided augmentation. It generates stochastic augmentations, pools predictions via correlation-weighted aggregation with outlier detection, and includes adaptive mixing to preserve clean performance under adversarial attacks. Evaluations on PACS, VLCS, and OfficeHome show improved resilience without sacrificing clean accuracy.
A new method evaluates physical consistency in generated videos without requiring human voting or ground-truth references. It uses DROID-SLAM and SEA-RAFT to detect inconsistencies, improving task success rates by over 8% and enabling spatio-temporal localization of physical artifacts.
A large language model (LLM) assisted in identifying label-report discordance in the CT-RATE chest CT dataset. GPT-5.4 achieved 96.4% agreement with existing labels, with radiologist adjudication supporting LLM-derived labels in 74.2% of general and 91.9% of lymphadenopathy discordances. Multi-LLM majority-vote labels outperformed others in F1 score and kappa, and the cleaned dataset will be publicly released.
PlanBench-XL evaluates long-horizon planning in LLM agents across 1,665 tools through 327 retail tasks. It introduces a blocking mechanism to simulate real-world tool failures, revealing that agents like GPT-5.4 drop from 51.90% to 11.36% accuracy under severe disruptions, highlighting vulnerabilities in recovery and error handling.
Gold Points Sniper (GPS) enables lightweight vision-language models to perform self-guided multimodal reasoning for fine-grained human action understanding. By integrating a Gold Points Extractor, Selective Socratic Questioner, and Semantic Entailment Evaluator, GPS achieves performance comparable to GPT-4o while maintaining superior factual accuracy on CAP benchmark-based instruction-tuning data.
A structural codebase index in coding agents enhances localization and resolve performance without increasing cost per cell. It outperforms agentic-grep baselines in both metrics and achieves lower cost per solved task, especially in workloads with multi-file changes.
The FFASR Leaderboard was launched to evaluate speech recognition systems in real-world conditions. It provides a benchmark for assessing the performance of automatic speech recognition models across diverse environments and use cases.
MMGist is a curated multimodal benchmark with 7,262 items, designed to address flaws in existing vision-language benchmarks. It reduces evaluation size by 69% and improves cross-model discrimination by 78%, while preserving model rankings with a Spearman correlation of 0.98. The benchmark highlights visual logic as a key weakness and emphasizes the importance of visual dependency, discriminative power, and reliability in evaluation.
A benchmark using efficient multimodal large language models evaluates PE diagnosis and risk prediction on the INSPECT dataset. Results show Gemma4 E4B and E2B outperform others when EHR data is available, with PE diagnosis achieving higher accuracy than prognostic tasks like readmission prediction.
A fully differentiable emulator of the Atari 2600 VCS is presented, reproducing all 64 ALE games with bit-for-bit accuracy in RAM and screen output. The system enables gradient-based explainable AI by providing a complex, fully known ground truth, with both Julia and JAX implementations validated against a reference emulator and capable of high-throughput differentiable rollouts on GPU.
This study compares characters in LLM-generated and human-written stories using narratological dimensions. It finds that while LLMs produce characters with similar basic traits, they lack diversity in complex character features like stylization and wholeness. The research highlights key differences in character depth and variety between human and machine-generated narratives.
PRIME introduces a framework to analyze how large language models handle conflicting instructions by generating calibrated conflicts in response length, format, and reasoning. The study finds that conflict type has a greater impact on model behavior than model size, revealing diverse failure modes across conflict categories. Results highlight the need for conflict awareness and suggest instruction following cannot be reliably assessed through isolated benchmarks alone.
FACTOR introduces an inference-time model that adapts verification criteria based on claim-level uncertainty. It improves factuality and reduces verification cost by dynamically allocating effort to high-risk claims, demonstrating effective and model-agnostic performance on the FactScore benchmark.
A rental-search assistant with LLMs and multi-market support faced persistent user defects despite 1,553 passing automated tests. Analysis of 252 bug-fix commits showed 44% resolved issues in four unseen seams: live browser runtime, non-default market, end-to-end flows, and whole-system level. A simple practice was adopted to identify the seam with most fixes.
Hi-Seg, a human-in-the-loop framework built on SAM, achieves a mean Dice score of almost 85% in pulmonary nodule segmentation. It outperforms five state-of-the-art deep learning models and 13 SAM variants, with non-medical annotators matching junior medical student performance, reducing clinician workload and enabling scalable annotation.
The author benchmarks LLMs in simulating wetting behaviour using Surface Evolver, a 1992 tool for modeling liquid surfaces. LLMs are evaluated objectively by comparing their generated datafiles against reference implementations, with results showing pass counts and token costs for each model.
Talos is an open-source tool that automates iterative reanalysis of genomic data to identify rare disease diagnoses. It achieved a 90% recovery rate of in-scope diagnoses with only 1.3 candidate variants per patient, and delivered 241 new diagnoses across 5,000 undiagnosed patients, with most new findings emerging within 32 days of evidence publication.
The method combines a learnable world model with high- and low-level policies to enable safe exploration in long-horizon tasks. The high-level policy guides exploration toward safe subgoals, while the low-level policy uses imagined rollouts to prevent unsafe behaviors, outperforming existing Safe RL methods in success rate and constraint satisfaction across diverse tasks.
Context compaction in long-horizon LLM agents silently removes in-context safety constraints, leading to prohibited tool actions. Across 1,323 episodes, compaction increases policy violations from 0% to 30% and up to 59% for some models, with violations reaching 38% when constraints are dropped. Constraint Pinning, a training-free method, restores zero violations by isolating governance constraints from compaction.