arxiv arXiv cs.CL · 2d ago · src: 4d ago · research

Rep2Act aligns VLM representations to improve abstention on new VAD-R benchmark

from English

Researchers introduce Visual Answerability Diagnosis with Rationales (VAD-R), a new benchmark designed to evaluate vision-language models' ability to abstain from unanswerable questions without shortcut cues. The study reveals that while hidden states can distinguish answerability, current models fail to translate this into explicit decisions.

VAD-R uses a two-stage pipeline of shortcut filtering and quality verification with step-by-step rationales.
State-of-the-art open-source VLMs show limited spontaneous abstention with average recall rates of only 11.4%.
The authors propose Rep2Act, a method to align latent answerability awareness with explicit abstention actions.
Rep2Act improves action accuracy on VAD-R from roughly 56-59% to over 86-88% for Qwen2.5-VL models.
On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3%, surpassing GPT-4 Turbo and GPT-4o.

This approach bridges the gap between representation and action, enabling VLMs to effectively recognize when they should not answer a question.

Importance 2/3 Beats a top-lab benchmark arXiv cs.CL Alibaba (Qwen) Evaluation & benchmarks Research paper

Related articles

media Hugging Face Forums · 17d ago · 29 views

Study identifies FragileTokens that fail contextual copying despite passing isolation probes

A study characterizes "FragileTokens," vocabulary entries in open-weight language models that successfully copy when isolated but exhibit errors when embedded in surrounding text. The research highlights that literal identity preservation is not guaranteed by standard isolation tests, as tokens can be deleted, substituted, or truncated within sequences.

arxiv arXiv cs.LG · 79d ago · 17 views

Test-time scaling improves small VLMs on multilingual visual MCQ

The study examines whether test-time scaling (TTS) transfers to small open vision-language models using the EXAMS-V benchmark, comparing methods like self-consistency and PRM-guided beam search on Qwen2.5-VL-7B-Instruct and Qwen3.5-4B.

arxiv arXiv cs.AI · 82d ago · 14 views

AMALIA-9B fails to validly measure moral authority despite high human agreement

A study evaluates Portugal's AMALIA, a 9B-parameter European Portuguese model, finding it cannot validly measure the moral foundation of authority despite high agreement with human coders. While AMALIA matches open models eight to thirteen times its size in raw agreement scores, it fails to recover performance when prompts are decomposed into atomic clauses, suggesting reliance on surface correlates like moral outrage rather than theoretical constructs.

arxiv arXiv cs.CL · 82d ago · 14 views

Qwen3-Next models struggle to lower cognitive demand in Bloom-aligned educational control

Researchers introduce a Bloom-aligned framework to measure the ability of Large Language Models to preserve instructional intent while shifting cognitive demand toward specific learning objectives. The study evaluates Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next across 2,520 programming tasks from three benchmarks.

arxiv arXiv cs.CL · 89d ago · 18 views

LLMs show distinct policies under skeptical pressure rather than sycophantic retreat

A study testing Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B across climate, vaccine, and evolution domains finds that models do not sycophantically retreat from scientific consensus when users signal doubt. Instead, the models exhibit three distinct policies: reactive assertion where consensus increases (Llama), surface hedging with softened tone (Qwen), and non-response (Mistral).