media Hugging Face Forums · 2d ago · open_models

Study identifies FragileTokens that fail contextual copying despite passing isolation probes

from English

A study characterizes "FragileTokens," vocabulary entries in open-weight language models that successfully copy when isolated but exhibit errors when embedded in surrounding text. The research highlights that literal identity preservation is not guaranteed by standard isolation tests, as tokens can be deleted, substituted, or truncated within sequences.

Tested eight open-weight models from the Qwen and OLMo families using fixed token IDs inserted into random common-word sequences of varying lengths.
Found that a probability-based screen flags 0.4% to 10.9% of isolation-passing tokens as susceptible in at least 10% of tested contexts.
Demonstrated specific failure modes including deletion, substitution, truncation, and translation, with some errors occurring even under concentrated output distributions.
Evaluated geometric signals, finding that input- and output-embedding proximity achieved an AUROC of 0.744 to 0.881 on seven models for identifying these tokens.
Showed that selected fragile candidates are preserved less often than stable controls in instruction and simulated tool-use tasks.

The findings motivate the use of contextual verification protocols, arguing that passing an isolation probe should not be treated as a general certificate of literal reliability.

Importance 1/3 Trust 1/3 Hugging Face Forums Allen AI Alibaba (Qwen) Evaluation & benchmarks Research paper

Related articles

arxiv arXiv cs.LG · 64d ago · 2 views

Test-time scaling improves small VLMs on multilingual visual MCQ

The study examines whether test-time scaling (TTS) transfers to small open vision-language models using the EXAMS-V benchmark, comparing methods like self-consistency and PRM-guided beam search on Qwen2.5-VL-7B-Instruct and Qwen3.5-4B.

arxiv arXiv cs.AI · 67d ago · 1 view

AMALIA-9B fails to validly measure moral authority despite high human agreement

A study evaluates Portugal's AMALIA, a 9B-parameter European Portuguese model, finding it cannot validly measure the moral foundation of authority despite high agreement with human coders. While AMALIA matches open models eight to thirteen times its size in raw agreement scores, it fails to recover performance when prompts are decomposed into atomic clauses, suggesting reliance on surface correlates like moral outrage rather than theoretical constructs.

arxiv arXiv cs.CL · 67d ago · 1 view

Qwen3-Next models struggle to lower cognitive demand in Bloom-aligned educational control

Researchers introduce a Bloom-aligned framework to measure the ability of Large Language Models to preserve instructional intent while shifting cognitive demand toward specific learning objectives. The study evaluates Qwen3-Next-80B-A3B-Instruct and Qwen3-Coder-Next across 2,520 programming tasks from three benchmarks.

arxiv arXiv cs.CL · 73d ago · 1 view

LLMs show distinct policies under skeptical pressure rather than sycophantic retreat

A study testing Llama-3.1-8B, Qwen2.5-7B, and Mistral-7B across climate, vaccine, and evolution domains finds that models do not sycophantically retreat from scientific consensus when users signal doubt. Instead, the models exhibit three distinct policies: reactive assertion where consensus increases (Llama), surface hedging with softened tone (Qwen), and non-response (Mistral).

arxiv arXiv cs.CL · 74d ago · 7 views

Will Scaling Improve Social Simulation with LLMs?

This study investigates whether current language model scaling paradigms can close fidelity gaps in social simulations across opinion modeling, behavioral simulation, and longitudinal forecasting. Using 85 Qwen3 transformer models trained on the DCLM corpus under fixed-compute budgets from $10^{18}$ to $10^{20}$ FLOPs, the authors analyze the relationship between compute scale and simulation accuracy.