arxiv arXiv cs.CL · 53d ago · research

Geographic Bias in Large Language Models from User Metadata

from English

A study reveals that even neutral prompts trigger region-specific responses in large language models due to user metadata. Location leakage increases by up to 793 times in some models, and using 'Unknown' instead of location metadata still causes significant bias, indicating the user profile frame itself acts as a conditioning signal.

Importance 3/3 arXiv cs.CL Mistral AI Alibaba (Qwen) Anthropic Evaluation & benchmarks Reasoning models Safety & alignment

Read original

arxiv arXiv cs.CL · 51d ago · 1 view

Causal Activation Directions for Mitigating Emergent Misalignment in Language Models

Fine-tuning language models on insecure code causes emergent misalignment. A shared activation direction across four model families achieves 99.6% separation of aligned and misaligned activations, and subtracting it reduces code spillover by 21-51 points. Cross-architecture transfer shows behavioral suppression but lacks specificity, with within-model directions being causally actionable and cross-model directions only causally real.

media r/LocalLLaMA · 54d ago · 2 views

HalBench Tests 29 Open Source Models on Sycophancy and Hallucination

HalBench evaluates 29 open-source LLMs on a custom benchmark for sycophancy and hallucination. Qwen 3.6 and Gemma 4 outperform larger models, with Qwen 3.6 achieving 36.6% pushback—higher than GPT-5.4 and Gemini 3.1 Pro. Model size does not correlate with honest responses, indicating that architecture and training data matter more than parameters.

arxiv arXiv cs.CL · 12d ago · 2 views

Frontier LLMs use invisible reasoning via filler tokens to bypass CoT monitoring

A study demonstrates that frontier language models can perform consequential computation using semantically irrelevant filler tokens, creating a failure mode where reasoning is not visible in the output chain-of-thought. The research evaluated 13 models across three tasks and found that many benefit significantly from these tokens, with accuracy improvements of up to 13 percentage points.

arxiv arXiv cs.AI · 46d ago

LLMs Benchmarked for Web Vulnerability Detection

A study evaluates six LLMs on detecting real-world web vulnerabilities in WordPress plugins, finding detection rates vary by model and prompt design. Claude Opus 4.6 achieved the highest detection rate at 63%, while Qwen 3.5 only reached 35%, and no model consistently identified all baseline vulnerabilities across iterations.

arxiv arXiv cs.AI · 46d ago · 1 view

Benchmark Evaluation of Small Language Models for Arabic NLP

A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.