arxiv arXiv cs.CL · 51d ago · research

LLM Alignment Using Implicit User Feedback

from English

A new dataset, IFLLM, collects mouse trajectories and eye gazing data from users interacting with LLMs. It shows that implicit feedback significantly improves LLM alignment, boosting text-based reward model accuracy from 55% to 64% and nearly tripling response quality improvements after DPO training on eight LLMs.

Importance 3/3 New harness with differentiators arXiv cs.CL OpenAI Google DeepMind Meta AI Evaluation & benchmarks Research paper Safety & alignment

Read original

arxiv arXiv cs.LG · 51d ago

LLM Alignment Using Implicit User Feedback

arxiv arXiv cs.AI · 51d ago

MACR: Explicit Conflict Resolution for LLM Inference

MACR introduces a multi-agent reasoning framework to resolve knowledge conflicts in LLM inference by jointly assessing internal and external knowledge. It uses semantic entropy to measure confidence and employs three specialized agents to induce rules, detect conflicts, and resolve inconsistencies across contexts. Empirical results show MACR outperforms state-of-the-art methods and provides interpretable conflict resolutions.

arxiv arXiv cs.CL · 51d ago · 3 views

Control-Window Law for Single-Neuron Steering in Language Models

A new framework defines when single-neuron interventions coherently control model behaviors without output collapse. The control window, based on alignment and norm ratios, predicts behavior triggers and collapse ceilings using forward pass data, with high accuracy on held-out neurons. On refusal, control is typed: coherent bypass occurs without actionable content, while genuine actionable reach appears only in specific cases and at later rollout stages.

arxiv arXiv cs.CL · 52d ago · 1 view

Steerable Cultural Preference Optimization of Reward Models

This paper introduces SCPO, a novel reward model training algorithm that balances diverse cultural preferences across subcommunities. SCPO improves minority reward model performance by up to 7 points on two datasets and seven countries, while being up to 280% more training data-efficient than full-data fine-tuning. Analysis shows reduced bias through targeted subcommunity preference evaluation.

arxiv arXiv cs.CL · 52d ago · 2 views

Misfired Alignment in LLMs: A Quantitative Study

A new study introduces VETO, a benchmark of 2,032 BBQ-derived contrastive pairs, to quantify misfired alignment in large language models. It defines the Misfired Alignment Rate (MAR) and finds that all benchmarked LLMs exhibit MARs between 4.7% and 18.9%, while human participants achieve 0%. The research shows alignment cues can amplify these failures, with evidence suppression occurring in late layers of models and emerging after instruction training.