A study of sequential Direct Preference Optimization finds that later training does not uniformly degrade earlier learned preferences. The effect varies by objective relationship, signal strength, and training order, ranging from partial degradation to positive transfer. Pair-level analysis reveals heterogeneous changes, with high-confidence preference pairs sometimes improving despite aggregate metric stability.
arxiv
arXiv cs.CL
·
51d ago
·
src: 52d ago
·
research
Sequential DPO Shows Variable Preference Impact Across Settings
from English
Related articles
arxiv
arXiv cs.AI
·
51d ago
MACR: Explicit Conflict Resolution for LLM Inference
MACR introduces a multi-agent reasoning framework to resolve knowledge conflicts in LLM inference by jointly assessing internal and external knowledge. It uses semantic entropy to measure confidence and employs three specialized agents to induce rules, detect conflicts, and resolve inconsistencies across contexts. Empirical results show MACR outperforms state-of-the-art methods and provides interpretable conflict resolutions.
arxiv
arXiv cs.LG
·
51d ago
·
1 view
VIMPO: Critic-Free Policy Optimization for LLMs
VIMPO introduces a critic-free policy optimization method that derives a policy-implied value function from KL-regularized reinforcement learning. It enables verifiable reward incorporation without training a critic and outperforms GRPO on mathematical benchmarks, especially under noisy rewards.
arxiv
arXiv cs.LG
·
51d ago
·
1 view
LLM-based Hierarchical Control in Multi-Agent Games
A hierarchical system using a pretrained LLM to select RL skill policies outperforms flat RL in a 2v2 King of the Hill environment. It matches hand-crafted behavior tree performance in win rate and is perceived as more human-like by 60% of users, highlighting effective coordination and adaptability without manual rule design.
arxiv
arXiv cs.CL
·
51d ago
·
4 views
Bayesian Curriculum Learning on LLM Latent Manifolds
Manifold Bandits introduces Bayesian Manifold Curriculum (BMC), a framework that models problem sampling as a structured bandit problem in LLMs' latent space. BMC organizes tasks into a hierarchical tree and uses Bayesian learning to guide sampling, revealing tradeoffs between learning signal, task diversity, and evaluation relevance. Prioritizing difficulty alone fails to achieve strong downstream performance, underscoring the need for structure and type-aware sampling.
arxiv
arXiv cs.CL
·
51d ago
·
3 views
Control-Window Law for Single-Neuron Steering in Language Models
A new framework defines when single-neuron interventions coherently control model behaviors without output collapse. The control window, based on alignment and norm ratios, predicts behavior triggers and collapse ceilings using forward pass data, with high accuracy on held-out neurons. On refusal, control is typed: coherent bypass occurs without actionable content, while genuine actionable reach appears only in specific cases and at later rollout stages.