Researchers propose MARGO (Mixed-Mode Advantage Regularization for Grounded Optimization), a reinforcement learning framework designed to mitigate "thinking-induced hallucination" in large reasoning models. This failure mode occurs when explicit thinking traces overturn correct non-thinking answers, leading to factual drift.
- MARGO uses non-thinking rollouts as same-model references during advantage estimation to evaluate whether explicit thinking adds factual value beyond direct answering.
- The framework constructs mixed-mode rollout groups containing both thinking and non-thinking trajectories to suppress hallucination-prone thinking while preserving beneficial behaviors.
- Experiments on factuality-oriented QA benchmarks demonstrate improved factual reliability over strong baselines, while evaluations on mathematical benchmarks show preserved general reasoning ability.