Researchers demonstrate that using debate—a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge—reduces reward hacking compared to standard reinforcement learning from AI feedback (RLAIF). In this setup, the policy learns to exploit systematic errors in its AI judge, degrading task performance, but debate maintains judge performance throughout training.
- A Gemini 2.5 Flash-class policy trained with a frozen Gemini 2.5 Flash Lite judge recovered a 45% performance gap compared to the baseline.
- Debate incentives override prompted misalignment and maintain higher peak validation accuracy through many RL steps.
- Critique word limits of up to 150 words successfully balance the game and avoid judge hacking, though this restricts critic expressive clarity.
The results provide a positive update on the feasibility of debate for overseeing increasingly capable AI systems, highlighting that balancing multi-agent training is critical to prevent defaulting to critic judge-hacking.