The Agon framework enables reinforcement learning from verifiable rewards by making two competing models grade each other's reasoning traces, rather than grading only final answers. In this setup, one model drafts a solution while the other solves it in alternating roles, rewarding the agent that out-reasons its rival without requiring process labels or a separate reward model.
- Both models are optimized simultaneously, facing a progressively stronger rival that single-model RL cannot provide.
- At inference, the pair deploys as a two-stage cascade where one model drafts and the other answers after reading the draft.
- On the hard split of DeepMath with Qwen3, this approach doubles GRPO's pass@1 performance.
- The ordering replicates on competitive-programming code and across model families including Qwen3.5 and Gemma 4.
This method allows reasoning to be judged implicitly during training, addressing the issue where hard problems currently train models to write more rather than think better.