Direct-OPD distills weak-to-strong RL gains for stronger models

Researchers propose Direct On-Policy Distillation (Direct-OPD) to transfer reinforcement learning with verifiable rewards from a smaller, cheaper teacher model to a larger target model. This method treats the log-ratio between the teacher's post-RL and pre-RL policies as a dense implicit reward for the student, avoiding the need for explicit reward models or sparse-reward RL on the target.

• Direct-OPD compares the post-RL teacher with its own pre-RL reference to create a dense implicit reward signal.
• The approach applies this signal on the stronger student's own on-policy states without training an explicit reward model.
• It enables the sequential composition of multiple policy shifts and outperforms step-matched direct RL.

This technique allows RL outcomes to be reused across model scales as implicit reward signals, significantly improving reasoning capabilities in larger models.