Researchers propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment method that reuses pre-collected teacher trajectories as replayed prefixes to address the high cost of fully online on-policy distillation. This approach allows the student to act at selected steps while the teacher provides dense per-step supervision without executing new environment interactions.
- ReOPD addresses the "prefix trap" in multi-turn OPD, where improving student relevance can query the teacher on unreliable histories.
- It implements a step-decaying sampling schedule that emphasizes early, lower-shift prefixes to manage distribution shift.
- Across mathematical reasoning and search environments, ReOPD preserves or improves accuracy while using zero tool calls during training.
- The method is at least 4x faster per training step than standard OPD by turning agent-environment interaction into a reusable offline resource.