TurnOPD improves long-horizon agent training via turn-aware on-policy distillation

The authors propose TurnOPD, a turn-level budgeting strategy designed to address inefficiencies in vanilla on-policy distillation (OPD) for long-horizon agentic tasks. The method introduces adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting to optimize training resources.

• Full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision.
• Trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained.
• TurnOPD uses probe-based turn statistics to determine rollout length and gradually shifts KL weighting from token-level to turn-balanced supervision.

Experiments on ALFWorld, WebShop, and Multi-Hop Search show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets compared to vanilla OPD.