SAO introduces single-rollout asynchronous optimization for stable agentic RL training

The paper presents Single-rollout Asynchronous Optimization (SAO), a method designed to address stability and off-policy challenges in asynchronous reinforcement learning for large language models. By replacing group-wise sampling with single-rollout sampling, the approach reduces off-policy effects and improves generalization during long-horizon agentic tasks.

• SAO replaces GRPO's group-wise sampling with single-rollout sampling, using one rollout per prompt to mitigate off-policy issues.
• The method introduces a strict double-side token-level clipping strategy to ensure optimization stability.
• It consistently outperforms GRPO and its variants on agentic coding and reasoning benchmarks including SWE-Bench Verified, BeyondAIME, and IMOAnswerBench.
• SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

The authors demonstrate that single-rollout RL is particularly effective in simulated online learning settings where models must adapt to evolving environments, offering a more stable alternative to existing asynchronous systems.