The authors propose Selective Importance Sampling (SIS), a plug-in method that addresses the variance explosion caused by token-level importance ratios in off-policy reinforcement learning post-training. SIS treats the off-policy model as a proposal distribution and applies a token-level rejection test to convert accepted tokens into on-policy data, thereby eliminating the need for importance correction scores for those tokens.
- SIS reduces the gap between token-level and sequence-level off-policy gradient estimators through theoretical proof.
- The method modifies only the importance ratio in the policy loss, adding negligible wall-clock overhead.
- It is compatible with a wide variety of RL post-training algorithms.
- Experiments on dense and MoE LLMs across math and agent benchmarks show consistent improvements and stronger robustness under off-policy data.
The approach provides substantially stronger robustness under off-policy data while consistently improving objectives without significant computational cost.