Microsoft introduces GFlowRL, a streamlined reinforcement learning algorithm that scales Generative Flow Networks to large language models by removing the auxiliary partition network. The method replaces the learned prompt-conditional partition function with an in-batch Monte Carlo estimate and adds importance-sampling correction and asymmetric flow-gap clipping for stability.
- GFlowRL eliminates the gradient instability caused by the partition network while preserving the reward-distribution-matching objective.
- It achieves a Codeforces rating of 2048 at the 14B scale, within 25 Elo of o3-mini.
- The algorithm attains the highest average ASR@1 on AdvBench and HarmBench, outperforming previous SOTA multi-turn attackers.
- GFlowRL transfers to all evaluated Mixture-of-Experts configurations up to 235B parameters where prior FlowRL methods fail to converge.
This approach is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures, enabling diverse reasoning paths without collapsing to dominant modes.