A study investigates the fixed-budget decision problem of allocating Reinforcement Learning (RL) post-training resources, specifically examining whether to prioritize larger policies, extended training, more rollout search, or stronger reward feedback.
- The authors introduce a FLOP-accounting framework for GRPO post-training that decomposes compute into rollout/search, policy-update/learning, and reward-model evaluation.
- Analysis of LoRA-adapted Qwen2.5 policies reveals conditional allocation frontiers where the optimal strategy depends on model size, budget, and reward systems.
- Larger policies consume more per-token compute, resulting in fewer updates or rollouts under the same total budget compared to smaller models.
- Reward system architecture significantly impacts accounting; rule-based rewards spend most non-update compute on rollouts, while PRM-style feedback allocates visible budget to inference.
- The paper proposes RACE as a diagnostic pilot-grid protocol to identify allocation regimes before expensive validation runs.
The authors argue that RL post-training papers should report total FLOPs alongside the specific division of compute among model size, search, learning, and feedback.