RL post-training builds compositional reasoning strategies from primitive skills

Research investigates whether reinforcement learning (RL) post-training merely amplifies latent primitive skills or composes them into new higher-level strategies. Using a fully observable rewrite-grammar environment, the study finds that RL reorganizes primitive competence through a phased compositional mechanism.

• RL solves held-out problems rarely solved by the pretrained model, even with larger sampling budgets, while rejection fine-tuning plateaus early.
• RL strengthens primitive reductions before discovering valid composed procedures, including sequential and parallel compositions.
• These composed procedures are reused and consolidated into a stable repertoire rather than remaining isolated samples.
• The key difference between RL and rejection fine-tuning is selectivity; RL concentrates exploration into valid reusable structure.

The emergence of compositional strategies depends on pretraining organizing primitive competence into reduction procedures that RL can compress, allowing it to build reliable higher-level strategies from weak procedural ingredients.