RL post-training builds compositional reasoning strategies from primitive skills

Research investigates whether reinforcement learning (RL) post-training merely amplifies latent primitive skills or composes them into new higher-level strategies. Using a Transformer pretrained on symbol-rewrite chains and post-trained on a Trace-based reasoning task, the study finds that RL solves held-out problems more effectively than rejection fine-tuning.

• RL reorganizes primitive competence through a phased compositional mechanism, first strengthening primitive reductions then discovering valid composed procedures.
• Composed procedures include sequential compositions that collapse ordered chains and parallel compositions that combine independent contractions in a single step.
• These strategies are reused and consolidated into a stable repertoire rather than remaining isolated samples.
• RL concentrates exploration into valid reusable structure, whereas rejection fine-tuning produces many invalid shortcut-like rewrites.

The emergence of compositional strategies is gated by whether pretraining organizes primitive competence into reduction procedures that RL can compress. The base model provides weak procedural ingredients, which RL builds into reliable higher-level strategies.