Researchers propose SPyCE (Skill-Policy Co-evolution), a framework that distills multimodal reasoning trajectories into a hierarchical skill library that evolves alongside the policy during reinforcement learning. This approach addresses limitations in existing methods by avoiding scalar rewards and static memory retrieval, instead creating a closed loop where improved policies yield better skills and evolving skills provide stronger priors.
- Execution skills capture local visual operations while workflow skills encode high-level tool-use priors.
- The policy conditions on retrieved skills to guide rollouts, while the skill library updates using valuable rollouts generated by the policy.
- Experiments across eight benchmarks show SPyCE consistently outperforms RL-based and memory-based baselines.
- Analysis confirms that both hierarchical skill design and the co-evolution mechanism are critical to the framework's success.
The authors consider this joint skill-policy optimization a promising paradigm for building capable multimodal agents.