PACT combines a reactive RL policy with a 2B-parameter Small Language Model to generate and validate action plans. The SLM plan is executed directly if verified as safe, feasible, and complete, bypassing the RL policy. PACT outperforms baselines on three increasingly difficult FrozenLake environments.
arxiv
arXiv cs.AI
·
54d ago
·
research
PACT: Small Language Model Deliberation for Reactive Reinforcement Learning
from English
Importance 2/3
New feature vs. leaders
New harness with differentiators
arXiv cs.AI
OpenAI
Google DeepMind
Meta AI
AI agents
Reasoning models
Training methods
Related articles
arxiv
arXiv cs.LG
·
51d ago
·
1 view
LLM-based Hierarchical Control in Multi-Agent Games
A hierarchical system using a pretrained LLM to select RL skill policies outperforms flat RL in a 2v2 King of the Hill environment. It matches hand-crafted behavior tree performance in win rate and is perceived as more human-like by 60% of users, highlighting effective coordination and adaptability without manual rule design.
arxiv
arXiv cs.CL
·
52d ago
GraphPO: Graph-based Policy Optimization for Reasoning Models
GraphPO introduces a directed acyclic graph framework to represent reasoning rollouts, merging semantically equivalent paths to reduce redundant exploration. It assigns efficiency and correctness advantages to edges, improving inference efficiency and process supervision while reducing advantage-estimation variance. Experiments show GraphPO outperforms chain- and tree-based methods on three LLMs across reasoning and agentic search tasks under identical token or response budgets.
arxiv
arXiv cs.AI
·
52d ago
RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
RODS addresses sample depletion in multi-turn tool-use RL by using reward variance to detect capability boundaries. It synthesizes new data in real time, matching structural complexity of boundary samples, and maintains a dynamic replay buffer that co-evolves with the policy. RODS achieves performance comparable to a 17K-sample offline pipeline with 20x fewer trajectories.
arxiv
arXiv cs.LG
·
53d ago
Compositional Generalization in Language Model Reasoning
A hierarchical latent selection model shows that supervised fine-tuning and reinforcement learning work together to enable compositional generalization in language models. SFT provides raw module materials, while RL identifies and recombines atomic modules from compound traces to solve new problems. Training on compound traces leads to stronger generalization than isolated module training, and an effective protocol is found where SFT ensures module coverage and RL drives exploration of novel compositions.
arxiv
arXiv cs.CL
·
53d ago
·
2 views
d-OPSD: On-policy Self-distillation for Diffusion LLMs
d-OPSD is the first on-policy self-distillation framework designed for diffusion LLMs. It uses self-generated answers as suffix conditioning and step-level supervision, enabling efficient post-training with only about 10% of RLVR's optimization steps while outperforming RLVR and SFT baselines on four reasoning benchmarks.