TRACE improves long-horizon agent tool-use via turn-level credit estimation

The authors propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning that addresses the sparsity and high variance of outcome rewards in long-horizon tasks. By representing rollouts as state transitions at tool-call boundaries and deriving per-action rewards from log-ratio state values, TRACE enables effective training without additional critics or supervised fine-tuning stages.

• On the BrowseComp-Plus benchmark, TRACE raises Qwen3-4B scores from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6.
• The method improves base-model tool-use ability using pure reinforcement learning, eliminating the need for cold-start supervised fine-tuning or live-web data training.
• Learned search behavior transfers to open-web benchmarks, with learning curves showing earlier improvement and faster convergence during RL training.

This approach allows long-horizon agents to learn effectively from dense feedback, significantly boosting performance on complex search tasks compared to traditional outcome-only reward signals.