TACO tail-aware credit calibration mitigates positive-credit contamination in LLM reinforcement learning

Researchers propose Tail-Aware Credit calibratiOn (TACO), a method that addresses the "Positive-Credit Contamination" failure mode in critic-free reinforcement learning for large language models. This issue arises when uniform credit assignment indiscriminately reinforces erroneous low-probability tokens alongside plausible ones.

• TACO computes a tail-risk score based on local generation context to distinguish unexpected rarity from uncertainty-driven exploration.
• The method tunes positive credit for risky tokens, progressively dampening incidental noise while allowing recurring useful rare patterns to accumulate reinforcement.
• Experiments across three LLMs and eight benchmarks show TACO consistently outperforms GRPO-style baselines.
• The approach improves training stability, supporting sustained performance gains in long-horizon RL.

The authors consider this important because it mitigates the reinforcement of flawed reasoning behavior, enabling more stable and effective learning for complex reasoning tasks.