Three-stage analysis shows SFT, RL, and OPD shape CoT confidence differently

A study analyzes how supervised fine-tuning (SFT), reinforcement learning (RL), and on-policy distillation (OPD) reshape model confidence during chain-of-thought reasoning. The authors introduce a three-stage calibration framework evaluating confidence before, during, and after reasoning to assess difficulty estimation, early termination, and answer aggregation.

• OPD provides the most useful pre-reasoning confidence for difficulty estimation.
• SFT offers the strongest online signal for early stopping.
• RL produces the most reliable trace-level signal for answer aggregation.
• Confidence reliability is position-dependent; RL becomes informative after a path-commitment phase, while OPD can become inversely calibrated later.

The authors propose PosConf, a position-aware strategy that improves RL answer aggregation by 6.1 points over majority voting and boosts OPD early stopping gains up to 4.3 points under tight token budgets.