Study finds on-policy distillation success depends on signal quality, not teacher scale

A systematic study of on-policy distillation (OPD) identifies it as an exploration catalyst that steers students via dense token-level guidance without expanding the capability ceiling. The research confirms that OPD effectiveness relies entirely on guiding signal quality rather than prompt diversity or per-problem sampling numbers.

• Student-Teacher Mismatch: A large distributional gap causes misalignment with task correctness, steering exploration counterproductively.
• Length Exploitation: Aggregated objectives create shortcuts allowing students to game rewards via truncation or padding.
• Signal Regulations: Advantage clipping and log-scale compression tame these pathologies by ensuring faithful guidance.

Experiments across seven benchmarks show that these regulations alleviate length exploitation and stably surpass OPD variants and RLVR baselines, confirming that well-regulated signal quality governs successful exploration.