DemoPSD mitigates privileged information leakage in policy self-distillation

Researchers introduce DemoPSD, a framework for on-policy self-distillation that resolves privileged information leakage and overfitting by selectively adopting teacher guidance. The method steers the student toward a reverse-KL barycenter target, balancing learning from the teacher with preserving the student's reasoning capacity.

• DemoPSD uses distribution discrepancy to adaptively control blending at each token position.
• It provably achieves leakage attenuation and exploration preservation under dense token-level distillation.
• Experiments on SciKnowEval show it outperforms GRPO and SDPO across four scientific fields.
• The approach maintains higher training entropy and robustly generalizes to out-of-distribution GPQA benchmarks.

DemoPSD effectively mitigates the suppression of exploration and cross-domain generalization issues inherent in standard teacher-student self-distillation.