DemoPSD mitigates privileged information leakage in policy self-distillation

Researchers introduce DemoPSD, a framework that addresses privileged information leakage and overfitting in on-policy self-distillation (OPSD) for large language models. The method uses selective adoption of teacher guidance by steering the student toward a reverse-KL barycenter target, which balances learning from the teacher with preserving the student's reasoning capacity.

• DemoPSD measures distribution differences to adaptively control blending at each token position.
• The approach provably achieves leakage attenuation and exploration preservation under dense token-level distillation.
• Experiments on SciKnowEval across four scientific fields show DemoPSD outperforms GRPO and SDPO.
• The method maintains higher training entropy and robustly generalizes to out-of-distribution GPQA benchmarks.

DemoPSD resolves fundamental issues in OPSD by preventing the student from encoding answer-dependent shortcuts unavailable at test time, thereby improving cross-domain generalization.