The authors propose ShortOPD, a short-to-long on-policy distillation schedule designed to recover the generative capabilities of structured-pruned large language models that collapse during free-form generation. By detecting teacher-confirmed repetitive suffixes and treating the surviving prefix as the effective rollout length, the method allocates training budget more efficiently than standard approaches.
- ShortOPD raises compressed model scores to approximately 9 times their unrecovered value and 1.6–4.4 times those of standard recovery recipes like SFT without KD, KD, and SeqKD across math, code, and open-ended generation tasks.
- The method matches a fixed 8192-token rollout horizon within two points while using only a quarter of the training time (8.5 vs. 35.9 hours) and 71% fewer rollout tokens.
This approach aims to move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, bringing compressed models closer to deployment-ready generation quality.