This article presents a practical investigation into training-free relaxed speculative decoding techniques, which accelerate autoregressive LLM sampling by using an auxiliary model to draft tokens. The authors unify existing approaches within a shared framework and benchmark them on contemporary settings to distill empirical findings for practitioners.
- Relaxing the strict lossless guarantee of standard speculative decoding can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains.
- Evaluating the capabilities required by relaxation is more considerable than in lossless speculative decoding.
- Many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.
The findings highlight that while relaxation offers performance benefits, it introduces significant evaluation requirements and constraints on the choice of drafting models.