A practical investigation examines training-free relaxed speculative decoding, which accelerates autoregressive LLM sampling by using a faster auxiliary model to draft tokens that are then verified in parallel. The study unifies existing approaches within a shared framework and benchmarks them on contemporary settings to distill empirical findings for practitioners.
- Relaxation of the strict lossless guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains.
- Evaluating capability is required for relaxed methods, unlike standard lossless speculative decoding.
- Many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.