The authors propose BlockPilot, a sample-adaptive policy that dynamically selects the optimal inference block size for diffusion-based speculative decoding. Unlike existing methods that use a fixed block size, this approach formulates block size selection as a lightweight policy learning problem based on prefilling representations.
- Predicts the optimal block size from the prefilling stage representation only once after prefilling.
- Exploits the local structure of optimal values to reduce the decision space.
- Achieves an acceptance length of 5.92 and a 4.20x speedup on Qwen3-4B under temperature T=1.
The method is described as plug-and-play with minimal overhead, consistently improving efficiency by adapting to individual samples.