The llama.cpp project has fused the RMS_NORM and SCALE operations into a single CUDA kernel to reduce host/driver overhead. Previously, separate SCALE nodes added extra kernel launches that were negligible in plain batch processing but significantly increased latency during draft-mtp speculative decoding.

  • The fusion adds a do_scale flag to rms_norm_f32, allowing the fused path to share the kernel, reduction, and launcher while maintaining bit-for-bit numerical equivalence.
  • On a 2x GTX 1080 Ti setup with Qwen3.8-27B, cold prefill throughput improved by 4.2% for 8000 tokens and 4.8% for 20000 tokens under draft-mtp conditions.
  • Launch counts per ubatch decreased from 1032.9 + 841.7 back to 978.9 + 799.7, matching previous baseline counts without changing GPU operation sums.
  • All backend tests for RMS_NORM_SCALE, NORM_SCALE, and related operations pass on both GPUs, with perplexity remaining identical to the unfused build.

This optimization improves inference speed specifically in speculative decoding scenarios by reducing the cumulative cost of kernel launches.