The llama.cpp project released version b10704, which updates the CUDA backend to use the optimized mm_ids_helper path for any number of used experts (n_expert_used). Previously, this fast path was restricted by warp size divisibility rules, forcing most counts to fall back to a slower generic implementation.

  • The optimization generalizes padding to the next power of two, allowing cases like n_expert_used = 10 to utilize the fast path.
  • Measured on Qwen3.8-Flash-Next with 512 experts and 10 used at 55k context on an RTX PRO 6000, prompt processing speed increased from 2334 to 2600 tokens per second.
  • Token generation performance remains unaffected as the optimization targets batched token processing.

This change improves inference latency for models using Mixture of Experts architectures by reducing overhead during prompt processing.