The llama.cpp project released build b10751, which introduces a fused kernel for the MoE (Mixture of Experts) weighted expert reduction on CUDA. This optimization replaces the previous baseline that ran two physical kernels with a single weighted-reduction kernel to reduce intermediate global-memory traffic.

  • The fused kernel handles k=2..15 experts via one runtime-k kernel, supporting both unscaled and scaled expert weighting graphs.
  • It matches structural op sequences, shapes, strides, and left-to-right ADD chains while maintaining the same reduction order.
  • Allocator integration ensures experts, router weights, and optional scales remain live until the fused destination is written.
  • Unrecognized or unsafe graphs fall back to the existing per-op path, which can be disabled via GGML_CUDA_MOE_WEIGHTED_REDUCTION=0.

This change improves performance for MoE models on CUDA by minimizing memory bandwidth usage during the expert combination phase.