The llama.cpp b9931 release introduces an OpenCL optimization for Mixture of Experts (MoE) prefill that skips padded expert tiles to improve performance.

  • The optimization groups tokens into TILESIZE_N=32 per-expert tiles and skips the second dotx16_reduce8 half when a tile's upper 16 slots are all padding.
  • It replaces two half-tile dotx16_reduce8 calls with four dotx8_reduce4 calls, allowing independent skipping of empty trailing skip-groups.
  • The feature is controlled by the GGML_OPENCL_MOE_RAGGED_GRAN environment variable, which accepts values 8, 16, or 32 for quarter, half, or off granularity respectively.
  • The release includes binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

This optimization allows users to reduce computation on padded data in MoE models via the new environment variable settings.