The llama.cpp project released version b10321, which includes a critical fix for GGML_OP_NORM and GGML_OP_RMS_NORM operations on Metal. The update resolves an issue where threadgroups sized to leave a partial SIMD group resulted in incorrect mean and variance calculations due to dropped partial sums.

  • The bug occurred when row lengths were not multiples of the SIMD group size, causing the last lane's contribution to be ignored during cross-SIMD group reduction.
  • The fix rounds the threadgroup size up to the nearest whole number of SIMD groups, ensuring all partial sums are correctly aggregated without excessive idle lanes.
  • Test results on M3 Pro show NORM and RMS_NORM tests passing 50/50 and 51/51 respectively, compared to previous failures (25/50 and 26/51).
  • The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, Vulkan, OpenCL, ROCm, OpenVINO, SYCL, HIP), and openEuler.

This update ensures numerical correctness for norm operations on non-standard vector lengths, which is essential for models using channel dimensions that do not divide evenly by the vectorization factor.