github llama.cpp · 2d ago · inference

llama.cpp b11265 fixes Vulkan MOE matmul tile selection

from English

The llama.cpp b11265 release includes a fix for the Vulkan backend's handling of Mixture of Experts (MoE) models. The update corrects how `mat_mul_id` selects matrix multiplication tiles, addressing a performance issue where workers were left idle during dispatch.

Previously, tile selection relied on total token count, which was incorrect for MoE dispatch grids where the true N per workgroup is per-expert rows.
On models like Sarvam 30B with pp128, this error caused the picker to select tiles for approximately 6 live rows instead of 128.
The fix ensures that all workers in a group have work to do, eliminating wasted time that previously accounted for 55% of the job's duration.

This change improves inference efficiency for MoE models on Vulkan by ensuring proper workload distribution across GPU workers.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 19h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 1d ago · 8 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 2d ago · 4 views

llama.cpp b11266 loads F32 A matrix 2 at a time for Vulkan on Intel

The llama.cpp project released build b11266, which includes a Vulkan optimization that loads F32 A matrices two elements at a time when they are 2-aligned. This change addresses performance issues on Intel hardware where loading floats one by one is inefficient, and it also corrects a missing validation for `a_offset` alignment in the original patch.

github llama.cpp · 3d ago · 4 views

llama.cpp b11238 adds ggml_pad_ext for left padding and skips DFlash2 taps

The llama.cpp project released build b11238, introducing a unified left-padding implementation via the new `ggml_pad_ext` function. This change consolidates previous methods used by audio encoders like Parakeet, LFM2-Audio, Granite Speech, and Gemma 4, ensuring bit-identical embeddings for Gemma 4 while simplifying backend support.

github llama.cpp · 3d ago · 5 views

llama.cpp b11232 enables tiled flash attention on x86

The llama.cpp project released build b11232, which includes updates to the ggml-cpu backend. The release focuses on enabling tiled flash attention for non-vector-multiple head dimensions on x86 architectures.