github llama.cpp · 4d ago · inference

llama.cpp b11224 fixes Vulkan mul_mat wrong results on strided KV cache views

from English

The llama.cpp b11224 release addresses incorrect computation results in the Vulkan backend when matrix multiplication operations read slices of larger, strided tensors such as KV caches.

The batch stride for in-place src0 and src1 tensors is now correctly derived from nb[2] instead of ne00*ne01, ensuring proper row access across all heads.
In-place A and B ranges are sized by their strided extent using ggml_nbytes or ggml_vk_subbuffer to prevent reading zeroed-out data or hanging on NVIDIA hardware without coopmat2.
mul_mat_id and mul_mm paths are updated to handle strided expert views and partial tiles correctly, avoiding hangs on NVFP4 operations.

This fix ensures accurate inference results for decoder self-attention mechanisms that rely on strided memory layouts in the Vulkan backend.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 19h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 1d ago · 8 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 2d ago · 4 views

llama.cpp b11266 loads F32 A matrix 2 at a time for Vulkan on Intel

The llama.cpp project released build b11266, which includes a Vulkan optimization that loads F32 A matrices two elements at a time when they are 2-aligned. This change addresses performance issues on Intel hardware where loading floats one by one is inefficient, and it also corrects a missing validation for `a_offset` alignment in the original patch.

github llama.cpp · 2d ago · 1 view

llama.cpp b11265 fixes Vulkan MOE matmul tile selection

The llama.cpp b11265 release includes a fix for the Vulkan backend's handling of Mixture of Experts (MoE) models. The update corrects how `mat_mul_id` selects matrix multiplication tiles, addressing a performance issue where workers were left idle during dispatch.

github llama.cpp · 3d ago · 4 views

llama.cpp b11238 adds ggml_pad_ext for left padding and skips DFlash2 taps

The llama.cpp project released build b11238, introducing a unified left-padding implementation via the new `ggml_pad_ext` function. This change consolidates previous methods used by audio encoders like Parakeet, LFM2-Audio, Granite Speech, and Gemma 4, ensuring bit-identical embeddings for Gemma 4 while simplifying backend support.