github llama.cpp · 5d ago · inference

llama.cpp b11195 adds tiled mul_mat for k-quants with 3-6x speedup

from English

The llama.cpp project released version b11195, introducing a tiled matrix multiplication implementation for k-quants in the CPU backend. This change unpacks quantized data into 256x256 int8 tiles and computes results using microkernels to improve performance on large matrix operations.

Tiled mul_mat provides a 3-6x speed improvement for large matrix multiplications, with break-even at 4096x64 * 64x4096 dimensions.
Error rates remain trivial, with maximum errors around 1-e04 and RMSE around 1-e05.
The release includes fixes for ARM and Windows builds, unified IQP support for Q5_K and IQ4_XS benchmarks, and optimized AVX2 kernels.

The update improves inference speed for large matrix operations while maintaining numerical accuracy across supported platforms.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 11h ago · 2 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 20h ago · 2 views

llama.cpp b11266 loads F32 A matrix 2 at a time for Vulkan on Intel

The llama.cpp project released build b11266, which includes a Vulkan optimization that loads F32 A matrices two elements at a time when they are 2-aligned. This change addresses performance issues on Intel hardware where loading floats one by one is inefficient, and it also corrects a missing validation for `a_offset` alignment in the original patch.

github llama.cpp · 22h ago · 1 view

llama.cpp b11265 fixes Vulkan MOE matmul tile selection

The llama.cpp b11265 release includes a fix for the Vulkan backend's handling of Mixture of Experts (MoE) models. The update corrects how `mat_mul_id` selects matrix multiplication tiles, addressing a performance issue where workers were left idle during dispatch.

github llama.cpp · 2d ago · 4 views

llama.cpp b11238 adds ggml_pad_ext for left padding and skips DFlash2 taps

The llama.cpp project released build b11238, introducing a unified left-padding implementation via the new `ggml_pad_ext` function. This change consolidates previous methods used by audio encoders like Parakeet, LFM2-Audio, Granite Speech, and Gemma 4, ensuring bit-identical embeddings for Gemma 4 while simplifying backend support.

github llama.cpp · 2d ago · 5 views

llama.cpp b11232 enables tiled flash attention on x86

The llama.cpp project released build b11232, which includes updates to the ggml-cpu backend. The release focuses on enabling tiled flash attention for non-vector-multiple head dimensions on x86 architectures.