github llama.cpp · 1h ago · inference

llama.cpp b11463 accelerates GLM MLA prefill with MKL flash attention

from English

The llama.cpp b11463 release includes a SYCL update that enables MKL flash attention for GLM-4.7 Flash's specific Multi-Latent Attention (MLA) shape, which was previously rejected by the dispatcher.

The change admits the 576/576/512 GQA-20 F16 shape to the MKL pipeline and handles the V cache as a strided view of K rows.
An attempt to store MKL flash attention scores in F16 to reduce memory traffic was included but subsequently reverted in this build.
On an Intel Arc Pro B70, the MLA optimization improved prompt processing (pp8192) from 432.80 to 1292.29 tok/s, a 2.99x speedup.
The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 2d ago · 27 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 7d ago · 44 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 10d ago · 33 views

llama.cpp b11207 adds tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon

The llama.cpp project released build b11207, which introduces support for tiled Q4_0 and Q8_0 GET_ROWS operations on the Hexagon backend. This update also includes fixes to macros, register spills, and the DMA pipeline, alongside improvements to kernel selection logic and re-enabled vectorization.

github llama.cpp · 14d ago · 47 views

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.

github llama.cpp · 14d ago · 53 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.