github llama.cpp · 2h ago · inference

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

from English

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

The new approach splits rows into contiguous segments processed by subgroups or threads, using ballot counting over coalesced loads to maintain an ascending index list.
Pre-built binaries are available for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL, Snapdragon), Windows (CPU, Vulkan, CUDA, OpenVINO, SYCL, ROCm), Android, and iOS.
The release also includes the llama.cpp UI and attestations for security verification.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 5d ago · 37 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 8d ago · 30 views

llama.cpp b11207 adds tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon

The llama.cpp project released build b11207, which introduces support for tiled Q4_0 and Q8_0 GET_ROWS operations on the Hexagon backend. This update also includes fixes to macros, register spills, and the DMA pipeline, alongside improvements to kernel selection logic and re-enabled vectorization.

github llama.cpp · 12d ago · 30 views

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.

github llama.cpp · 12d ago · 41 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 13d ago · 37 views

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.