github llama.cpp · 3h ago · inference

llama.cpp extends few-row MMA matmul to BF16, Q2_K, and other types

from English

The llama.cpp project has expanded its generic few-row Matrix Multiply (MMA) kernel to support additional source data types, including BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0, and various IQ types. This update allows the optimized kernel to handle any type with a 16-weight dequantizer, applying specific row count thresholds on M3 Ultra hardware where performance begins to exceed current kernels.

The new support covers BF16, Q1_0, Q2_0, MXFP4, Q2_K, Q3_K, TQ2_0, and IQ types.
Performance thresholds vary by type: 5 rows for TQ2_0, 4 for BF16, 3 for MXFP4/Q2_0/Q2_K/IQ4_NL, and 2 for others.
Benchmark results on M3 Ultra show performance ratios ranging from 0.23 to 1.01 depending on row count and kernel overlap.

This change improves inference efficiency for a wider range of quantization formats by leveraging the MMA kernel's capabilities at low row counts.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp adds GLM5-Next MTP support and optimizes NextN graph

The llama.cpp project has added support for the GLM5-Next multi-token-prediction (MTP) head, implementing a new graph structure to handle its specific architectural needs. This update includes optimizations for the NextN block, allowing it to skip dead compute during headless forwards and correctly extract hidden rows.

github llama.cpp · 7h ago · 1 view

llama.cpp b11463 accelerates GLM MLA prefill with MKL flash attention

The llama.cpp b11463 release includes a SYCL update that enables MKL flash attention for GLM-4.7 Flash's specific Multi-Latent Attention (MLA) shape, which was previously rejected by the dispatcher.

github llama.cpp · 2d ago · 27 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 7d ago · 44 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 10d ago · 33 views

llama.cpp b11207 adds tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon

The llama.cpp project released build b11207, which introduces support for tiled Q4_0 and Q8_0 GET_ROWS operations on the Hexagon backend. This update also includes fixes to macros, register spills, and the DMA pipeline, alongside improvements to kernel selection logic and re-enabled vectorization.