github llama.cpp · 1h ago · inference

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

from English

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

The new kernel dequantizes Q8_0 KV tensors into a contiguous F16 scratch buffer, allowing the use of existing F16 Flash Attention kernels instead of in-kernel dequantization.
Support was extended to cover all quantized KV types supported by Metal, including Q4_0, Q4_1, Q5_0, and Q5_1.
For MLA-based models where V is a view of K, the implementation skips redundant V dequantization and reads V from the K F16 buffer.
Verification on an M2 Ultra showed that perplexity for Qwen2.5-0.5B with q8_0 KV matches the F16 reference exactly (PPL 1.0008).

This change enables efficient Flash Attention execution on Metal devices by leveraging contiguous F16 buffers for quantized KV caches, ensuring accuracy parity with standard F16 implementations.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 8h ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

github llama.cpp · 20d ago · 35 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 29d ago · 26 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

llama.cpp b10538 release with dequantized KV cache for large batches

llama.cpp b10533 adds graceful fallback for unsupported JSON schema regex patterns