github llama.cpp · just now · inference

llama.cpp b10236 implements Metal DSv4 Lightning Indexer

from English

The llama.cpp b10236 release introduces a Metal implementation of the DSv4 Lightning Indexer, specifically supporting 128-dimensional inputs with F32 queries and weights alongside F16 keys and masks.

The update includes tiled and tail kernels for handling KV lengths near 8- and 64-element boundaries.
K tiles are staged and dequantized in F16 threadgroup memory before simdgroup matrix loads, supporting F32, F16, BF16, and various quantization formats.
Benchmarks show improved prefill throughput (pp512) at sequence lengths of 10k, 20k, and 30k tokens compared to previous versions.

This optimization enhances inference performance on Apple Silicon devices by improving memory access patterns for the Lightning Indexer.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 3d ago · 8 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 5d ago · 1 view

llama.cpp b10181 disables MMQ on devices with less than 48 KiB shared memory

The llama.cpp project released build b10181, which includes a fix in ggml-cuda to disable Matrix Multiplication Quantization (MMQ) on GPUs with less than 48 KiB of per-block shared memory. Previously, MMQ configurations assumed a minimum of 48 KiB, causing fatal errors on devices like NVIDIA Pascal GPUs and Moore Threads MTT S70 where the tile size exceeded available memory.

media Hugging Face Forums · 5d ago

Hugging Face fixes DynamicCache to prevent wrong output in generate()

A bug in Hugging Face's `generate()` function causes incorrect outputs when `past_key_values` is an externally-built `DynamicCache`. This issue affects KV cache compression methods like SnapKV, H2O, and KIVI that reconstruct caches from prefill data. The root cause is the failure to propagate positional context, leading to a 10.7pp accuracy loss on Qwen3-VL-8B.

github llama.cpp · 6d ago · 19 views

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

The llama.cpp project has released build b10165, which restores iq4_nl quantization support for the Flash Attention (FA) Vulkan backend. This change addresses issue #23681 by re-enabling the feature after concerns about shared memory usage were deemed negligible.

github llama.cpp · 6d ago

llama.cpp b10164 adds chunked SSD matmul for Mamba-2 prefill acceleration

The llama.cpp project released version b10164, introducing a new chunked SSD matrix multiplication implementation designed to accelerate the prefill phase of Mamba-2 models.