github llama.cpp · 1h ago · inference

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

from English

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.

Routes large q6_K lm_head tensors to flat GEMV instead of gemv_noshuffle.
Adds a direct size condition to prevent slowdowns with models like gemma-4 E2B, which has dimensions [1536, 262144].
Provides binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

This change ensures that large weight matrices are processed efficiently on OpenCL devices without triggering performance bottlenecks.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 19h ago · 6 views

llama.cpp b10236 implements Metal DSv4 Lightning Indexer

The llama.cpp b10236 release introduces a Metal implementation of the DSv4 Lightning Indexer, specifically supporting 128-dimensional inputs with F32 queries and weights alongside F16 keys and masks.

github llama.cpp · 3d ago · 8 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 5d ago · 2 views

llama.cpp b10181 disables MMQ on devices with less than 48 KiB shared memory

The llama.cpp project released build b10181, which includes a fix in ggml-cuda to disable Matrix Multiplication Quantization (MMQ) on GPUs with less than 48 KiB of per-block shared memory. Previously, MMQ configurations assumed a minimum of 48 KiB, causing fatal errors on devices like NVIDIA Pascal GPUs and Moore Threads MTT S70 where the tile size exceeded available memory.

media Hugging Face Forums · 5d ago

Hugging Face fixes DynamicCache to prevent wrong output in generate()

A bug in Hugging Face's `generate()` function causes incorrect outputs when `past_key_values` is an externally-built `DynamicCache`. This issue affects KV cache compression methods like SnapKV, H2O, and KIVI that reconstruct caches from prefill data. The root cause is the failure to propagate positional context, leading to a 10.7pp accuracy loss on Qwen3-VL-8B.

github llama.cpp · 6d ago · 19 views

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

The llama.cpp project has released build b10165, which restores iq4_nl quantization support for the Flash Attention (FA) Vulkan backend. This change addresses issue #23681 by re-enabling the feature after concerns about shared memory usage were deemed negligible.