github llama.cpp · 1h ago · inference

llama.cpp b11405 tiles the lightning indexer over keys and tokens for 4 heads

from English

The llama.cpp b11405 release updates the CUDA backend by tiling the lightning indexer over keys and tokens for 4 heads, addressing performance issues with small head counts.

The kernel stages queries and weights for all heads at once, widening each half-precision key element once to feed all heads without a float round trip.
Queries remain in float within shared memory to prevent overflow from half2 products when q * k exceeds the f16 range.
Each thread now scores two keys for a single token, reducing shared reads and keeping gfx908 at 63 VGPRs with no register spill.
This optimization makes the kernel 36x faster on an R9700 and slightly faster on CUDA hardware.

The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU (CUDA, ROCm, Vulkan, OpenCL), and NPU backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b11403 uses MMVF for thin f16/bf16 mul_mat at small batch size

The llama.cpp project released version b11403, which includes a CUDA optimization to use the MMVF kernel for thin f16 and bf16 matrix multiplications at small batch sizes.

github llama.cpp · 19h ago · 28 views

llama.cpp b11390 fixes CUDA MMQ memory fault

The llama.cpp project has released version b11390, which includes a fix for a CUDA MMQ memory fault that occurred when the number of experts was significantly larger than the micro-batch size.

github llama.cpp · 2d ago · 24 views

llama.cpp b11375 fixes graph reallocation aborts under GGML_SCHED_NO_REALLOC

The llama.cpp project released build b11375, which addresses a critical issue where non-contiguous ubatch cells forced graph reallocations that would abort when the GGML_SCHED_NO_REALLOC flag was set.

github llama.cpp · 2d ago · 29 views

llama.cpp b11372 halves qwen4exp indexer memory and adds CUDA 4-head support

The llama.cpp b11372 release introduces optimizations for the qwen4exp model's lightning indexer, primarily halving the memory required for indexer scores during long context processing. This is achieved by computing head scores in place rather than materializing separate tensors for each head.

github llama.cpp · 2d ago · 30 views

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.