github llama.cpp · 1h ago · inference

llama.cpp b11403 uses MMVF for thin f16/bf16 mul_mat at small batch size

from English

The llama.cpp project released version b11403, which includes a CUDA optimization to use the MMVF kernel for thin f16 and bf16 matrix multiplications at small batch sizes.

Adjusted kernel selection logic for improved performance on NVIDIA GPUs.
Provided binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL, Snapdragon), Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), Android, and openEuler.
Included iOS XCFramework and UI builds alongside the standard binaries.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b11405 tiles the lightning indexer over keys and tokens for 4 heads

The llama.cpp b11405 release updates the CUDA backend by tiling the lightning indexer over keys and tokens for 4 heads, addressing performance issues with small head counts.

github llama.cpp · 19h ago · 28 views

llama.cpp b11390 fixes CUDA MMQ memory fault

The llama.cpp project has released version b11390, which includes a fix for a CUDA MMQ memory fault that occurred when the number of experts was significantly larger than the micro-batch size.

github llama.cpp · 2d ago · 24 views

llama.cpp b11375 fixes graph reallocation aborts under GGML_SCHED_NO_REALLOC

The llama.cpp project released build b11375, which addresses a critical issue where non-contiguous ubatch cells forced graph reallocations that would abort when the GGML_SCHED_NO_REALLOC flag was set.

github llama.cpp · 2d ago · 29 views

llama.cpp b11372 halves qwen4exp indexer memory and adds CUDA 4-head support

The llama.cpp b11372 release introduces optimizations for the qwen4exp model's lightning indexer, primarily halving the memory required for indexer scores during long context processing. This is achieved by computing head scores in place rather than materializing separate tensors for each head.

github llama.cpp · 2d ago · 30 views

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.