github llama.cpp · 1h ago · inference

llama.cpp b11390 fixes CUDA MMQ memory fault

from English

The llama.cpp project has released version b11390, which includes a fix for a CUDA MMQ memory fault that occurred when the number of experts was significantly larger than the micro-batch size.

Resolves a memory fault in the CUDA backend related to Mixture-of-Experts (MMQ) handling.
Provides binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL, Snapdragon), Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), Android, and openEuler.
Includes an updated UI build.

This update ensures stability for users running Mixture-of-Experts models on CUDA hardware by preventing the specific memory access error.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1d ago · 12 views

llama.cpp b11375 fixes graph reallocation aborts under GGML_SCHED_NO_REALLOC

The llama.cpp project released build b11375, which addresses a critical issue where non-contiguous ubatch cells forced graph reallocations that would abort when the GGML_SCHED_NO_REALLOC flag was set.

github llama.cpp · 1d ago · 15 views

llama.cpp b11372 halves qwen4exp indexer memory and adds CUDA 4-head support

The llama.cpp b11372 release introduces optimizations for the qwen4exp model's lightning indexer, primarily halving the memory required for indexer scores during long context processing. This is achieved by computing head scores in place rather than materializing separate tensors for each head.

github llama.cpp · 1d ago · 17 views

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

github llama.cpp · 2d ago · 16 views

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.

github llama.cpp · 2d ago · 16 views

llama.cpp b11351 adds alloc_buffer_n and get_alloc_size_n to ggml backend buffer type interface

The llama.cpp project released build b11351, introducing new methods to the ggml backend buffer type interface. The update adds `alloc_buffer_n` and `get_alloc_size_n` to allow for more granular control over buffer allocation and size querying.