github llama.cpp · just now · inference

llama.cpp b11375 fixes graph reallocation aborts under GGML_SCHED_NO_REALLOC

from English

The llama.cpp project released build b11375, which addresses a critical issue where non-contiguous ubatch cells forced graph reallocations that would abort when the GGML_SCHED_NO_REALLOC flag was set.

The fix gathers recurrent states once so the reserve covers every split, ensuring the worst-case reserve size is sufficient.
A single get_rows now gathers n_rs states, with ubatch and extra states treated as views of this single allocation.
Custom getters like mamba ssm_scan gather from the second state to avoid copying state for single sequence ubatches.
Views are built once per graph to keep host overhead unchanged.

This change ensures stable execution in constrained scheduling environments by preventing aborts caused by memory reallocation constraints.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b11372 halves qwen4exp indexer memory and adds CUDA 4-head support

The llama.cpp b11372 release introduces optimizations for the qwen4exp model's lightning indexer, primarily halving the memory required for indexer scores during long context processing. This is achieved by computing head scores in place rather than materializing separate tensors for each head.

github llama.cpp · 11h ago · 1 view

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

github llama.cpp · 14h ago · 1 view

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.

github llama.cpp · 18h ago · 4 views

llama.cpp b11351 adds alloc_buffer_n and get_alloc_size_n to ggml backend buffer type interface

The llama.cpp project released build b11351, introducing new methods to the ggml backend buffer type interface. The update adds `alloc_buffer_n` and `get_alloc_size_n` to allow for more granular control over buffer allocation and size querying.

github llama.cpp · 1d ago · 2 views

llama.cpp b11338 adds shared strided DMA copy for CPY and CONCAT

The llama.cpp project released build b11338, introducing a shared strided Direct Memory Access (DMA) copy mechanism for the CPY and CONCAT operations. This update enables any-dimension CONCAT via DMA on Hexagon architectures.