github llama.cpp · 1d ago · inference

llama.cpp b10690 fixes K cache crash during context shift

from English

The llama.cpp project has released version b10690, which includes a critical fix for unquantized key-value caches. The update prevents crashes that occurred during context shifts by ensuring the Hadamard matrix is copied to the k_rot tensor only when it has an assigned buffer.

Fixes crash in unquantized K cache during context shift by conditionally copying Hadamard matrix
Provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, ROCm), Android, and openEuler
Includes UI updates and attestations for the release

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 4h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 10h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.

github llama.cpp · 17h ago · 6 views

llama.cpp b10705 improves TENSOR_READ_LAZY handling

The llama.cpp project has released version b10705, which includes an improvement to the TENSOR_READ_LAZY handling mechanism. This update forces lazy tensors on CPU when the lazy setting is enabled.

github llama.cpp · 18h ago · 4 views

llama.cpp b10702 optimizes Q2_0 dot-product for AMD gfx1201

The llama.cpp project released version b10702, which includes a specific optimization for the Q2_0 quantization dot-product path on AMD's gfx1201 architecture. This update broadens the HIP implementation to use native amdgcn permutation instructions, improving performance for this specific hardware configuration.