github llama.cpp · 1h ago · inference

llama.cpp b11007 enables CUDA graph for MTP draft

from English

The llama.cpp project has released version b11007, which introduces support for CUDA graphs in Multi-Token Prediction (MTP) drafts. This update improves the efficiency of the drafting process by leveraging CUDA graph execution.

Enable CUDA graph for MTP draft to improve usage efficiency.
Provide binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.
Include UI binaries for all supported platforms.

This release allows users to benefit from optimized inference performance on compatible NVIDIA GPUs through the new MTP draft implementation.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.

github llama.cpp · 1d ago · 8 views

llama.cpp b10982 adds Vulkan sparse Flash Attention support for DSV4/GLM

The llama.cpp project has released build b10982, which introduces support for sparse Flash Attention via the Vulkan backend. This update specifically targets DSV4 and GLM models, allowing them to leverage sparse attention mechanisms on compatible GPU hardware.

github llama.cpp · 1d ago · 5 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 2d ago · 12 views

llama.cpp 0.4.1 adds Maple 20B-A1B, Tencent Hy 4, Spark2.5 support

The llama.cpp project released version 0.4.1, introducing support for the Maple 20B-A1B ternary Mixture of Experts architecture on CPU, as well as preview support for Tencent Hy 4 and Spark2.5 models.

github llama.cpp · 3d ago · 3 views

llama.cpp b10951 moves llama_n_rs_seq before llama_decode to avoid unnecessary calls

The llama.cpp project released version b10951, which includes a code optimization in the decoding logic. The `llama_n_rs_seq` function call is now executed before `llama_decode`, allowing the system to return directly if the check passes.