github llama.cpp · 5h ago · inference

llama.cpp b10715 fuses DFlash encoder into KV cache injection

from English

The llama.cpp project released build b10715, which includes a key optimization to the DFlash implementation. The DFlash encoder is now fused directly into the KV cache injection process during decoding.

Previously, running the encoder as a separate `llama_encode` call forced a device-to-host round trip for its output before the injection decode could re-upload it, along with a second graph build per round. This change folds the encoder into the decoder's embedding branch, allowing target features to be fed directly to a single `llama_decode` call.

The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b10721 fixes WebGPU crash and adds Windows CUDA 13.4 support

The llama.cpp project has released build b10721, which includes a critical fix for the WebGPU backend and expands hardware support on Windows.

github llama.cpp · 2h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 6h ago · 1 view

llama.cpp b10714 tunes Vulkan mat-vec rows for Strix Halo

The llama.cpp project released build b10714, which includes specific optimizations for the Vulkan backend targeting AMD Strix Halo hardware. The update adjusts matrix-vector multiplication row counts to improve performance during batched inference.

github llama.cpp · 12h ago · 3 views

llama.cpp b10706 adds SWIGLU_CLAMP and Vulkan shader support

The llama.cpp project has released build b10706, introducing new functionality to the ggml backend. This update includes the addition of SWIGLU_CLAMP and a new Vulkan shader implementation.

github llama.cpp · 13h ago · 9 views

llama.cpp b10707 optimizes KV cache cell scanning for faster generation

The llama.cpp project released version b10707, which includes a performance optimization to the key-value (KV) cache handling. The update modifies the `for_each_token_in` function to stop scanning once all sequences within a specific cell have been seen, rather than iterating through all possible maximum sequences.