github llama.cpp · just now · inference

llama.cpp b10906 fixes speculation after an image in the server

from English

The llama.cpp project released version b10906, which includes a fix for the server component regarding speculative decoding after image inputs. The change ensures that the actual position is passed to the drafter instead of the token count, affecting all drafters rather than just DFlash.

Renamed the draft parameter `n_past` to `pos0` to accurately reflect its role as a position indicator rather than a token count.
Provided pre-built binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, OpenCL, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This update corrects a parameter naming inconsistency and ensures proper position handling in the server's speculative decoding logic.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 16h ago · 2 views

llama.cpp b10899 adds Vulkan optimizations for Qwen and split_k support

The llama.cpp project released version b10899, introducing specific performance improvements to its Vulkan backend. The update focuses on optimizing matrix operations for the Qwen model family and enhancing general small M performance.

github llama.cpp · 19h ago · 1 view

llama.cpp b10896 fixes DFlash decoding failure with vision models

The llama.cpp project released build b10896, which addresses a critical issue where the drafter memory failed to allocate new tokens when using DFlash with vision models. This failure occurred because images reported a fixed offset, preventing proper token allocation.

github llama.cpp · 3d ago · 2 views

llama.cpp disables lazy tensor loading by default on iGPUs

The llama.cpp project has disabled lazy tensor loading by default for integrated GPUs (iGPUs) in release b10867. This change addresses a regression introduced in the previous "AUTO" lazy mode setting.

github llama.cpp · 3d ago · 4 views

llama.cpp b10864 fixes checkpoint eviction for hybrid models

The llama.cpp project released build b10864, which includes a fix for the checkpoint spacing eviction logic in `create_checkpoint()`. Previously, the eviction rule kept the oldest checkpoint and erased later ones within `checkpoint_min_step`, causing prompts shorter than that threshold to drop the checkpoint needed for resumption. This forced hybrid and recurrent models to unnecessarily re-prefill from the previous checkpoint.

github llama.cpp · 4d ago · 11 views

llama.cpp b10844 adds Vulkan DeepSeek-V4 hyper-connection fused ops

The llama.cpp project released version b10844, introducing support for DeepSeek-V4 hyper-connection fused operations (DSV4_HC_COMB/PRE/POST) in the Vulkan backend. This update brings Vulkan to parity with CUDA and Metal regarding these specific optimizations.