github llama.cpp · 1h ago · inference

llama.cpp b10781 fixes Vulkan FA dequant path engagement

from English

The llama.cpp project released build b10781, which includes a fix for the Vulkan fast attention (FA) dequantization path. The update addresses an issue where the shader incorrectly checked `nb[3]` even when `ne[3] == 1`, causing the path to only engage when the cache was full.

Skips the `nb[3]` check when `ne[3] == 1` because the shader never reads it for a single stream.
Resolves a logic error where cache views carried the full-buffer stride, reducing the old check to `n_kv == kv_size`.
Provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This fix ensures the Vulkan FA dequant path engages correctly during inference rather than being restricted to full cache states.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10780 enhances SYCL API to support peer-to-peer copy

The llama.cpp project has released version b10780, which includes a specific enhancement to the SYCL backend. This update modifies the API to enable peer-to-peer copy capabilities for SYCL devices.

github llama.cpp · 2h ago · 1 view

llama.cpp adds support for NVIDIA Nemotron-3-Puzzle-75B-A9B

llama.cpp introduces support for the NVIDIA Nemotron-3-Puzzle-75B-A9B model, enabling inference for this Mixture of Experts architecture. The update implements infrastructure for variable-per-layer expert feed-forward sizes and top-k routing, which are required because the model has 40 MoE layers with distinct n_ff_exp and top-k values.

github llama.cpp · 2h ago · 1 view

llama.cpp b10777 reduces redundant work in Q4_K multi-column MMVQ

The llama.cpp project released build b10777, which includes optimizations to the SYCL backend for Q4_K multi-column matrix-vector multiplication (MMVQ). The update focuses on reducing redundant computational work through weight unpacking and activation reuse strategies.

github llama.cpp · 8h ago · 1 view

ggml-org releases llama.cpp b10774 with finetune KV cache fix

The ggml-org project has released version b10774 of llama.cpp, which includes a fix for the no KV cache issue in the finetune module.

github llama.cpp · 9h ago · 1 view

llama.cpp b10772 adds F16 ABS support to Hexagon HTP backend

The llama.cpp release b10772 extends the Hexagon Tensor Processor (HTP) backend's F16 unary operation coverage by adding support for the absolute value (ABS) function.