github llama.cpp · just now · inference

llama.cpp b10777 reduces redundant work in Q4_K multi-column MMVQ

from English

The llama.cpp project released build b10777, which includes optimizations to the SYCL backend for Q4_K multi-column matrix-vector multiplication (MMVQ). The update focuses on reducing redundant computational work through weight unpacking and activation reuse strategies.

Optimizes Q4_K weight unpacking and reuses data between destination columns.
Implements subgroup reuse of activations for small N values (N=2..4) across two output rows.
Gates the two-row reuse pattern specifically for N=2 to improve efficiency.
Updates the magic number threshold to Q4_K_MMVQ_ROW_PAIR_MIN_NROWS=6272 and adds performance tests for Q4_K MUL_MAT coverage.

This release provides pre-built binaries for macOS, Linux, Android, Windows, and openEuler across various hardware backends including CPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 13h ago · 2 views

llama.cpp b10766 adds DeepSeek 4 vision support

The llama.cpp project has released build b10766, which includes a fix to correctly support input vision for the DeepSeek 4 model.

github llama.cpp · 14h ago · 4 views

llama.cpp b10762 adds DeepSeek-V4-Flash-Vision-Exp support

The llama.cpp project has released build b10762, which introduces support for the DeepSeek-V4-Flash-Vision-Exp model. This update also includes handling of minimum and maximum token counts from the command line interface.

github llama.cpp · 22h ago · 1 view

llama.cpp b10758 adds MUL_MAT and MUL_MAT_ID fusion for Hexagon

The llama.cpp project released build b10758, introducing MUL_MAT and MUL_MAT_ID fusion capabilities for the Hexagon DSP. This update also includes several fixes to improve stability and performance on Qualcomm hardware.

github llama.cpp · 22h ago · 3 views

llama.cpp b10759 avoids KleidiAI buffer type init on dispatch

The llama.cpp project released build b10759, which includes a fix to avoid KleidiAI buffer type initialization on dispatch.

github llama.cpp · 1d ago · 1 view

llama.cpp b10756 restricts Vulkan shader_bfloat16 extension request

The llama.cpp project released build b10756, which modifies Vulkan backend behavior to only request the VK_KHR_shader_bfloat16 extension when it is supported by the hardware.