github llama.cpp · 11h ago · inference

llama.cpp b10757 improves Vulkan IQ3_S mat-vec performance for larger batch sizes

from English

The llama.cpp project released build b10757, which includes a specific optimization for the Vulkan backend. This update handles larger batch sizes greater than 4 efficiently for IQ3_S matrix-vector multiplication when NUM_COLS is greater than 4.

Achieves 5x performance improvement at n=8 for the specified Vulkan IQ3_S mat-vec configuration.
Adds two cases per quant type at k=16*256 to the all_types mat-vec sweep.
Provides binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, OpenCL, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This release enables users to run llama.cpp on a wide variety of hardware platforms with updated Vulkan optimizations for specific quantization types.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10766 adds DeepSeek 4 vision support

The llama.cpp project has released build b10766, which includes a fix to correctly support input vision for the DeepSeek 4 model.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10762 adds DeepSeek-V4-Flash-Vision-Exp support

The llama.cpp project has released build b10762, which introduces support for the DeepSeek-V4-Flash-Vision-Exp model. This update also includes handling of minimum and maximum token counts from the command line interface.

github llama.cpp · 9h ago · 1 view

llama.cpp b10758 adds MUL_MAT and MUL_MAT_ID fusion for Hexagon

The llama.cpp project released build b10758, introducing MUL_MAT and MUL_MAT_ID fusion capabilities for the Hexagon DSP. This update also includes several fixes to improve stability and performance on Qualcomm hardware.

github llama.cpp · 9h ago · 3 views

llama.cpp b10759 avoids KleidiAI buffer type init on dispatch

The llama.cpp project released build b10759, which includes a fix to avoid KleidiAI buffer type initialization on dispatch.

github llama.cpp · 11h ago · 1 view

llama.cpp b10756 restricts Vulkan shader_bfloat16 extension request

The llama.cpp project released build b10756, which modifies Vulkan backend behavior to only request the VK_KHR_shader_bfloat16 extension when it is supported by the hardware.