github llama.cpp · 9h ago · inference

llama.cpp b10758 adds MUL_MAT and MUL_MAT_ID fusion for Hexagon

from English

The llama.cpp project released build b10758, introducing MUL_MAT and MUL_MAT_ID fusion capabilities for the Hexagon DSP. This update also includes several fixes to improve stability and performance on Qualcomm hardware.

Fuses QKV and FFN matmuls that land on HMX.
Removes the hardcoded ne[1] < 32K restriction.
Corrects overhead sizing to prevent exceeding vtcm budget for large dimensions.
Fuses MUL_MAT_ID into MUL_MAT_ID_NX variants where possible.
Updates opbatch and opqueue sizing to reduce trace buffer allocation overhead.
Adds virtual address space defragmentation to avoid aborts due to fragmentation.

These changes optimize memory usage and execution efficiency for Hexagon-based devices.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10766 adds DeepSeek 4 vision support

The llama.cpp project has released build b10766, which includes a fix to correctly support input vision for the DeepSeek 4 model.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10762 adds DeepSeek-V4-Flash-Vision-Exp support

The llama.cpp project has released build b10762, which introduces support for the DeepSeek-V4-Flash-Vision-Exp model. This update also includes handling of minimum and maximum token counts from the command line interface.

github llama.cpp · 9h ago · 3 views

llama.cpp b10759 avoids KleidiAI buffer type init on dispatch

The llama.cpp project released build b10759, which includes a fix to avoid KleidiAI buffer type initialization on dispatch.

github llama.cpp · 11h ago · 1 view

llama.cpp b10756 restricts Vulkan shader_bfloat16 extension request

The llama.cpp project released build b10756, which modifies Vulkan backend behavior to only request the VK_KHR_shader_bfloat16 extension when it is supported by the hardware.

github llama.cpp · 11h ago · 1 view

llama.cpp b10757 improves Vulkan IQ3_S mat-vec performance for larger batch sizes

The llama.cpp project released build b10757, which includes a specific optimization for the Vulkan backend. This update handles larger batch sizes greater than 4 efficiently for IQ3_S matrix-vector multiplication when NUM_COLS is greater than 4.