github llama.cpp · just now · inference

llama.cpp b10908 fixes idle threads in Metal iq mul_mv kernels

from English

The llama.cpp project released version b10908, which includes a fix for idle threads in the remaining iq mul_mv kernels for ne00 < 1024. This change generalizes the row split logic to six specific kernels: iq1_s, iq1_m, iq2_xxs, iq2_xs, iq2_s, and iq3_s.

The fix addresses cases where rows have fewer than 32 chunks by allowing threads to share chunks and take slices of the rows.
It offsets the src0 row pointer once in the iq mul_mv kernels since q2, dh, sc, qh, and signs are derived from xr.
The implementation folds the iq mul_mv row split into offset0 to keep x and its derived pointers on the existing path.

This release provides binaries for macOS, Linux, Android, Windows, and openEuler across various architectures including CPU, CUDA, ROCm, Vulkan, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10907 fixes MTP context kv cache allocation for deepseek2, glm4moe, cohere2moe

The llama.cpp project released version b10907, which includes a fix for Multi-Token Prediction (MTP) context key-value cache allocation. This update addresses issues affecting the deepseek2, glm4moe, and cohere2moe architectures.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10905 tunes Flash Attention for AMD RDNA4 and revises stream-k logic

The llama.cpp project released build b10905, which includes specific tuning of Flash Attention for CUDA/HIP on the gfx1201 architecture. This update focuses on optimizing performance for AMD RDNA4 GPUs.

github llama.cpp · 1d ago · 5 views

llama.cpp b10888 adds Vulkan command-buffer debug labels for GPU profilers

The llama.cpp project has released build b10888, which includes a new feature to add command-buffer debug labels for GPU profilers in the Vulkan backend.

github llama.cpp · 1d ago · 1 view

llama.cpp b10886 adds Q1_0 vector intrinsic support for s390x

The llama.cpp project released build b10886, which introduces Q1_0 vector intrinsic support for the s390x architecture. This update includes the implementation of `ggml_vec_dot_q1_0_q8_0` and updates documentation to reflect the new capability.

github llama.cpp · 2d ago · 2 views

llama.cpp b10883 fixes Vulkan matrix multiplication and restores Ampere optimizations

The llama.cpp project released build b10883, which includes significant updates to the Vulkan backend. The primary change involves using spec constants for matrix-matrix multiplication A-type operations to improve shader compilation and performance.