github llama.cpp · 1h ago · inference

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

from English

The llama.cpp project has released build b10165, which restores iq4_nl quantization support for the Flash Attention (FA) Vulkan backend. This change addresses issue #23681 by re-enabling the feature after concerns about shared memory usage were deemed negligible.

Adds iq4_nl support back to the Vulkan FA implementation.
Adds q1_0 support for non-coopmat2 configurations.
Fixes GitHub issue #23681 regarding the removal of q1_0 FA support.

The release provides binaries for macOS, Linux, Windows, and Android across CPU, CUDA, ROCm, OpenVINO, SYCL, and Vulkan backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10164 adds chunked SSD matmul for Mamba-2 prefill acceleration

The llama.cpp project released version b10164, introducing a new chunked SSD matrix multiplication implementation designed to accelerate the prefill phase of Mamba-2 models.

github llama.cpp · 6h ago · 1 view

llama.cpp b10159 adds FWHT kernel for Metal backend

The llama.cpp project released build b10159, which introduces a Fast Walsh-Hadamard Transform (FWHT) kernel for the Metal backend. This update includes work-in-progress implementation details along with shape guards and code formatting adjustments.

github llama.cpp · 8h ago · 2 views

llama.cpp b10158 adds eagle3-v3 support for gpt-oss model

The llama.cpp project has released build b10158, which introduces support for the eagle3-v3 architecture within the gpt-oss model.

github llama.cpp · 5d ago · 2 views

llama.cpp b10098 updates Hexagon activation ops

The llama.cpp project released version b10098, which includes an update to the Hexagon activation operations. This release provides binaries for macOS, Linux, Android, Windows, and openEuler across various CPU and GPU backends.

lab Hugging Face Blog · 5d ago · 1 view

Diffusers adds native Nunchaku Lite 4-bit inference support

Hugging Face Diffusers now supports loading Nunchaku Lite checkpoints natively via `from_pretrained()`, eliminating the need for a separate inference engine or local CUDA compilation. This integration utilizes the SVDQuant method to run transformer layers with 4-bit weights and activations (W4A4), significantly reducing memory usage while improving inference speed.