github llama.cpp · just now · inference

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

from English

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.

Implementation of q6k and q4k kernels for Hexagon DSPs.
Improved unpack accuracy for the hex-q6k kernel.
Added support for Q4_K kernels on Hexagon hardware.

The release provides pre-built binaries for macOS, Linux, Windows, Android, and openEuler across various CPU and GPU backends including CUDA, ROCm, Vulkan, and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 1d ago · 5 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 9d ago · 21 views

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).

github llama.cpp · 10d ago · 21 views

llama.cpp b10827 fixes OpenCL weights pack selection for q4_K and q5_K mul_mat

The llama.cpp project has released version b10827, which includes a fix to properly choose the weights pack for q4_K and q5_K matrix multiplication operations using OpenCL.

github llama.cpp · 13d ago · 28 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

lab Hugging Face Blog · 15d ago · 20 views

Hugging Face releases 207 WebGPU kernels and a browser benchmarking tool

Hugging Face has released @huggingface/kernels, a library for loading and running optimized WebGPU kernels from the Hugging Face Hub, alongside an initial collection of 207 kernels. The release also introduces Fleet, an in-browser GPU benchmarking suite that crowdsources performance and correctness evidence from users' devices.