github llama.cpp · 3d ago · inference

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

from English

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).

Branchless unpacking prevents scale re-execution for every column in mmvq, improving performance at batch sizes greater than 1.
L2 prefetch logic is gated specifically for DGX Spark hardware to ensure gains are realized.
The mmvq L2 prefetch guard is extended to support MUSA and HIP backends alongside CUDA.
Switch points for Q4_K are updated to accommodate a wider range of models.

These changes provide measurable performance improvements for users running llama.cpp on DGX Spark systems with batch sizes exceeding one.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 4d ago · 9 views

llama.cpp b10827 fixes OpenCL weights pack selection for q4_K and q5_K mul_mat

The llama.cpp project has released version b10827, which includes a fix to properly choose the weights pack for q4_K and q5_K matrix multiplication operations using OpenCL.

github llama.cpp · 7d ago · 21 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

lab Hugging Face Blog · 9d ago · 13 views

Hugging Face releases 207 WebGPU kernels and a browser benchmarking tool

Hugging Face has released @huggingface/kernels, a library for loading and running optimized WebGPU kernels from the Hugging Face Hub, alongside an initial collection of 207 kernels. The release also introduces Fleet, an in-browser GPU benchmarking suite that crowdsources performance and correctness evidence from users' devices.

github llama.cpp · 10d ago · 19 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 12d ago · 19 views

llama.cpp enables xmem GEMM and bypasses tiled f32 on Adreno GPUs

The llama.cpp project has updated its OpenCL backend to improve matrix multiplication performance on two generations of Adreno GPUs. The changes primarily target the X2E generation by defaulting the xmem F16xF32 GEMM path and optimizing handling for the A7X generation.