github llama.cpp · 4d ago · inference

llama.cpp b10827 fixes OpenCL weights pack selection for q4_K and q5_K mul_mat

from English

The llama.cpp project has released version b10827, which includes a fix to properly choose the weights pack for q4_K and q5_K matrix multiplication operations using OpenCL.

The release provides binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x), Windows (CPU, CUDA 12/13, Vulkan, ROCm, SYCL), Android, and iOS.
OpenCL Adreno support is available on Windows arm64, while KleidiAI remains disabled for macOS Apple Silicon.
The update also includes a new UI build and attestation links for verification.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 3d ago · 7 views

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).

github llama.cpp · 7d ago · 21 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

lab Hugging Face Blog · 9d ago · 13 views

Hugging Face releases 207 WebGPU kernels and a browser benchmarking tool

Hugging Face has released @huggingface/kernels, a library for loading and running optimized WebGPU kernels from the Hugging Face Hub, alongside an initial collection of 207 kernels. The release also introduces Fleet, an in-browser GPU benchmarking suite that crowdsources performance and correctness evidence from users' devices.

github llama.cpp · 10d ago · 19 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 12d ago · 19 views

llama.cpp enables xmem GEMM and bypasses tiled f32 on Adreno GPUs

The llama.cpp project has updated its OpenCL backend to improve matrix multiplication performance on two generations of Adreno GPUs. The changes primarily target the X2E generation by defaulting the xmem F16xF32 GEMM path and optimizing handling for the A7X generation.