github llama.cpp · just now · inference

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

from English

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

Optimizes opencl quant lm_head / decode GEMV and medium-batch GEMM for speculative decoding/MTP.
Guards q4_K/q6_K tiled_ns convert-kernel registration for non-Adreno builds.
Gates q4_K MUL_MAT+GLU fusion dispatch specifically to Adreno hardware.
Requires the noshuffle weight layout in the q4_K GLU fusion gate.
Prevents taking the vectorized f16 mrow GEMV path on unaligned row strides.
Enables q4_K split-K decode GEMV only where performance gains are measured.
Restricts tiled lm_head/embed GEMV defaults to X2E/A8X architectures.

The release provides binaries for macOS, iOS, Linux, Windows, Android, and openEuler across CPU, GPU, and various accelerator backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

lab Hugging Face Blog · 2d ago · 6 views

Hugging Face releases 207 WebGPU kernels and a browser benchmarking tool

Hugging Face has released @huggingface/kernels, a library for loading and running optimized WebGPU kernels from the Hugging Face Hub, alongside an initial collection of 207 kernels. The release also introduces Fleet, an in-browser GPU benchmarking suite that crowdsources performance and correctness evidence from users' devices.

github llama.cpp · 3d ago · 10 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 5d ago · 9 views

llama.cpp enables xmem GEMM and bypasses tiled f32 on Adreno GPUs

The llama.cpp project has updated its OpenCL backend to improve matrix multiplication performance on two generations of Adreno GPUs. The changes primarily target the X2E generation by defaulting the xmem F16xF32 GEMM path and optimizing handling for the A7X generation.

github llama.cpp · 6d ago · 11 views

llama.cpp b10670 routes quantized KV decode to TILE on Xe2 (BMG)

The llama.cpp project released version b10670, which includes a specific optimization for Intel's Xe2 architecture. The update routes the quantized key-value (KV) decode operation to use the TILE backend exclusively for BMG hardware, while keeping the VEC backend on other architectures until they are validated.

github llama.cpp · 7d ago · 30 views

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.