github llama.cpp · 2h ago · inference

llama.cpp b10718 extends MOE fusion to specdec and multi-token

from English

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

CUDA: Extended MOE fusion to specdec, removing the single-token restriction for earlier MOE glu and topk-router fusions.
Added SWIGLU_CLAMP case support to the multi-token MOE fusion implementation.
Addressed review comments from the contributing pull request #27621.

This update improves performance for models using speculative decoding by enabling more efficient fused operations on NVIDIA GPUs.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 2d ago · 8 views

llama.cpp enables xmem GEMM and bypasses tiled f32 on Adreno GPUs

The llama.cpp project has updated its OpenCL backend to improve matrix multiplication performance on two generations of Adreno GPUs. The changes primarily target the X2E generation by defaulting the xmem F16xF32 GEMM path and optimizing handling for the A7X generation.

github llama.cpp · 3d ago · 9 views

llama.cpp b10670 routes quantized KV decode to TILE on Xe2 (BMG)

The llama.cpp project released version b10670, which includes a specific optimization for Intel's Xe2 architecture. The update routes the quantized key-value (KV) decode operation to use the TILE backend exclusively for BMG hardware, while keeping the VEC backend on other architectures until they are validated.

github llama.cpp · 4d ago · 23 views

llama.cpp enables xmem GEMM and bypasses tiled f32 on Adreno GPUs

llama.cpp b10670 routes quantized KV decode to TILE on Xe2 (BMG)

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

llama.cpp b10653 adds TENSOR_READ_LAZY for model loading

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2