github llama.cpp · 1h ago · inference

ggml-openvino updates to 2026.4.1 with MoE fusion and device listing fixes

from English

The ggml-openvino backend for llama.cpp has been updated to version 2026.4.1, introducing performance optimizations, expanded operator support, and improved device enumeration.

Fusion of Mixture-of-Experts (MoE) routing and GDN QK normalization, with GPU MoE fusion enabled by default.
Support for fused gate-up weights in models like gemma-4, achieving a prefill throughput increase from 66.16 to 1608.73 t/s on Arc B390.
Fix for rank-3 axis handling in stateful execution, resolving graph build aborts for MoE models.
Implementation of PRD-compliant device enumeration and memory reporting, ensuring accurate GPU/IGPU distinction.
Addition of the k-requant option q4_asym64 and cache_only mode for importing compiled models directly from disk.

These changes improve inference performance for MoE architectures and provide more reliable hardware detection and configuration for OpenVINO users.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11375 fixes graph reallocation aborts under GGML_SCHED_NO_REALLOC

The llama.cpp project released build b11375, which addresses a critical issue where non-contiguous ubatch cells forced graph reallocations that would abort when the GGML_SCHED_NO_REALLOC flag was set.

github llama.cpp · 2h ago · 1 view

llama.cpp b11372 halves qwen4exp indexer memory and adds CUDA 4-head support

The llama.cpp b11372 release introduces optimizations for the qwen4exp model's lightning indexer, primarily halving the memory required for indexer scores during long context processing. This is achieved by computing head scores in place rather than materializing separate tensors for each head.

github llama.cpp · 8h ago · 2 views

llama.cpp b11370 fuses shared experts into MMVQ

The llama.cpp project has released build b11370, which includes a CUDA optimization to fuse shared experts into the Mixture of MoE with Vector Quantization (MMVQ) architecture. This change is part of pull request #29184 and also addresses buffer null checks and argument handling for stride_col_dst.

github llama.cpp · 11h ago · 1 view

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

github llama.cpp · 14h ago · 1 view

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.