The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

  • Fixes stateful decode failures for Gemma-4 by correctly handling per-layer-type head counts and sliding-window layers.
  • Improves stateful decoding performance on GPU, with gemma-4-12B increasing from 6.27 to 9.11 t/s at depth 8192.
  • Adds support for GGML_OPENVINO_REQUANT_KQUANT to select a 4-bit requant target and GGML_OPENVINO_SPILL_DIR to spill weight buffers to disk.
  • Optimizes GPU MoE inference by fusing the expert block into MOECompressed and requantizing grouped 8-bit experts.
  • Enables cacheless encoder models on NPU by accepting packed QKV RoPE views and detecting cacheless attention from masks.
  • Fixes mixed-dtype ADD/SWIGLU_CLAMP operations by upcasting mismatched operand types to f32 to preserve precision.

These changes ensure accurate inference for complex model architectures like Gemma-4 while providing measurable speedups for stateful decoding and MoE workloads on compatible hardware.