github llama.cpp · 1h ago · inference

llama.cpp OpenVINO backend adds Qwen3.5 support and memory optimizations

from English

The llama.cpp project has merged significant updates to its OpenVINO backend, primarily enabling support for the Qwen3.5 model family and introducing several memory optimization techniques.

Enabled GPT-OSS MoE, MXFP4, FILL, SIGMOID, SQR, SQRT, and multi-dim set rows operations.
Fixed accuracy issues in Gemma3n, Phi-3-mini (NEOX RoPE), MPT, Kimi-linear, and Minimax-m3 architectures.
Implemented GGML_OPENVINO_RELEASE_WEIGHTS to drop host weight RSS after compilation, reducing steady-state memory usage from ~1555 MB to ~710 MB for Llama-3.2-1B-Q4_K_M on Arc iGPU.
Added streaming weight requantization to reduce compile-time peak RSS by 1.06 GB for 1B models and 2.0 GB for 8B models.
Introduced a frontend model cache (GGML_OPENVINO_MODEL_CACHE_DIR) to skip graph conversion and compilation for repeated loads.

These changes improve compatibility with newer architectures like Qwen3.5 and significantly reduce both transient compile-time and steady-state memory footprints for GPU inference.

Importance 1/3 Trust 1/3 llama.cpp NVIDIA Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10418 adds SYCL host pinned memory support

The llama.cpp project released version b10418, introducing support for host pinned memory to improve SYCL Host-to-Device memory access. This update addresses thread-safety issues within the backend implementation.

github llama.cpp · 4h ago · 1 view

llama.cpp b10414 adds TQ2_0 support and optimizes mul_mv kernel

The llama.cpp project released build b10414, introducing support for the GGML_TYPE_TQ2_0 ternary quantization type in the Metal backend. This update also includes optimizations to the contiguous tensor (cont) mul_mv kernel.

github llama.cpp · 3d ago · 1 view

llama.cpp b10354 fixes CPU affinity mask on Android

The llama.cpp project has released version b10354, which includes a fix for the CPU affinity mask being ignored on Android devices.

github llama.cpp · 3d ago · 2 views

llama.cpp b10344 adds MTP support for Nemotron models

The llama.cpp project has released build b10344, which introduces Multi-Token Prediction (MTP) support specifically for the Nemotron and Nemotron Nano models.

github llama.cpp · 3d ago · 1 view

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.