github llama.cpp · 2d ago · inference

llama.cpp b11227 optimizes causal_attn toggling for faster vision inference

from English

The llama.cpp project released build b11227, which includes performance optimizations specifically targeting the handling of the `causal_attn` flag during multimodal processing.

The scheduler no longer re-reserves memory when toggling `causal_attn`, eliminating expensive passes that previously slowed down vision inputs for models like Gemma.
The qwen4exp indexer bias shape is now independent of `causal_attn`, preventing buffer reallocation failures under `GGML_SCHED_NO_REALLOC` constraints.
Benchmarks on llama-server with gemma-4-26B-A4B show significant latency reductions, such as a 7.61x speedup for 24 images on RTX 4090 and up to 3.40x on H200 with large context settings.

These changes improve inference speed for multi-image and video inputs without altering the generated output or graph rebuilding behavior.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11298 adds dflash support to convert and feature extraction

The llama.cpp project has released build b11298, which introduces support for the dflash format. This update enables both conversion of models to this format and subsequent feature extraction capabilities.

github llama.cpp · 7h ago · 2 views

llama.cpp b11284 fixes OpenVINO GET_ROWS on quantized weight views

The llama.cpp project released build b11284, which includes a fix for the OpenVINO backend to correctly handle GET_ROWS operations on quantized weight views.

github llama.cpp · 11h ago · 2 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 13h ago · 2 views

llama.cpp b11272 optimizes Hexagon concat op

The llama.cpp project released build b11272, which includes optimizations for the Hexagon backend. The primary change focuses on optimizing the concatenation operation to improve performance on Qualcomm Snapdragon hardware.

github llama.cpp · 1d ago · 7 views

llama.cpp b11260 adds FP32 GELU_ERF and GEGLU_ERF support

The llama.cpp project has released version b11260, introducing support for FP32 GELU_ERF and GEGLU_ERF activation functions in the Hexagon backend. This update also includes optimizations to reduce register pressure within the associated kernels.