github llama.cpp · 4d ago · inference

llama.cpp b11223 allows RANK pooling batch splitting for causal LLM rerankers

from English

The llama.cpp server now permits RANK pooling batch splitting for causal LLM rerankers like Qwen3 and Qwen3-VL, resolving previous limitations that broke long-document and multimodal reranking.

Previously, the server rejected RANK-pooling inputs larger than n_ubatch, forcing all tokens into a single physical batch.
The fix exposes llama_get_causal_attn and llama_model_is_causal to dynamically check attention types instead of hardcoding architecture checks.
can_split() now allows chunked prefill for RANK pooling when the context is causal, removing duplicated logic in the graph builder.

This change enables efficient processing of long documents and multimodal inputs by leveraging chunked prefill capabilities inherent to causal models.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 19h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 1d ago · 8 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 2d ago · 4 views

llama.cpp b11266 loads F32 A matrix 2 at a time for Vulkan on Intel

The llama.cpp project released build b11266, which includes a Vulkan optimization that loads F32 A matrices two elements at a time when they are 2-aligned. This change addresses performance issues on Intel hardware where loading floats one by one is inefficient, and it also corrects a missing validation for `a_offset` alignment in the original patch.

github llama.cpp · 2d ago · 1 view

llama.cpp b11265 fixes Vulkan MOE matmul tile selection

The llama.cpp b11265 release includes a fix for the Vulkan backend's handling of Mixture of Experts (MoE) models. The update corrects how `mat_mul_id` selects matrix multiplication tiles, addressing a performance issue where workers were left idle during dispatch.

github llama.cpp · 3d ago · 4 views

llama.cpp b11238 adds ggml_pad_ext for left padding and skips DFlash2 taps

The llama.cpp project released build b11238, introducing a unified left-padding implementation via the new `ggml_pad_ext` function. This change consolidates previous methods used by audio encoders like Parakeet, LFM2-Audio, Granite Speech, and Gemma 4, ensuring bit-identical embeddings for Gemma 4 while simplifying backend support.