github llama.cpp · 2d ago · inference

llama.cpp b11273 adds classifier_pooling support for ModernBERT rerankers

from English

The llama.cpp project released build b11273, which introduces support for the `classifier_pooling` method in reranking models, specifically targeting ModernBERT architectures.

The update modifies `load_hparams` to read the classifier pooling type from model configurations.
Conversion logic now restricts accepted `classifier_pooling` values to only 'cls' and 'mean'.
The internal parameter name `classifier_pooling_type` was renamed to `pooling_type_cls` for clarity.
ModernBERT models fall back to mean pooling if the specific type is not explicitly specified in the config.

This change enables users to correctly load and utilize ModernBERT rerankers within llama.cpp by respecting their specific pooling requirements.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media Hugging Face Forums · 4h ago

Proposal to retrieve LLM-native latent memory instead of text in RAG

A discussion on Hugging Face proposes replacing traditional text retrieval in RAG systems with LLM-native latent representations. The idea involves encoding documents into internal model states during indexing and retrieving those states for inference, potentially bypassing the cost of re-processing retrieved text.

github llama.cpp · 18h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 18h ago · 1 view

llama.cpp b11310 guards iq4_nl dequantize kernel against short rows

The llama.cpp project released version b11310, which includes a fix for the CUDA backend's IQ4_NL dequantization logic. The update addresses a memory safety issue where threads could read and write past the end of a row when processing blocks shorter than QK_K.

github llama.cpp · 21h ago · 1 view

llama.cpp b11307 preserves batch order for speculative decoding

The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.

github llama.cpp · 22h ago · 7 views

llama.cpp b11306 fixes graph reallocation abort by toggling causal_attn

The llama.cpp project released build b11306, which addresses a crash caused by graph shape changes during decoding. The update modifies the handling of the `causal_attn` flag to prevent reallocation failures under specific scheduling constraints.