github llama.cpp · 17h ago · inference

llama.cpp b11310 guards iq4_nl dequantize kernel against short rows

from English

The llama.cpp project released version b11310, which includes a fix for the CUDA backend's IQ4_NL dequantization logic. The update addresses a memory safety issue where threads could read and write past the end of a row when processing blocks shorter than QK_K.

Guards the iq4_nl dequantize row kernel against short rows by skipping sub-blocks that start at or past the row length.
Provides pre-built binaries for macOS (Apple Silicon and Intel), Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon backends.

This fix prevents potential out-of-bounds memory access during inference with IQ4_NL quantized models on CUDA devices.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media Hugging Face Forums · 3h ago

Proposal to retrieve LLM-native latent memory instead of text in RAG

A discussion on Hugging Face proposes replacing traditional text retrieval in RAG systems with LLM-native latent representations. The idea involves encoding documents into internal model states during indexing and retrieving those states for inference, potentially bypassing the cost of re-processing retrieved text.

github llama.cpp · 17h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 20h ago · 1 view

llama.cpp b11307 preserves batch order for speculative decoding

The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.

github llama.cpp · 21h ago · 7 views

llama.cpp b11306 fixes graph reallocation abort by toggling causal_attn

The llama.cpp project released build b11306, which addresses a crash caused by graph shape changes during decoding. The update modifies the handling of the `causal_attn` flag to prevent reallocation failures under specific scheduling constraints.

github llama.cpp · 22h ago · 1 view

llama.cpp b11301 fixes integer overflow in ggml/gguf

The llama.cpp project has released build b11301, which includes a fix for an integer overflow issue within the ggml and gguf components. This update addresses validation logic to prevent potential errors when handling zero-element tensors.