media Hugging Face Forums · 2h ago · open_models

Hugging Face fixes DynamicCache to prevent wrong output in generate()

from English

A bug in Hugging Face's `generate()` function causes incorrect outputs when `past_key_values` is an externally-built `DynamicCache`. This issue affects KV cache compression methods like SnapKV, H2O, and KIVI that reconstruct caches from prefill data. The root cause is the failure to propagate positional context, leading to a 10.7pp accuracy loss on Qwen3-VL-8B.

Add `DynamicCache.from_kv_tensors()` classmethod to safely reconstruct caches with proper internal state.
Require explicit `attention_mask` and `position_ids` during generation to restore baseline quality.
The new method pre-allocates internal lists via an optional `num_hidden_layers` parameter for backward compatibility.

This fix ensures that KV compression workflows produce reliable outputs by correctly handling sequence length and attention masks.

Importance 1/3 Trust 1/3 Hugging Face Forums Hugging Face Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10181 disables MMQ on devices with less than 48 KiB shared memory

The llama.cpp project released build b10181, which includes a fix in ggml-cuda to disable Matrix Multiplication Quantization (MMQ) on GPUs with less than 48 KiB of per-block shared memory. Previously, MMQ configurations assumed a minimum of 48 KiB, causing fatal errors on devices like NVIDIA Pascal GPUs and Moore Threads MTT S70 where the tile size exceeded available memory.

github llama.cpp · 1d ago · 1 view

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

The llama.cpp project has released build b10165, which restores iq4_nl quantization support for the Flash Attention (FA) Vulkan backend. This change addresses issue #23681 by re-enabling the feature after concerns about shared memory usage were deemed negligible.

github llama.cpp · 1d ago

llama.cpp b10181 disables MMQ on devices with less than 48 KiB shared memory

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

llama.cpp b10164 adds chunked SSD matmul for Mamba-2 prefill acceleration

llama.cpp b10159 adds FWHT kernel for Metal backend

llama.cpp b10158 adds eagle3-v3 support for gpt-oss model