github llama.cpp · just now · inference

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

from English

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.

The matmul2d op now uses dynamic_extent for K instead of a static tile size.
Operand tensor views are clamped to the remaining valid K range on every iteration.
This change eliminates out-of-bounds reads that previously caused NaN values or data corruption.
Test cases were added to exercise the unaligned K path and verify the fix.

This fix ensures numerical stability for models with non-aligned matrix dimensions on Apple Silicon devices, preventing silent errors during inference.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 2h ago · 1 view

llama.cpp b10538 release with dequantized KV cache for large batches

The llama.cpp project has released version b10538, introducing a key optimization to the metal backend. This update enables dequantization of the key-value (KV) cache exclusively for large batch sizes.

github llama.cpp · 2h ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 7h ago · 1 view

llama.cpp b10519 refactors sleep handling to allow metrics access

The llama.cpp project released version b10519, which includes a significant refactor of the server's sleep handling mechanism. This change allows the /metrics endpoint to remain accessible while the server is in a sleeping state.

github llama.cpp · 10h ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.