github llama.cpp · 1h ago · inference

llama.cpp b10538 release with dequantized KV cache for large batches

from English

The llama.cpp project has released version b10538, introducing a key optimization to the metal backend. This update enables dequantization of the key-value (KV) cache exclusively for large batch sizes.

The primary change is in the metal backend, which now applies dequantized KV caching only when processing large batches.
Binaries are available for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.
KleidiAI support on macOS Apple Silicon and ROCm 7.14 on Ubuntu are disabled in this release.

This optimization aims to improve memory efficiency or performance for workloads involving large batch sizes on compatible hardware.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.

github llama.cpp · 2h ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 6h ago · 1 view

llama.cpp b10519 refactors sleep handling to allow metrics access

The llama.cpp project released version b10519, which includes a significant refactor of the server's sleep handling mechanism. This change allows the /metrics endpoint to remain accessible while the server is in a sleeping state.

github llama.cpp · 9h ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

media r/LocalLLaMA · 14h ago · 2 views

G9v3-39A5B MoE model shows strong results on artificialanalysis

The G9v3-39A5B model is drawing attention for its performance on artificialanalysis, where it reportedly ranks above Qwen 3.6 27B.