github llama.cpp · 1h ago · inference

llama.cpp b10505 adds dedup-cache-models preset option

from English

The llama.cpp project has released version b10505, which introduces a new `dedup-cache-models` preset option for the server component.

The release includes binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x, Vulkan, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.
macOS Apple Silicon KleidiAI support is disabled in this build.
Ubuntu x64 ROCm 7.14 support is also disabled.
A new UI package is available for download alongside the platform-specific binaries.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

media Hugging Face Forums · 1d ago · 3 views

Fourier Magnitude KV Cache Quantization doubles context length at same memory

ntrillard demonstrates that quantizing the Fourier magnitude spectrum of K and V cache values to 4-bit while preserving phase at full precision achieves 96.9% token match with fp16 reference on Gemma-3-1B, a 5-10x improvement over standard min-max quantization.

github llama.cpp · 4d ago · 26 views

llama.cpp adds Kimi-K3 model support with MXFP4 repacking and chat template

The llama.cpp project has added support for the Kimi-K3 text model, implementing its hybrid KDA (linear) + MLA (full) attention architecture along with five specific architectural features not present in the older Kimi-Linear-48B.

github llama.cpp · 4d ago · 21 views

llama.cpp b10447 redesigns yield_to_queue thread model

The llama.cpp project has released build b10447, which includes a re-design of the `yield_to_queue` thread model. This change involves running `common_speculative_process` in the worker and swapping the worker-to-main thread design.

github llama.cpp · 4d ago · 21 views

llama.cpp b10444 adds MTP assistant model support and drops eagle3

The llama.cpp project has released version b10444, introducing support for loading Multi-Token Prediction (MTP) assistant models via the `--models-dir` flag. This update also includes strict prefix checking for MTP models in presets and removes support for the eagle3 draft type.

github llama.cpp · 5d ago · 19 views

llama.cpp b10442 adds Vulkan SHMEM_STRIDE_PAD for Intel Xe and fixes OOB read

The llama.cpp project released version b10442, introducing specific Vulkan optimizations for Intel Xe hardware while addressing memory safety issues in matrix multiplication routines.