github llama.cpp · just now · inference

llama.cpp b10669 binds f16 KV cache in place for oneDNN SDPA path

from English

The llama.cpp project released version b10669, which includes a performance optimization for the oneDNN SDPA path by binding the f16 KV cache in place.

The change reduces memory traffic during prefill chunks; for example, on Qwen3.8 27B with a live KV length of 34816, it lowers traffic per ubatch from 4.56 GB to less by avoiding staged copies.
Binaries are available for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, OpenCL, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This update provides users with optimized inference performance on supported hardware while maintaining broad platform compatibility.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 6h ago · 1 view

llama.cpp b10666 fixes state save/load tests and on-device seq-copy

The llama.cpp b10666 release expands the test-save-load-state suite to run across all architectures and resolves several critical bugs in state management and graph handling. Key changes include fixing dangling references in minimax-01, aligning chunk copying for on-device sequences, and correcting indexer head counts for deepseek4.

github llama.cpp · 16h ago · 1 view

llama.cpp b10662 adds ctx-per-slot for unified KV cache

The llama.cpp project released build b10662, introducing a new `--kv-unified-per-slot` argument that allows users to specify context per slot for the unified KV cache. This change replaces the previous context fraction mechanism with context pool slots and refactors the underlying implementation.

github llama.cpp · 17h ago · 7 views

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.

github llama.cpp · 17h ago · 16 views

llama.cpp b10658 adds DFlash2 support and fixes mrope bug

The llama.cpp project released build b10658, introducing support for the DFlash2 specification which includes local convolution and a candidate selector. This update also addresses a specific mrope bug within the codebase.

github llama.cpp · 18h ago · 1 view

llama.cpp bundles HIP runtime DLLs with Windows ROCm release

The llama.cpp project has updated its Windows ROCm release to bundle the necessary HIP runtime DLLs directly alongside the binaries. This change addresses issue #26929 by ensuring the correct HIP runtime loads instead of potentially conflicting with the driver's copy in System32.