github llama.cpp · 1h ago · inference

llama.cpp b10232 implements DeepSeek V4 hyper-connections in Metal

from English

The llama.cpp project released build b10232, which introduces support for DeepSeek V4 hyper-connections via the Metal backend. This update adds implementation of GGML_OP_DSV4_HC_COMB, GGML_OP_DSV4_HC_PRE, and GGML_OP_DSV4_HC_POST operations.

The new kernels utilize SIMDgroup register and shuffle optimizations.
Metal dispatch plumbing was added to support production Sinkhorn iteration counts and embedding widths.
Pre-built binaries are available for macOS (Apple Silicon and Intel), iOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, Vulkan, and OpenVINO.

This release enables efficient inference of DeepSeek V4 models on Apple Silicon devices using the Metal framework.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1d ago · 7 views

llama.cpp b10219 persists reasoning_content in chat history

The llama.cpp project released version b10219, which includes a fix for the CLI to persist reasoning content within the chat history. Previously, while `llama-cli` collected reasoning from the stream for display, it only stored assistant content in messages, preventing the `--reasoning-preserve` flag from re-injecting prior thoughts on later turns.

github llama.cpp · 2d ago · 8 views

llama.cpp b10213 adds support for rotated kv cache quantization

The llama.cpp project has released version b10213, introducing support for rotated key-value cache quantization. This update is available across a wide range of platforms and hardware accelerators.

github llama.cpp · 2d ago · 3 views

llama.cpp b10212 loads MTP tensors only when used

The llama.cpp project released version b10212, which includes a change to load Multi-Token Prediction (MTP) tensors only if they are actually used. This update ensures that MTP tensors are skipped during loading for remaining models that support the feature but do not require it.

github llama.cpp · 2d ago · 3 views

llama.cpp b10210 corrects accepted tokens during draft token replay

The llama.cpp project has released version b10210, which includes a correction to the server logic for handling accepted tokens when draft token replay is required.

github llama.cpp · 2d ago · 6 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.