github llama.cpp · 1h ago · inference

llama.cpp b10210 corrects accepted tokens during draft token replay

from English

The llama.cpp project has released version b10210, which includes a correction to the server logic for handling accepted tokens when draft token replay is required.

Corrects accepted tokens in the server component when draft token replay is needed.
Provides pre-built binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64/arm64/s390x with CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.
Includes a UI release package.

This update ensures accurate token handling in speculative decoding scenarios across all supported platforms.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1h ago Live

llama.cpp b10212 loads MTP tensors only when used

The llama.cpp project released version b10212, which includes a change to load Multi-Token Prediction (MTP) tensors only if they are actually used. This update ensures that MTP tensors are skipped during loading for remaining models that support the feature but do not require it.

github llama.cpp · 2h ago

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 4h ago

llama.cpp b10212 loads MTP tensors only when used

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

llama.cpp b10207 adds SYCL support for missed types in cpy

llama.cpp b10206 enforces same K and V cache types for DeepSeek V4 and MLA models

llama.cpp b10202 release with SYCL RMS_NORM fusion