github llama.cpp · 1h ago · inference

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

from English

The llama.cpp project has released version b10282, which introduces new monitoring capabilities for speculative decoding. The server now includes spec-decode counters in its /metrics endpoint, allowing users to track performance metrics related to this feature.

Added spec-decode counters to the /metrics endpoint for better observability.
Aligned parameter names exactly with vLLM following review feedback.
Provided binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

This update improves the ability to monitor speculative decoding performance while maintaining compatibility with vLLM parameter naming conventions.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1d ago

llama.cpp b10259 allows tensor reshape during load

The llama.cpp project has released build b10259, which introduces the ability to allow the reshape of tensors during the loading process.

github llama.cpp · 1d ago · 1 view

llama.cpp b10254 adds DeepSeek V4 Flash 0731 template and updates V4 templates

The llama.cpp project released version b10254, introducing a new chat template for the DeepSeek V4 Flash 0731 model alongside updates to existing DeepSeek V4 templates.

github llama.cpp · 2d ago

llama.cpp b10251 adds MTP support for GLM-4.7-Flash

The llama.cpp project has released build b10251, which introduces Multi-Token Prediction (MTP) support for the GLM-4.7-Flash model.

github llama.cpp · 2d ago

llama.cpp b10238 adds MTP support for Qwen3-Next

The llama.cpp project released build b10238, which introduces Multi-Token Prediction (MTP) support for the Qwen3-Next model family. This update includes specific implementation changes to handle MTP layers and fixes related to Python type checking within the codebase.

github llama.cpp · 2d ago · 5 views

llama.cpp b10237 adds MTP support for DeepSeek V3.2

The llama.cpp project released build b10237, which introduces Multi-Token Prediction (MTP) support for the DeepSeek V3.2 model.