github llama.cpp · 1h ago · inference

llama.cpp b10429 allows accessing /metrics and /slots during llama_decode()

from English

The llama.cpp project released version b10429, which modifies the server to allow accessing the /metrics and /slots endpoints while llama_decode() is executing.

The change updates server_queue::worker to call llama_decode inside yield_to_queue.
It also handles process_mtmd_chunk during this operation.
Binaries are available for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This update improves server responsiveness by preventing endpoint blocking during model inference.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago Live

llama.cpp b10430 allows virtual igpu devices

The llama.cpp project has released version b10430, which introduces support for virtual integrated GPU (igpu) devices. This update also includes improved comments within the codebase.

github llama.cpp · 6h ago

llama.cpp b10427 fuses SYCL FFN ops for up to 12.4% throughput gain

The llama.cpp project released version b10427, which includes a performance optimization for Intel Arc GPUs via SYCL. The update implements a fusion of the gate, up, and GLU operations within the q4_K dense feed-forward network (FFN) layer.

github llama.cpp · 8h ago · 1 view

llama.cpp b10425 fuses gated-delta-net state writeback for +1.2% speedup

The llama.cpp project has released version b10425, which includes a port of the SYCL optimization to fuse the gated-delta-net state writeback copy.

arxiv arXiv cs.CL · 11h ago

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Researchers propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along contraction dimensions without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off across language models ranging from 1B to 70B parameters.

github llama.cpp · 16h ago

llama.cpp b10418 adds SYCL host pinned memory support

The llama.cpp project released version b10418, introducing support for host pinned memory to improve SYCL Host-to-Device memory access. This update addresses thread-safety issues within the backend implementation.