github llama.cpp · 1h ago · inference

llama.cpp b10430 allows virtual igpu devices

from English

The llama.cpp project has released version b10430, which introduces support for virtual integrated GPU (igpu) devices. This update also includes improved comments within the codebase.

The release provides binaries and frameworks for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu with CPU, Vulkan, OpenVINO, SYCL, and disabled ROCm), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, and ROCm 7.14), and openEuler architectures.

This update enables users to run llama.cpp on systems utilizing virtual igpu configurations across a wide range of operating systems and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10429 allows accessing /metrics and /slots during llama_decode()

The llama.cpp project released version b10429, which modifies the server to allow accessing the /metrics and /slots endpoints while llama_decode() is executing.

github llama.cpp · 6h ago

llama.cpp b10427 fuses SYCL FFN ops for up to 12.4% throughput gain

The llama.cpp project released version b10427, which includes a performance optimization for Intel Arc GPUs via SYCL. The update implements a fusion of the gate, up, and GLU operations within the q4_K dense feed-forward network (FFN) layer.

github llama.cpp · 7h ago · 1 view

llama.cpp b10425 fuses gated-delta-net state writeback for +1.2% speedup

The llama.cpp project has released version b10425, which includes a port of the SYCL optimization to fuse the gated-delta-net state writeback copy.

arxiv arXiv cs.CL · 10h ago

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Researchers propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along contraction dimensions without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off across language models ranging from 1B to 70B parameters.

github llama.cpp · 15h ago

llama.cpp b10418 adds SYCL host pinned memory support

The llama.cpp project released version b10418, introducing support for host pinned memory to improve SYCL Host-to-Device memory access. This update addresses thread-safety issues within the backend implementation.