github llama.cpp · just now · inference

llama.cpp b10425 fuses gated-delta-net state writeback for +1.2% speedup

from English

The llama.cpp project has released version b10425, which includes a port of the SYCL optimization to fuse the gated-delta-net state writeback copy.

The change improves performance by approximately 1.2% on Arc Pro B70 hardware when running Qwen 3.6 27B with interleaved A/B passes.
Pre-fill latency (pp2048) remains flat, while token generation speed increases slightly.
Binaries are available for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

This release provides updated builds for various platforms and accelerators to support the latest model optimizations.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

arxiv arXiv cs.CL · 4h ago

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

Researchers propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along contraction dimensions without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off across language models ranging from 1B to 70B parameters.

github llama.cpp · 9h ago

Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference

llama.cpp b10418 adds SYCL host pinned memory support

llama.cpp OpenVINO backend adds Qwen3.5 support and memory optimizations

llama.cpp b10416 fixes stale UI cache on llama.app

llama.cpp b10413 adds auto-detection of draft spec type from GGUF metadata