github llama.cpp · 12h ago · inference

llama.cpp b10989 enables AllReduce for ROCm and provides multi-platform binaries

from English

The llama.cpp project has released version b10989, which introduces support for enabling AllReduce operations within the ROCm backend. This update is part of the ongoing development to expand hardware compatibility and distributed computing capabilities for large language model inference.

The release includes pre-built binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x), Windows, and Android.
GPU acceleration options are expanded to include CUDA 12.8/13.3, Vulkan, OpenVINO, SYCL FP32/FP16, and ROCm 10.0 across supported platforms.
Specific builds are provided for openEuler x86 and aarch64 architectures, including support for ACL Graph on 910b processors.
A standalone UI binary is also available alongside the core library releases.

This release allows users to leverage ROCm-based AMD GPUs with AllReduce functionality while providing updated binaries for various operating systems and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 4h ago · 1 view

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

github llama.cpp · 16h ago · 1 view

llama.cpp b10988 adds OpenCL MoE expert matmul selection for speculative decoding

The llama.cpp project released build b10988, introducing changes to the OpenCL backend specifically for Mixture of Experts (MoE) models. The update implements logic to choose the MoE expert matrix multiplication based on batch size during speculative decoding and multi-token prediction.

github llama.cpp · 21h ago · 1 view

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

github llama.cpp · 21h ago · 1 view

llama.cpp b10985 restricts RPC hash-cache to weights

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.

github llama.cpp · 1d ago · 4 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.