github llama.cpp · 4h ago · inference

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

from English

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

The release includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This update provides users with updated builds across multiple platforms and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 12h ago · 1 view

llama.cpp b10989 enables AllReduce for ROCm and provides multi-platform binaries

The llama.cpp project has released version b10989, which introduces support for enabling AllReduce operations within the ROCm backend. This update is part of the ongoing development to expand hardware compatibility and distributed computing capabilities for large language model inference.

github llama.cpp · 16h ago · 1 view

llama.cpp b10988 adds OpenCL MoE expert matmul selection for speculative decoding

The llama.cpp project released build b10988, introducing changes to the OpenCL backend specifically for Mixture of Experts (MoE) models. The update implements logic to choose the MoE expert matrix multiplication based on batch size during speculative decoding and multi-token prediction.

github llama.cpp · 21h ago · 1 view

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

github llama.cpp · 21h ago · 1 view

llama.cpp b10985 restricts RPC hash-cache to weights

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.

github llama.cpp · 1d ago · 4 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.