github llama.cpp · 1h ago · inference

llama.cpp b11001 fixes int16 transpose corruption on RISC-V

from English

The llama.cpp project released build b11001, which includes a critical fix for the `permute_transpose_impl` function handling int16 data on RISC-V architectures.

The previous version incorrectly called `rvv_transposed_s32_mn_to_nm` for 16-bit integers, causing corrupted transposition results where 14 out of 16 positions were wrong in a 4x4 matrix.
The fix replaces the erroneous call with the correct `rvv_transposed_s16_mn_to_nm` function, which was already available and used elsewhere in flash attention.
Binaries are provided for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, CUDA 12/13, ROCm 10.0, OpenVINO, SYCL), Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 10.0), Android, and openEuler.

This update ensures correct data transposition for int16 operations on RISC-V hardware, preventing calculation errors in affected models.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 7h ago · 1 view

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

github llama.cpp · 15h ago · 2 views

llama.cpp b10989 enables AllReduce for ROCm and provides multi-platform binaries

The llama.cpp project has released version b10989, which introduces support for enabling AllReduce operations within the ROCm backend. This update is part of the ongoing development to expand hardware compatibility and distributed computing capabilities for large language model inference.

github llama.cpp · 20h ago · 1 view

llama.cpp b10988 adds OpenCL MoE expert matmul selection for speculative decoding

The llama.cpp project released build b10988, introducing changes to the OpenCL backend specifically for Mixture of Experts (MoE) models. The update implements logic to choose the MoE expert matrix multiplication based on batch size during speculative decoding and multi-token prediction.

github llama.cpp · 1d ago · 1 view

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

github llama.cpp · 1d ago · 1 view

llama.cpp b10985 restricts RPC hash-cache to weights

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.