github llama.cpp · just now · inference

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

from English

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

Added CUDA support for row-contiguous SUM_ROWS.
Organized code to add GGML_OP_MEAN for row-contiguous tensors using a shared kernel.
Included tests for MEAN permute and slice operations.
Provided binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

The release ensures compatibility with various hardware accelerators and operating systems for local inference.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10985 restricts RPC hash-cache to weights

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.

github llama.cpp · 3h ago · 2 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 4h ago · 1 view

llama.cpp adds Metal FA kernels for MiniCPM3's HSK=96, HSV=64

The llama.cpp project has added Flash Attention (FA) kernel support for the specific configuration used by MiniCPM3 on Apple Silicon. This change addresses a crash where the `-fa auto` flag would abort due to a missing kernel instantiation for attention key length 96 and value length 64.

github llama.cpp · 18h ago · 3 views

llama.cpp b10970 release uses fp32 accumulation on MFMA devices

The llama.cpp project has released version b10970, which includes a technical update to the fattn-mma backend. This change implements the use of fp32 accumulators specifically for CDNA architecture devices.

github llama.cpp · 1d ago · 1 view

llama.cpp b10956 adds radix select for large k in SYCL top_k

The llama.cpp b10956 release introduces a radix select algorithm for the SYCL backend to handle large k values in the TOP_K operation, which previously fell back to CPU for k > 32. This change allows GPU-resident processing of large k by using histogram-based passes that keep shared local memory footprint independent of k.