github llama.cpp · 1h ago · inference

llama.cpp b10985 restricts RPC hash-cache to weights

from English

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.

The `ggml_backend_rpc_buffer_set_tensor` and `ggml_backend_rpc_set_tensor_async` functions now only hash transfers for buffers marked with `GGML_BACKEND_BUFFER_USAGE_WEIGHTS`.
A new `cache_flag` byte in the SET_TENSOR message signals when a cache entry should be saved, replacing the previous server-side pending state logic.
The RPC protocol version is bumped to `RPC_PROTO_MAJOR_VERSION` to reflect the wire format change.

This optimization prevents compute data from filling up disk space by ensuring only weights are cached via the hash path.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

github llama.cpp · 3h ago · 2 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 4h ago · 1 view

llama.cpp adds Metal FA kernels for MiniCPM3's HSK=96, HSV=64

The llama.cpp project has added Flash Attention (FA) kernel support for the specific configuration used by MiniCPM3 on Apple Silicon. This change addresses a crash where the `-fa auto` flag would abort due to a missing kernel instantiation for attention key length 96 and value length 64.

github llama.cpp · 18h ago · 3 views

llama.cpp b10970 release uses fp32 accumulation on MFMA devices

The llama.cpp project has released version b10970, which includes a technical update to the fattn-mma backend. This change implements the use of fp32 accumulators specifically for CDNA architecture devices.

github llama.cpp · 1d ago · 1 view

llama.cpp b10956 adds radix select for large k in SYCL top_k

The llama.cpp b10956 release introduces a radix select algorithm for the SYCL backend to handle large k values in the TOP_K operation, which previously fell back to CPU for k > 32. This change allows GPU-resident processing of large k by using histogram-based passes that keep shared local memory footprint independent of k.