github llama.cpp · 5h ago · inference

llama.cpp adds Metal FA kernels for MiniCPM3's HSK=96, HSV=64

from English

The llama.cpp project has added Flash Attention (FA) kernel support for the specific configuration used by MiniCPM3 on Apple Silicon. This change addresses a crash where the `-fa auto` flag would abort due to a missing kernel instantiation for attention key length 96 and value length 64.

Added tile kernels at (96, 64) for every K/V type that already supports (96, 96).
Added vec kernels for NE=4 configurations, as this is the only value where NL divides both DK/4 and DV/4.
Updated tests to avoid redundant FA vec slice coverage.

This update enables MiniCPM3 models to run with Flash Attention enabled on macOS and iOS devices without encountering kernel errors.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10985 restricts RPC hash-cache to weights

The llama.cpp project released build b10985, which modifies the RPC backend to limit hash-based caching exclusively to weight tensors. This change addresses an issue where activation data was unnecessarily hashed and stored, leading to excessive disk usage.

github llama.cpp · 3h ago · 2 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 18h ago · 3 views

llama.cpp b10970 release uses fp32 accumulation on MFMA devices

The llama.cpp project has released version b10970, which includes a technical update to the fattn-mma backend. This change implements the use of fp32 accumulators specifically for CDNA architecture devices.

github llama.cpp · 1d ago · 1 view

llama.cpp b10956 adds radix select for large k in SYCL top_k

The llama.cpp b10956 release introduces a radix select algorithm for the SYCL backend to handle large k values in the TOP_K operation, which previously fell back to CPU for k > 32. This change allows GPU-resident processing of large k by using histogram-based passes that keep shared local memory footprint independent of k.