github llama.cpp · 3d ago · inference

llama.cpp b11214 enables fattn-mma kernel on CDNA for large batch sizes

from English

The llama.cpp project has released version b11214, which includes a key update to the HIP backend. This release enables the fattn-mma kernel on CDNA architectures when dkq is greater than 256, specifically targeting performance improvements for large batch sizes.

The HIP backend now supports the fattn-mma kernel for dkq > 256 on CDNA hardware.
CI checks for hip-quality-check have been updated to ignore spills for very large mfma mma kernels.
Binaries are provided for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, CUDA, ROCm, OpenVINO, SYCL, Snapdragon), Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, ROCm), Android, and openEuler.

This update allows users running llama.cpp on AMD CDNA GPUs to utilize optimized attention kernels for larger batch sizes.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 11h ago · 2 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 20h ago · 2 views

llama.cpp b11266 loads F32 A matrix 2 at a time for Vulkan on Intel

The llama.cpp project released build b11266, which includes a Vulkan optimization that loads F32 A matrices two elements at a time when they are 2-aligned. This change addresses performance issues on Intel hardware where loading floats one by one is inefficient, and it also corrects a missing validation for `a_offset` alignment in the original patch.

github llama.cpp · 22h ago · 1 view

llama.cpp b11265 fixes Vulkan MOE matmul tile selection

The llama.cpp b11265 release includes a fix for the Vulkan backend's handling of Mixture of Experts (MoE) models. The update corrects how `mat_mul_id` selects matrix multiplication tiles, addressing a performance issue where workers were left idle during dispatch.

github llama.cpp · 2d ago · 4 views

llama.cpp b11238 adds ggml_pad_ext for left padding and skips DFlash2 taps

The llama.cpp project released build b11238, introducing a unified left-padding implementation via the new `ggml_pad_ext` function. This change consolidates previous methods used by audio encoders like Parakeet, LFM2-Audio, Granite Speech, and Gemma 4, ensuring bit-identical embeddings for Gemma 4 while simplifying backend support.

github llama.cpp · 2d ago · 5 views

llama.cpp b11232 enables tiled flash attention on x86

The llama.cpp project released build b11232, which includes updates to the ggml-cpu backend. The release focuses on enabling tiled flash attention for non-vector-multiple head dimensions on x86 architectures.