github llama.cpp · just now · inference

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

from English

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

Added `flash_attn_f32_f16_bin` kernel implementation for OpenCL.
Implemented guarded prefill logic for Flash Attention in the OpenCL backend.
Provided binaries for macOS (Apple Silicon and Intel), Linux (CPU, CUDA, ROCm, Vulkan, SYCL, OpenVINO), Windows (CPU, CUDA, Vulkan, SYCL, ROCm, OpenVINO), Android, and iOS.

This release expands hardware compatibility for attention mechanisms on OpenCL-capable devices while maintaining broad support across major operating systems and accelerators.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 2d ago · 4 views

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.

github llama.cpp · 2d ago · 7 views

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.

github llama.cpp · 4d ago · 10 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 11d ago · 24 views

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).

github llama.cpp · 12d ago · 23 views

llama.cpp b10827 fixes OpenCL weights pack selection for q4_K and q5_K mul_mat

The llama.cpp project has released version b10827, which includes a fix to properly choose the weights pack for q4_K and q5_K matrix multiplication operations using OpenCL.