github llama.cpp · 2d ago · inference

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

from English

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.

Fixes gguf_init_from_file_ptr to pad the data section from the GGUF start, preventing errors with unaligned embedded files.
Adds llama_adapter_lora_init_from_file_ptr to enable loading LoRA adapters via FILE* streams.
Disables mmap with a warning when an embedded data section is not aligned, replacing previous assertions in ggml.
Includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, CUDA, ROCm, OpenVINO, SYCL), Android, Windows, and openEuler.

This release ensures reliable operation of GGUF files embedded within other containers or at arbitrary file offsets.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

github llama.cpp · 2d ago · 7 views

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.

github llama.cpp · 4d ago · 10 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.

github llama.cpp · 11d ago · 24 views

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).

github llama.cpp · 12d ago · 23 views

llama.cpp b10827 fixes OpenCL weights pack selection for q4_K and q5_K mul_mat

The llama.cpp project has released version b10827, which includes a fix to properly choose the weights pack for q4_K and q5_K matrix multiplication operations using OpenCL.