github llama.cpp · just now · inference

llama.cpp b11460 adds IQ3_S multi-column MMVQ to SYCL

from English

The llama.cpp project has released version b11460, which introduces support for the IQ3_S multi-column MMVQ quantization format within its SYCL backend. This update is included in the new build alongside standard CPU and GPU acceleration options.

The release adds IQ3_S multi-column MMVQ support to the SYCL implementation via pull request #29500.
Binaries are provided for macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x), Windows, Android, and openEuler.
GPU acceleration is available via CUDA 12/13, Vulkan, ROCm 10.0, OpenVINO, and OpenCL Adreno across supported platforms.
Specific builds are included for Snapdragon devices on Linux and Android, supporting CPU, Adreno GPU, and Hexagon NPU.

This release allows users to access the latest SYCL optimizations and broadens hardware compatibility for running llama.cpp models on diverse architectures.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 9h ago · 6 views

llama.cpp b11454 adds K2 Horizon dense and MoVA support

The llama.cpp project released build b11454, which introduces support for the K2 Horizon model architecture, including both dense and Mixture-of-Value Attention (MoVA) variants.

github llama.cpp · 1d ago · 25 views

llama.cpp b11433 adds Hexagon pool op support

The llama.cpp project released version b11433, which introduces support for pooling operations on the Hexagon NPU. This update includes implementations for both 2D and 1D pooling layers.

github llama.cpp · 2d ago · 25 views

llama.cpp b11424 fixes Vulkan Flash Attention shmem write out of bounds

The llama.cpp project has released version b11424, which includes a fix for a shared memory write-out-of-bounds error in the Vulkan backend when using Flash Attention.

github llama.cpp · 2d ago · 22 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 2d ago · 29 views

llama.cpp b11405 tiles the lightning indexer over keys and tokens for 4 heads

The llama.cpp b11405 release updates the CUDA backend by tiling the lightning indexer over keys and tokens for 4 heads, addressing performance issues with small head counts.