github llama.cpp · 1h ago · inference

llama.cpp b11454 adds K2 Horizon dense and MoVA support

from English

The llama.cpp project released build b11454, which introduces support for the K2 Horizon model architecture, including both dense and Mixture-of-Value Attention (MoVA) variants.

Added K2 Horizon GGUF conversion code, tensor loading, compute graph implementation, and tokenizer registration.
Fixed Unicode regex splitting to correctly handle K2-Horizon's specific requirements on Windows and other platforms.
Implemented Jinja template support for sequence indices in selectattr and rejectattr filters.
Enabled chat support for K2 Horizon reasoning and tool calls, including response schema enforcement and YaRN beta metadata loading.

This update allows users to run K2 Horizon models locally using llama.cpp across various backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1d ago · 22 views

llama.cpp b11433 adds Hexagon pool op support

The llama.cpp project released version b11433, which introduces support for pooling operations on the Hexagon NPU. This update includes implementations for both 2D and 1D pooling layers.

github llama.cpp · 1d ago · 25 views

llama.cpp b11424 fixes Vulkan Flash Attention shmem write out of bounds

The llama.cpp project has released version b11424, which includes a fix for a shared memory write-out-of-bounds error in the Vulkan backend when using Flash Attention.

github llama.cpp · 2d ago · 21 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 2d ago · 25 views

llama.cpp b11405 tiles the lightning indexer over keys and tokens for 4 heads

The llama.cpp b11405 release updates the CUDA backend by tiling the lightning indexer over keys and tokens for 4 heads, addressing performance issues with small head counts.

github llama.cpp · 2d ago · 33 views

llama.cpp b11403 uses MMVF for thin f16/bf16 mul_mat at small batch size

The llama.cpp project released version b11403, which includes a CUDA optimization to use the MMVF kernel for thin f16 and bf16 matrix multiplications at small batch sizes.