github llama.cpp · 3d ago · inference

llama.cpp b11064 supports arbitrary hc in dsv4_hc_pre

from English

The llama.cpp project released build b11064, which modifies the metal backend to support arbitrary head counts (hc) in the dsv4_hc_pre kernel. Previously, these kernels hardcoded hc = 4, causing operations with other head counts to fall back to CPU execution.

The change passes n_hc as a function constant and loops over it in both pre kernels with direct loads.
Test-backend-ops cases were added for hc values of 1, 2, 3, 5, 8, and 65, covering both gated and not gated scenarios.
This update enables models like Kimi-K3 to run efficiently on GPU by supporting dynamic banked checkpoint counts in the cross-layer residual stack.

The release includes binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 1h ago · 11 views Live

llama.cpp b11120 hides Vulkan internal symbols to fix state destruction

The llama.cpp project released version b11120, which addresses a critical bug in the Vulkan backend by hiding internal symbols. This change prevents duplicate-dlopen state destruction issues that occurred when internal symbols were exported.

arxiv arXiv cs.AI · 4h ago · 9 views

CliffCompaction reduces coding agent costs by 50% while maintaining performance

Researchers introduce CliffCompaction, an autocompaction technique designed to reduce costs for long-horizon coding agents by up to 50% under a bounded context. The method maintains or improves performance on Terminal-Bench and achieves state-of-the-art results on KernelBench by keeping compacted information faithful through truncation rather than rephrasing.

arxiv arXiv cs.LG · 5h ago · 9 views

CliffCompaction reduces long-horizon coding agent costs by up to 50% while maintaining performance

Researchers introduce CliffCompaction, an autocompaction technique designed for agents handling millions of tokens, which cuts costs by up to 50% under bounded context. The method maintains or improves performance on Terminal-Bench and achieves state-of-the-art results on KernelBench by keeping information faithful through truncation rather than rephrasing.

github llama.cpp · 6h ago · 13 views

llama.cpp b11118 introduces direct-mapped DMA cache for HVX FA mask handling

The llama.cpp project has released build b11118, which includes a new hex-dma feature introducing a direct-mapped DMA cache better suited for Hexagon Vector eXtended (HVX) FA mask handling.