github llama.cpp · 1d ago · inference

llama.cpp b11095 adds HMX-optimized GATED_DELTA_NET and reduces DDR reads

from English

The llama.cpp b11095 release introduces a new HMX-optimized GATED_DELTA_NET (GDN) implementation, alongside various optimizations for Hexagon and Flash Attention kernels.

The GDN support includes pipelining, HVX threading, and vectorized operations to improve performance on Qualcomm chips.
Flash Attention updates reduce DDR reads by 20-30% during token generation through in-register accumulation and DMA pipeline improvements.
The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, Vulkan, and OpenVINO backends.

This update enhances inference efficiency on specific hardware architectures while maintaining broad compatibility with existing deployment environments.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 1h ago · 11 views Live

llama.cpp b11120 hides Vulkan internal symbols to fix state destruction

The llama.cpp project released version b11120, which addresses a critical bug in the Vulkan backend by hiding internal symbols. This change prevents duplicate-dlopen state destruction issues that occurred when internal symbols were exported.

arxiv arXiv cs.AI · 4h ago · 9 views

CliffCompaction reduces coding agent costs by 50% while maintaining performance

Researchers introduce CliffCompaction, an autocompaction technique designed to reduce costs for long-horizon coding agents by up to 50% under a bounded context. The method maintains or improves performance on Terminal-Bench and achieves state-of-the-art results on KernelBench by keeping compacted information faithful through truncation rather than rephrasing.

arxiv arXiv cs.LG · 5h ago · 9 views

CliffCompaction reduces long-horizon coding agent costs by up to 50% while maintaining performance

Researchers introduce CliffCompaction, an autocompaction technique designed for agents handling millions of tokens, which cuts costs by up to 50% under bounded context. The method maintains or improves performance on Terminal-Bench and achieves state-of-the-art results on KernelBench by keeping information faithful through truncation rather than rephrasing.

github llama.cpp · 6h ago · 13 views

llama.cpp b11118 introduces direct-mapped DMA cache for HVX FA mask handling

The llama.cpp project has released build b11118, which includes a new hex-dma feature introducing a direct-mapped DMA cache better suited for Hexagon Vector eXtended (HVX) FA mask handling.