github llama.cpp · 1d ago · inference

llama.cpp b10692 splits long rows in TOP_K SYCL kernel

from English

The llama.cpp project has released build b10692, which includes a modification to the SYCL implementation for the TOP_K operation. The change involves splitting long rows within the kernel rather than assigning one work-group per row.

This update is available across multiple platforms and hardware backends, including macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm 7.14, OpenVINO, and SYCL FP32/FP16), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, and ROCm 7.14), Android, and iOS.

The release provides binaries for various architectures such as x64, arm64, and s390x, along with an updated UI package.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 4h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 10h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.

github llama.cpp · 17h ago · 6 views

llama.cpp b10705 improves TENSOR_READ_LAZY handling

The llama.cpp project has released version b10705, which includes an improvement to the TENSOR_READ_LAZY handling mechanism. This update forces lazy tensors on CPU when the lazy setting is enabled.

github llama.cpp · 18h ago · 4 views

llama.cpp b10702 optimizes Q2_0 dot-product for AMD gfx1201

The llama.cpp project released version b10702, which includes a specific optimization for the Q2_0 quantization dot-product path on AMD's gfx1201 architecture. This update broadens the HIP implementation to use native amdgcn permutation instructions, improving performance for this specific hardware configuration.