github llama.cpp · 17h ago · inference

llama.cpp b10705 improves TENSOR_READ_LAZY handling

from English

The llama.cpp project has released version b10705, which includes an improvement to the TENSOR_READ_LAZY handling mechanism. This update forces lazy tensors on CPU when the lazy setting is enabled.

Improved TENSOR_READ_LAZY handling in llama.cpp
Forced lazy tensor behavior on CPU when lazy mode is active

This release provides updated binaries for macOS, Linux, Windows, Android, and iOS across various hardware backends including CPU, CUDA, ROCm, Vulkan, and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 4h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 10h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.

github llama.cpp · 18h ago · 4 views

llama.cpp b10702 optimizes Q2_0 dot-product for AMD gfx1201

The llama.cpp project released version b10702, which includes a specific optimization for the Q2_0 quantization dot-product path on AMD's gfx1201 architecture. This update broadens the HIP implementation to use native amdgcn permutation instructions, improving performance for this specific hardware configuration.

github llama.cpp · 1d ago · 6 views

llama.cpp b10691 fixes Metal null-pipeline crash for F16 mul_mat

The llama.cpp project released build b10691, which addresses a critical stability issue in the Metal backend. The update specifically resolves a null-pipeline crash that occurred during F16 source multiplication operations.