github llama.cpp · just now · inference

llama.cpp b10723 tunes OpenCL quant paths for Intel Xe-LP GPUs

from English

The llama.cpp project released version b10723, which includes specific optimizations for OpenCL on Intel Xe-LP graphics hardware. The update focuses on improving tensor generation (TG) and prefill (PP) performance by adjusting quantization kernels.

Tuned Q4_K and Q5_K mul_mv operations to increase N_DST values, enabling 2x activation reuse on Intel devices.
Implemented 8x8 tile configurations for both Q4_K and Q5_K mul_mm operations specifically for Intel architectures.
Increased N_DST from 8 to 16 for Q4_K mul_mv to further optimize data handling.

This release provides updated binaries for macOS, Linux, Windows, Android, and iOS across various backends including CUDA, Vulkan, ROCm, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 3h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 5h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 11h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.

github llama.cpp · 18h ago · 6 views

llama.cpp b10705 improves TENSOR_READ_LAZY handling

The llama.cpp project has released version b10705, which includes an improvement to the TENSOR_READ_LAZY handling mechanism. This update forces lazy tensors on CPU when the lazy setting is enabled.

github llama.cpp · 19h ago · 4 views

llama.cpp b10702 optimizes Q2_0 dot-product for AMD gfx1201

The llama.cpp project released version b10702, which includes a specific optimization for the Q2_0 quantization dot-product path on AMD's gfx1201 architecture. This update broadens the HIP implementation to use native amdgcn permutation instructions, improving performance for this specific hardware configuration.