github llama.cpp · 1h ago · inference

llama.cpp b10726 speeds up large batch IQ model prompt processing with AVX2

from English

The llama.cpp project released version b10726, which introduces significant performance optimizations for processing large batch sizes in IQ quantized models. The primary update leverages AVX2 instructions to accelerate grid IQ quants through batched GEMM operations.

Batched GEMM implementation for grid IQ quants
Vectorization of IQ panel decode with a lowered threshold for speedup
Introduction of the ggml_gemm_iqp_8x8_q8_K_p4 kernel and removal of the gather buffer
Single-source gather layout, gate bias, and vectorized interleave for IQ panels
Addition of NUMA fallback support and 10-row batch tests for IQP coverage

This release provides faster inference capabilities for users running IQ models on compatible hardware.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10723 tunes OpenCL quant paths for Intel Xe-LP GPUs

The llama.cpp project released version b10723, which includes specific optimizations for OpenCL on Intel Xe-LP graphics hardware. The update focuses on improving tensor generation (TG) and prefill (PP) performance by adjusting quantization kernels.

github llama.cpp · 6h ago · 1 view

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 8h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 14h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.

github llama.cpp · 21h ago · 6 views

llama.cpp b10705 improves TENSOR_READ_LAZY handling

The llama.cpp project has released version b10705, which includes an improvement to the TENSOR_READ_LAZY handling mechanism. This update forces lazy tensors on CPU when the lazy setting is enabled.