github llama.cpp · just now · inference

llama.cpp b10730 optimizes Qwen4exp prompt processing by summing indexer heads by slices

from English

The llama.cpp project released build b10730, which includes optimizations for the Qwen4exp model architecture. The primary change involves summing indexer heads by slices instead of using a transpose and sum_rows operation, reducing memory copy overhead.

Prompt processing speed increased from 2170 to 2366 tokens per second on an RTX PRO 6000 with Qwen3.8-Flash-Next UD-Q4_K_XL.
The optimization removes redundant contiguous tensor requirements for the indexer query, as rope returns a freshly allocated contiguous tensor.
Generation speed remains unaffected, but the gain in prompt processing scales with context length and ubatch size.
Greedy output is unchanged token for token compared to previous versions.

This update improves efficiency for long-context inference tasks involving Qwen4exp models by reducing computational waste during the prompt processing phase.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 9h ago · 3 views

llama.cpp b10727 adds concat support for quantized types

The llama.cpp project has released build b10727, which introduces concatenation support for quantized data types. This update is assisted by DeepSeek-V4-Flash-0731 and includes binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, and Vulkan.

github llama.cpp · 11h ago · 6 views

llama.cpp b10726 speeds up large batch IQ model prompt processing with AVX2

The llama.cpp project released version b10726, which introduces significant performance optimizations for processing large batch sizes in IQ quantized models. The primary update leverages AVX2 instructions to accelerate grid IQ quants through batched GEMM operations.

github llama.cpp · 13h ago · 7 views

llama.cpp b10723 tunes OpenCL quant paths for Intel Xe-LP GPUs

The llama.cpp project released version b10723, which includes specific optimizations for OpenCL on Intel Xe-LP graphics hardware. The update focuses on improving tensor generation (TG) and prefill (PP) performance by adjusting quantization kernels.

github llama.cpp · 16h ago · 9 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 18h ago · 6 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.