github llama.cpp · just now · inference

llama.cpp b10727 adds concat support for quantized types

from English

The llama.cpp project has released build b10727, which introduces concatenation support for quantized data types. This update is assisted by DeepSeek-V4-Flash-0731 and includes binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, and Vulkan.

The release provides precompiled binaries for multiple platforms and architectures:

macOS Apple Silicon (arm64) and Intel (x64)
iOS via XCFramework
Linux distributions including Ubuntu x64, arm64, s390x, and variants with Vulkan, ROCm 7.14, OpenVINO, and SYCL support
Windows x64 and arm64 with CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, and ROCm 7.14 options
Android arm64 (CPU)
openEuler x86 and aarch64 configurations

This release enables users to utilize the new quantized concat functionality across a wide range of supported operating systems and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b10726 speeds up large batch IQ model prompt processing with AVX2

The llama.cpp project released version b10726, which introduces significant performance optimizations for processing large batch sizes in IQ quantized models. The primary update leverages AVX2 instructions to accelerate grid IQ quants through batched GEMM operations.

github llama.cpp · 4h ago · 2 views

llama.cpp b10723 tunes OpenCL quant paths for Intel Xe-LP GPUs

The llama.cpp project released version b10723, which includes specific optimizations for OpenCL on Intel Xe-LP graphics hardware. The update focuses on improving tensor generation (TG) and prefill (PP) performance by adjusting quantization kernels.

github llama.cpp · 7h ago · 1 view

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 9h ago · 3 views

llama.cpp b10717 release enhances SYCL memory info retrieval

The llama.cpp project has released version b10717, which includes a specific enhancement to the SYCL backend for Intel GPUs. This update improves the ability to retrieve free memory information by leveraging the L0 SYCL API.

github llama.cpp · 15h ago · 3 views

llama.cpp b10712 adds Vulkan top-k radix select for Qwen 3.8 Flash Next

The llama.cpp project released build b10712, introducing a new Vulkan top-k radix sort shader designed to support k values greater than or equal to 1024. This update specifically targets the Qwen 3.8 Flash Next model and includes corresponding tests as well as top-k qsa fusion.