github llama.cpp · just now · inference

llama.cpp b10670 routes quantized KV decode to TILE on Xe2 (BMG)

from English

The llama.cpp project released version b10670, which includes a specific optimization for Intel's Xe2 architecture. The update routes the quantized key-value (KV) decode operation to use the TILE backend exclusively for BMG hardware, while keeping the VEC backend on other architectures until they are validated.

Quantized KV decode is now directed to TILE on Xe2 (BMG).
Other architectures continue to use the VEC backend pending validation.
Binaries are provided for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm 7.14, OpenVINO, SYCL FP32/FP16), Android, Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 7.14), and openEuler.

This change optimizes memory access patterns for specific Intel hardware generations.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 18h ago · 7 views

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.

github llama.cpp · 1d ago · 8 views

llama.cpp b10653 adds TENSOR_READ_LAZY for model loading

The llama.cpp project has released build b10653, introducing the TENSOR_READ_LAZY feature to its model loader. This change includes the addition of the TENSOR_GET_ROW_LAZY function and a corresponding --tensor-read-lazy command-line flag.

github llama.cpp · 7d ago · 12 views

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2

The llama.cpp project released build b10568, which updates the model handling to use the `ggml_rope_set_offset()` function. This change is partially applied to support DeepSeek 2.

github llama.cpp · 7d ago · 17 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 8d ago · 8 views

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.