github llama.cpp · just now · inference

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

from English

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

The new argument is supported by a backwards-compatible `MTMD_BACKEND_DEVICE` environment variable.
A short flag `-mmdev` is also available for quick access to this functionality.
The release includes binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x with CPU, Vulkan, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 7.14), Android, and openEuler.
An updated UI binary is also provided in this release.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 10h ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

github llama.cpp · 20d ago · 35 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 29d ago · 26 views

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.