github llama.cpp · 2h ago · inference

llama.cpp b10539 fixes Vulkan FA MMQ overflow and adds Windows CUDA 13 support

from English

The llama.cpp project has released version b10539, which addresses a critical numerical stability issue in the Vulkan backend while expanding hardware support for Windows users.

Vulkan FA MMQ now uses fp32 for Q quantization calculations to prevent overflow when qd is a denorm.
Added Windows x64 binaries for CUDA 13 and preview support for Windows arm64 with CUDA 13.4.
Included builds for Ubuntu s390x (CPU), macOS Intel (x64), and iOS XCFramework.
OpenCL Adreno support is now available for Windows arm64.

This update ensures reliable performance on Vulkan devices by preventing arithmetic errors and provides users with the latest CUDA 13 compatibility options.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 2h ago · 1 view

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.

github llama.cpp · 3h ago · 1 view

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

llama.cpp b10538 release with dequantized KV cache for large batches

llama.cpp b10532 adds Metal KV dequantization for Flash Attention