github llama.cpp · 1h ago · inference

llama.cpp b10533 adds graceful fallback for unsupported JSON schema regex patterns

from English

The llama.cpp project has released build b10533, which includes a fix to gracefully handle unsupported regular expression patterns within JSON schemas.

The primary code change allows the system to fall back gracefully when encountering regex patterns that are not supported by the underlying implementation.
Binaries are available for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x with CPU, Vulkan, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 7.14), Android, and openEuler.
macOS Apple Silicon KleidiAI support is disabled in this release, as is Ubuntu x64 ROCm 7.14 support.
An iOS XCFramework and a standalone UI package are also included in the distribution.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 8h ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

github llama.cpp · 19h ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

llama.cpp b10509 adds ggml_rope_set_offset support to Metal, CUDA, and Vulkan

llama.cpp b10427 fuses SYCL FFN ops for up to 12.4% throughput gain

llama.cpp b10413 adds auto-detection of draft spec type from GGUF metadata