github llama.cpp · 2h ago · inference

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

from English

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

Vulkan now falls back instead of aborting when FA scratch exceeds maxStorageBufferRange.
The release requires a specific KV-cache layout in the FA dequant path.
Tightened permutation checks for the FA path and corrected prealloc_x_need_sync timing.
Added tests for contiguously-allocated quant K/V FA scenarios.

This update enhances stability and performance for Vulkan users by addressing edge cases in memory management and hardware compatibility.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 20d ago · 34 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 29d ago · 26 views

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

media r/LocalLLaMA · 7h ago · 2 views

G9v3-39A5B MoE model shows strong results on artificialanalysis

The G9v3-39A5B model is drawing attention for its performance on artificialanalysis, where it reportedly ranks above Qwen 3.6 27B.

media Hugging Face Forums · 7h ago · 1 view

Celestium pipeline enables SDXL on 4GB-8GB GPUs via adaptive FP16 and CPU fallback

A diagnostic using Celestium’s hardware-agnostic inference pipeline demonstrates that Stable Diffusion XL can run stably on low-VRAM GPUs (4GB, 6GB, and 8GB) by leveraging adaptive FP16 precision and autonomous memory cleanup. The approach prevents fragmentation spikes during multi-stage pipelines and utilizes an emergency CPU mode to maintain generation when GPU VRAM is exhausted.

github llama.cpp · 12h ago · 1 view

llama.cpp b10509 adds ggml_rope_set_offset support to Metal, CUDA, and Vulkan

The llama.cpp project released build b10509, which introduces support for the `ggml_rope_set_offset` function across multiple hardware backends. This update includes CPU, Metal, CUDA, and Vulkan implementations of the new operation.