github llama.cpp · just now · inference

llama.cpp b10891 fixes PowerVR Vulkan crash by falling back to shared-memory reduction

from English

The llama.cpp project released version b10891, which addresses a critical stability issue on PowerVR GPUs. The update modifies the Vulkan backend to fall back to shared-memory reduction for dequantized matrix-vector multiplication (dmmv) operations.

The change resolves a crash caused by the Imagination proprietary Vulkan compiler returning VK_ERROR_UNKNOWN when building compute pipelines with subgroup-only reductions.
This fix applies to k-quants, i-quants, TQ2_0, MXFP4, and NVFP4 quantization formats that previously failed on devices like the Pixel 11 Pro.
The shared-memory reduction variant compiles successfully and matches CPU reference results for q2_K through q6_K quantizations.
A hybrid variant is also available but incurs a significant performance penalty, reducing token throughput to 27% of the standard rate.

This update ensures that models using these specific quantization formats can run on PowerVR hardware without crashing during the initial token generation phase.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

github llama.cpp · 3d ago · 12 views

llama.cpp b10839 fixes Vulkan GET_ROWS misaligned offset crashes

The llama.cpp project released version b10839, which resolves hard crashes in the Vulkan backend caused by misaligned offsets during tensor row retrieval operations. Previously, models like Qwen3-TTS and Qwen3-VL would fail when using `ggml_view` with non-zero view offsets because the shader asserted on alignment violations relative to `minStorageBufferOffsetAlignment`. This update implements native support for aligned offsets in both quantized and non-quantized paths, eliminating the need for CPU fallbacks.

arxiv arXiv cs.LG · 6d ago · 10 views

FlashAttention-4 introduces Direct-P for hardware-aware FP4 inference and training

FlashAttention-4 addresses the performance limitations of Blackwell's 4-bit floating-point (FP4) tensor cores by introducing new paths for noncausal inference and causal training. The method, called Direct-P, maps scores directly to FP4 probabilities to bypass softmax conversion overheads that typically dominate once matrix products shrink.

lab NVIDIA Research · 12d ago · 14 views

Twill automatically derives optimal software pipelining and warp specialization schedules for Tensor Core GPUs

Researchers have introduced Twill, a system that formulates software pipelining (SWP) and warp specialization (WS) as a joint optimization problem solvable by off-the-shelf constraint solvers. This approach automates the derivation of optimal execution schedules for iterative programs on GPU architectures.

github llama.cpp · 13d ago · 26 views

llama.cpp b10675 adds Vulkan hoisting support for row IDs and expert count in shaders

The llama.cpp project released version b10675, which includes a key update to its Vulkan backend. This release adds hoisting support for row IDs and expert counts within the shader code.

github llama.cpp · 13d ago · 12 views

llama.cpp adds fa-vec tunings for Apple M4 Pro

The llama.cpp project has added fast attention vector (fa-vec) tunings for the Apple M4 Pro GPU via a new Metal backend contribution. This update follows previous work to support additional Apple GPUs and was generated using the `ggml-metal-tuning` tool on a 14-inch MacBook Pro from November 2024.