github llama.cpp · just now · inference

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

from English

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

Supported KV types include Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, and F32, while BF16 and IQ types are excluded due to missing conversion kernels.
Non-FP16 support is gated by requirements that K >= 1024 and Q >= 32 during the prefill phase.
The release includes a stream sync fix ensuring unconditional wait_and_throw behavior and removes V_is_K_view aliasing to ensure K and V are always dequantized to separate buffers.
Binaries are provided for macOS, Linux, Windows, Android, and openEuler across CPU, GPU (CUDA, Vulkan, ROCm, OpenCL), and specialized backends like SYCL and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Hardware & chips Inference efficiency

Read original

media r/LocalLLaMA · 30d ago

llama.cpp uses hipBLAS for dense prefill on gfx900 GPUs

A pull request to the llama.cpp repository introduces a change that utilizes hipBLAS for dense prefill operations on AMD gfx900 architecture GPUs. This modification is specifically targeted at legacy Vega GPU hardware, including models like the Radeon RX Vega 56/64 and Radeon Pro Vega series.

github llama.cpp · 7h ago

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.

github llama.cpp · 1d ago · 6 views

llama.cpp b10236 implements Metal DSv4 Lightning Indexer

The llama.cpp b10236 release introduces a Metal implementation of the DSv4 Lightning Indexer, specifically supporting 128-dimensional inputs with F32 queries and weights alongside F16 keys and masks.

github llama.cpp · 4d ago · 8 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

media r/LocalLLaMA · 5d ago

Lucebox and AMD beat Nvidia DGX Spark by 3.63x on DeepSeek V4 Flash

Lucebox has partnered with AMD to demonstrate a heterogeneous consumer hardware setup that outperforms the Nvidia DGX Spark in inference speed. The system combines an AMD Radeon AI PRO R9700 GPU with a Strix Halo processor to run the full 284B DeepSeek V4 Flash model.