github llama.cpp · 1h ago · inference

llama.cpp b10293 adds ROCm CI for gfx1151 and fixes RDNA3.5 inference

from English

The llama.cpp project released build b10293, which includes onboarding AMD ROCm continuous integration for the gfx1151 architecture and resolves correctness issues on integrated RDNA3.5 GPUs.

Fixes test-recurrent-state-rollback on gfx1151 by allowing integrated-GPU host output buffers in debug asserts.
Restores inference correctness on RDNA3.5 by enabling unified memory to work around GPU coherence issues with mmap-loaded weights.
Addresses async-execution errors in the HIP path by utilizing HIP_LAUNCH_BLOCKING=1 for serializing kernel launches.
Skips jamba architecture tests and top-k subtests on the HIP backend due to unsupported operations and incorrect output.

This update enables reliable testing and inference of llama.cpp models on AMD RDNA3.5 hardware via ROCm.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2d ago · 1 view

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

github llama.cpp · 2d ago

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.

github llama.cpp · 3d ago · 6 views

llama.cpp b10236 implements Metal DSv4 Lightning Indexer

The llama.cpp b10236 release introduces a Metal implementation of the DSv4 Lightning Indexer, specifically supporting 128-dimensional inputs with F32 queries and weights alongside F16 keys and masks.

github llama.cpp · 6d ago · 8 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 8d ago · 2 views

llama.cpp b10181 disables MMQ on devices with less than 48 KiB shared memory

The llama.cpp project released build b10181, which includes a fix in ggml-cuda to disable Matrix Multiplication Quantization (MMQ) on GPUs with less than 48 KiB of per-block shared memory. Previously, MMQ configurations assumed a minimum of 48 KiB, causing fatal errors on devices like NVIDIA Pascal GPUs and Moore Threads MTT S70 where the tile size exceeded available memory.