github llama.cpp · just now · inference

llama.cpp b11160 adds int8 coopmat1 matmul for AMD RDNA3/4 and IQ4_XS support

from English

The llama.cpp project released version b11160, introducing an int8 cooperative matrix (coopmat1) implementation for Vulkan on AMD RDNA3 and RDNA4 architectures. This update includes a dedicated shader that probes coopmat values directly to improve performance.

Adds IQ4_XS quantization support to the mul_mmq_cm1 integer matmul shader.
Implements inline scale application, scalar sums, and double buffering for the Vulkan backend.
Supports q8_0, q4_1, q5_0, q5_1, iq4_nl, mxfp4, q3_k, q4_k, q5_k, q6_k, and nvfp4 quantizations.
Includes binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, OpenVINO, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Hardware & chips Inference efficiency

Read original

media Hugging Face Forums · 29d ago · 38 views

EdgeFirst publishes Rust pipeline validating YOLO models on 20 edge platforms

EdgeFirst has published the EdgeFirst Model Zoo on Hugging Face, featuring YOLOv5, YOLOv8, YOLO11, and YOLO26 models validated across 20 distinct edge platforms. The validation relies on a single Rust pipeline that handles zero-copy DMA pre-processing, model decoding, and post-processing directly on the target hardware.

github llama.cpp · 51d ago · 35 views

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

media r/LocalLLaMA · 81d ago · 29 views

llama.cpp uses hipBLAS for dense prefill on gfx900 GPUs

A pull request to the llama.cpp repository introduces a change that utilizes hipBLAS for dense prefill operations on AMD gfx900 architecture GPUs. This modification is specifically targeted at legacy Vega GPU hardware, including models like the Radeon RX Vega 56/64 and Radeon Pro Vega series.

github llama.cpp · 4h ago · 2 views

llama.cpp b11158 enables KHR cooperative matrix support for Adreno GPUs

The llama.cpp project released build b11158, which includes a tune of KHR cooperative matrix support specifically for Adreno GPUs via the Vulkan backend.

github llama.cpp · 18h ago · 3 views

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.