github llama.cpp · 1h ago · inference

llama.cpp b10790 tunes MMVQ to MMQ crossover for SM87

from English

The llama.cpp project has released build b10790, which includes a specific tuning of the MMVQ to MMQ crossover for SM87.

This update provides pre-built binaries across multiple platforms and hardware configurations. Available builds include macOS Apple Silicon and Intel, Linux (Ubuntu) with CPU, Vulkan, ROCm 10.0, OpenVINO, and SYCL support, as well as Windows builds featuring CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, and ROCm 10.0. Additional releases cover Android arm64, iOS XCFramework, and openEuler x86/aarch64 architectures.

The release also includes the standard llama.cpp UI package for all supported operating systems.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media r/LocalLLaMA · 6h ago · 1 view

IFM releases K2-Horizon-MoVA-36B-A4B, a sparse MoE model with 4B active parameters

IFM has released the final checkpoint for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model featuring Mixture-of-Values attention that stores 36B parameters but runs only 4B per token. The release includes GGUF formats and links to smaller variants in the K2-Horizon family.

github llama.cpp · 8h ago · 1 view

llama.cpp adds sparse flash attention support for Metal prefill

The llama.cpp project has introduced sparse flash attention (FA) support for the Metal backend, specifically targeting the prefill phase of model inference. This update includes a new kernel that compacts finite mask entries into per-row index lists and extends the vector FA kernel with optional sparse index gathering.

github llama.cpp · 9h ago · 2 views

llama.cpp b10782 fixes multi-GPU CUDA graph optimization

The llama.cpp project released build b10782, which addresses a critical issue in the CUDA backend for multi-GPU setups. The update allows concurrent streams per split and enables CUDA graph optimization to run correctly across multiple devices.

github llama.cpp · 10h ago · 1 view

llama.cpp b10780 enhances SYCL API to support peer-to-peer copy

The llama.cpp project has released version b10780, which includes a specific enhancement to the SYCL backend. This update modifies the API to enable peer-to-peer copy capabilities for SYCL devices.

github llama.cpp · 11h ago · 2 views

llama.cpp b10781 fixes Vulkan FA dequant path engagement

The llama.cpp project released build b10781, which includes a fix for the Vulkan fast attention (FA) dequantization path. The update addresses an issue where the shader incorrectly checked `nb[3]` even when `ne[3] == 1`, causing the path to only engage when the cache was full.