github llama.cpp · just now · inference

llama.cpp b10795 fuses SYCL operations

from English

The llama.cpp project released version b10795, which includes a new optimization for the SYCL backend. This update enables the fusion of RMS_NORM+MUL+ADD and ADD+ADD residual chains under the GGML_SYCL_ENABLE_FUSION flag.

The fused ADD+ADD operation reuses the same broadcast indexing and type matrix as the standalone add() function, supporting f32, f16, i32, i16, bf16, and non-contiguous data.
Unsupported combinations in the fusion logic fall back to two separate add() launches to maintain compatibility.

This release provides binaries for macOS, Linux, Windows, Android, and openEuler across various hardware accelerators including CUDA, ROCm, Vulkan, and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 7h ago · 3 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

media r/LocalLLaMA · 14h ago · 3 views

IFM releases K2-Horizon-MoVA-36B-A4B, a sparse MoE model with 4B active parameters

IFM has released the final checkpoint for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model featuring Mixture-of-Values attention that stores 36B parameters but runs only 4B per token. The release includes GGUF formats and links to smaller variants in the K2-Horizon family.

github llama.cpp · 16h ago · 3 views

llama.cpp adds sparse flash attention support for Metal prefill

The llama.cpp project has introduced sparse flash attention (FA) support for the Metal backend, specifically targeting the prefill phase of model inference. This update includes a new kernel that compacts finite mask entries into per-row index lists and extends the vector FA kernel with optional sparse index gathering.

github llama.cpp · 19h ago · 1 view

llama.cpp b10780 enhances SYCL API to support peer-to-peer copy

The llama.cpp project has released version b10780, which includes a specific enhancement to the SYCL backend. This update modifies the API to enable peer-to-peer copy capabilities for SYCL devices.

github llama.cpp · 19h ago · 2 views

llama.cpp adds support for NVIDIA Nemotron-3-Puzzle-75B-A9B

llama.cpp introduces support for the NVIDIA Nemotron-3-Puzzle-75B-A9B model, enabling inference for this Mixture of Experts architecture. The update implements infrastructure for variable-per-layer expert feed-forward sizes and top-k routing, which are required because the model has 40 MoE layers with distinct n_ff_exp and top-k values.