github llama.cpp · just now · inference

llama.cpp b10817 adds SYCL memory tracing via GGML_SYCL_MEMTRACE

from English

The llama.cpp b10817 release introduces a new SYCL memory tracing feature to help developers understand device memory allocation patterns. This functionality is controlled by two new environment variables that allow for detailed logging of memory usage.

Setting GGML_SYCL_MEMTRACE=1 displays per-site memory usage, updating only when allocations increase by more than 64MiB.
Setting GGML_SYCL_MEMTRACE=2 logs every individual allocation and deallocation event.
The default reporting threshold can be customized using the GGML_SYCL_MEMTRACE_STEP variable.

This tooling is intended to assist in debugging memory-related issues and has already proven useful for improving the --fit algorithm.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 14h ago · 7 views

llama.cpp 0.4.0 adds Qwen3.8-Flash-Next and Nemotron-3-Puzzle support

llama.cpp version 0.4.0 introduces initial architecture support for the Qwen3.8-Flash-Next and NVIDIA Nemotron-3-Puzzle-75B-A9B models, alongside a bump of the underlying ggml library to version 0.23.0.

github llama.cpp · 1d ago · 2 views

llama.cpp b10795 fuses SYCL operations

The llama.cpp project released version b10795, which includes a new optimization for the SYCL backend. This update enables the fusion of RMS_NORM+MUL+ADD and ADD+ADD residual chains under the GGML_SYCL_ENABLE_FUSION flag.

github llama.cpp · 2d ago · 3 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

media r/LocalLLaMA · 2d ago · 6 views

IFM releases K2-Horizon-MoVA-36B-A4B, a sparse MoE model with 4B active parameters

IFM has released the final checkpoint for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model featuring Mixture-of-Values attention that stores 36B parameters but runs only 4B per token. The release includes GGUF formats and links to smaller variants in the K2-Horizon family.

github llama.cpp · 2d ago · 8 views

llama.cpp adds sparse flash attention support for Metal prefill

The llama.cpp project has introduced sparse flash attention (FA) support for the Metal backend, specifically targeting the prefill phase of model inference. This update includes a new kernel that compacts finite mask entries into per-row index lists and extends the vector FA kernel with optional sparse index gathering.