github llama.cpp · 1h ago · inference

llama.cpp b10456 fixes SYCL thread count for 7x throughput gain on Arc 70

from English

The llama.cpp project released version b10456, which includes a critical fix for the SYCL backend in quantized copy kernel launches. The update adjusts thread and block counts to be proportional to the quantization size, effectively reducing under- and over-subscription issues.

Fixes thread/block count in quantized cpy kernel launches (PR #27160).
Throughput for the q4_0 -> f32 path on an Arc 70 increases from 20.21 GB/s to 158.19 GB/s.
Other quantization paths show flat performance uplifts.
Binaries are available for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

This change significantly improves data transfer efficiency on Intel Arc GPUs using the SYCL backend, addressing a major performance bottleneck in quantized operations.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media Hugging Face Forums · just now Live

SpiralCoreAttention prototype selects token blocks for long-sequence QLoRA fine-tuning

A developer is seeking feedback on SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning. Instead of processing complete long sequences during every forward and backward pass, the system selects specific token blocks from the sequence to train on, while evaluating the resulting adapter on held-out full-context sequences.

media r/LocalLLaMA · 14h ago · 8 views

NVIDIA serves Qwen 3.8 2.4T on GB300 NVL72 at 4K tokens/s per GPU

NVIDIA has demonstrated serving the Qwen 3.8 2.4T parameter model with configurable reasoning capabilities on its GB300 NVL72 hardware platform.

github llama.cpp · 1d ago · 9 views

llama.cpp adds Kimi-K3 model support with MXFP4 repacking and chat template

The llama.cpp project has added support for the Kimi-K3 text model, implementing its hybrid KDA (linear) + MLA (full) attention architecture along with five specific architectural features not present in the older Kimi-Linear-48B.

github llama.cpp · 2d ago · 10 views

llama.cpp b10447 redesigns yield_to_queue thread model

The llama.cpp project has released build b10447, which includes a re-design of the `yield_to_queue` thread model. This change involves running `common_speculative_process` in the worker and swapping the worker-to-main thread design.

github llama.cpp · 2d ago · 9 views

llama.cpp b10444 adds MTP assistant model support and drops eagle3

The llama.cpp project has released version b10444, introducing support for loading Multi-Token Prediction (MTP) assistant models via the `--models-dir` flag. This update also includes strict prefix checking for MTP models in presets and removes support for the eagle3 draft type.