github llama.cpp · 1h ago · inference

llama.cpp b10256 parallelizes the SYCL non-contiguous concat kernel

from English

The llama.cpp b10256 release includes a change to parallelize the non-contiguous concatenation kernel for SYCL devices. The update modifies the launch geometry from a single-lane work-group to one utilizing `SYCL_CONCAT_BLOCK_SIZE`.

The SYCL non-contiguous concat kernel now launches a wider work-group to improve parallelism.
Benchmarks on an Arc Pro B70 with Qwen3.6-27B show a 9.4% throughput increase (920 to 1006 t/s).
The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

This optimization enhances inference performance on Intel GPUs using the SYCL backend.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago Live

llama.cpp b10254 adds DeepSeek V4 Flash 0731 template and updates V4 templates

The llama.cpp project released version b10254, introducing a new chat template for the DeepSeek V4 Flash 0731 model alongside updates to existing DeepSeek V4 templates.

github llama.cpp · 1h ago Live

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

github llama.cpp · 4h ago

llama.cpp b10251 adds MTP support for GLM-4.7-Flash

The llama.cpp project has released build b10251, which introduces Multi-Token Prediction (MTP) support for the GLM-4.7-Flash model.

media r/LocalLLaMA · 5h ago

Apple's AFM3 20B uses instruction-following pruning to activate ~20% of layers

Apple has introduced a new architecture for its third-generation foundation models (AFM3) that utilizes "Instruction-Following Pruning" to significantly reduce active parameter count. The model is designed to activate approximately 20% of its MLP layers by making routing decisions once per prompt rather than per token.

github llama.cpp · 6h ago

llama.cpp b10249 adds get_info tool to server

The llama.cpp project released version b10249, which introduces a new `get_info` tool to the server component. This update also includes documentation fixes for the `--rpc` flag.