github llama.cpp · just now · inference

llama.cpp b10734 enables Metal 4.0 tensor API on M5+/A19+

from English

The llama.cpp project released version b10734, which includes an update to enable the Metal 4.0 tensor API for devices with M5+ or A19+ chips.

Requests the Metal 4.0 language version for the tensor API.
Loads tensor API kernels from a separate metallib.
Fixes the metallib build order for the tensor API kernels.
Adds an external-metallib tensor API regression test.

This release provides binaries for macOS, iOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, Vulkan, OpenVINO, SYCL, and OpenCL.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

media r/LocalLLaMA · 5h ago · 1 view

ExLlamaV3 adds CPU offload for MoE experts and new GLM-5.3-Flash, Qwen3.8-Flash models

Turboderp has released significant updates to ExLlamaV3, introducing CPU offload capabilities for Mixture of Experts (MoE) experts. The update also includes support for the newly released GLM-5.3-Flash and Qwen3.8-Flash models.

github llama.cpp · 18h ago · 2 views

llama.cpp b10724 optimizes kv-cache state restore to reduce copy count from 1.3M to 224

The llama.cpp project released build b10724, which significantly improves the performance of restoring non-contiguous KV cache cells by batching scatter reads per contiguous run.

github llama.cpp · 20h ago · 3 views

llama.cpp b10721 fixes WebGPU crash and adds Windows CUDA 13.4 support

The llama.cpp project has released build b10721, which includes a critical fix for the WebGPU backend and expands hardware support on Windows.

github llama.cpp · 22h ago · 9 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 1d ago · 4 views

llama.cpp b10715 fuses DFlash encoder into KV cache injection

The llama.cpp project released build b10715, which includes a key optimization to the DFlash implementation. The DFlash encoder is now fused directly into the KV cache injection process during decoding.