media r/LocalLLaMA · 1h ago · open_models

ExLlamaV3 adds CPU offload for MoE experts and new GLM-5.3-Flash, Qwen3.8-Flash models

from English

Turboderp has released significant updates to ExLlamaV3, introducing CPU offload capabilities for Mixture of Experts (MoE) experts. The update also includes support for the newly released GLM-5.3-Flash and Qwen3.8-Flash models.

Implementation of CPU offload specifically for MoE experts.
Support for Qwen-3.8-Flash-Next with ngram disk offload capabilities.
Integration of GLM-5.3-Flash via the exl3 format.
Introduction of a new self-calibrated optimization technique.
Various other optimizations and improvements to the engine.

These updates expand the hardware flexibility and model compatibility of ExLlamaV3, allowing users to leverage CPU resources for specific model components.

Importance 2/3 New harness with differentiators r/LocalLLaMA Inference efficiency Local inference

Read original

github llama.cpp · 14h ago · 2 views

llama.cpp b10724 optimizes kv-cache state restore to reduce copy count from 1.3M to 224

The llama.cpp project released build b10724, which significantly improves the performance of restoring non-contiguous KV cache cells by batching scatter reads per contiguous run.

github llama.cpp · 16h ago · 3 views

llama.cpp b10721 fixes WebGPU crash and adds Windows CUDA 13.4 support

The llama.cpp project has released build b10721, which includes a critical fix for the WebGPU backend and expands hardware support on Windows.

github llama.cpp · 18h ago · 9 views

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 21h ago · 4 views

llama.cpp b10715 fuses DFlash encoder into KV cache injection

The llama.cpp project released build b10715, which includes a key optimization to the DFlash implementation. The DFlash encoder is now fused directly into the KV cache injection process during decoding.

github llama.cpp · 23h ago · 2 views

llama.cpp b10714 tunes Vulkan mat-vec rows for Strix Halo

The llama.cpp project released build b10714, which includes specific optimizations for the Vulkan backend targeting AMD Strix Halo hardware. The update adjusts matrix-vector multiplication row counts to improve performance during batched inference.