github llama.cpp · 2h ago · inference

llama.cpp b10338 fixes MoE model save/load roundtrip failure

from English

The llama.cpp b10338 release resolves a critical bug in the model saver that caused shared expert Mixture-of-Experts (MoE) models to fail loading after being saved. The issue stemmed from the saver incorrectly writing the `n_ff_chexp` value to the `LLM_KV_EXPERT_SHARED_FEED_FORWARD_LENGTH` key, which clobbered the correct tensor shape information.

The fix ensures the second call writes the correct `LLM_KV_EXPERT_CHUNK_FEED_FORWARD_LENGTH` key instead of overwriting the shared feed-forward length.
This resolves load failures for architectures with shared experts, including qwen2moe, qwen3-next, granite-moe, hunyuan-moe, ernie4.5, bailingmoe2, and nemotron-h.
The release includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, OpenVINO, SYCL, HIP), and openEuler.

This fix ensures that save-and-load roundtrips for MoE models with shared experts succeed without tensor shape mismatches.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago Live

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.

lab Hugging Face Blog · 3h ago · 2 views

Multiverse Computing cuts knowledge distillation VRAM with fused chunked KL loss

Multiverse Computing introduces a memory-efficient approach to large language model knowledge distillation by combining offline top-K logits caching with a fused chunked Kullback-Leibler (KL) divergence loss. This method eliminates the need to keep both teacher and student models in memory simultaneously, drastically reducing VRAM requirements.

github llama.cpp · 5h ago

llama.cpp b10336 release with ggml-webgpu refactor

The llama.cpp project has released version b10336, which includes a refactor of several wgsl files and simplification of flash_attn wgsl within the ggml-webgpu component.

media Hugging Face Forums · 1d ago · 8 views

vLLM Launcher provides a Windows desktop workbench for local LLM inference

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.

github llama.cpp · 2d ago · 4 views

llama.cpp b10331 fixes server get_info to report correct isolate working directory

The llama.cpp project released build b10331, which includes a fix for the server's `get_info` endpoint. Previously, when no explicit current working directory was provided, the function incorrectly fell back to the server process's working directory, even if a tools runtime was configured.