github llama.cpp · 1h ago · inference

llama.cpp b10247 fixes crashes with wide MoE models by using dynamic allocation

from English

The llama.cpp project released version b10247, which addresses stability issues when loading large Mixture of Experts (MoE) models on multi-backend setups. The update replaces fixed-size arrays in the backend scheduler with dynamically allocated buffers to handle graph splits exceeding 30 input tensors.

Replaces GGML_SCHED_MAX_SPLIT_INPUTS with dynamic allocation for split graph inputs.
Fixes crashes when loading wide MoE models such as Gemma 4, Qwen MoE, Mixtral, and DeepSeek.
Updates graph_size calculation to use actual input count instead of a fixed constant.
Provides binaries for macOS (Apple Silicon/Intel), Linux (CPU/Vulkan/ROCm/OpenVINO/SYCL), Windows (CPU/CUDA/Vulkan/OpenVINO/SYCL/HIP), Android, and iOS.

This change allows users to run complex MoE architectures without encountering scheduler-related crashes on systems with multiple backend devices.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.

github llama.cpp · 7h ago · 2 views

llama.cpp b10242 adds CUDA backend sampler for penalties

The llama.cpp project released version b10242, introducing a new backend sampler for penalties that allows penalty handling to be offloaded to the GPU. This change enhances the `common_sampler_init` function and updates the `llama_sampler_penalties` structure to inherit from `llama_sampler_backend`, enabling frequency and presence adjustments on the backend.

github llama.cpp · 10h ago · 1 view

llama.cpp b10241 fixes CUDA SMEM data-races in block_reduce

The llama.cpp project released version b10241, which addresses a critical bug in the CUDA backend. The update resolves data-races that occurred when reusing shared memory (SMEM) during block reductions.

media Hugging Face Forums · 13h ago · 1 view

Quesma publishes curated map of 218 tools and papers for LLM token cost efficiency

Quesma has released an open, curated list containing 218 entries that track where tokens are spent in LLM and agent workflows. The resource covers monitoring, semantic caching, cheap local models, context engineering, and cost governance.

github llama.cpp · 16h ago

llama.cpp b10238 adds MTP support for Qwen3-Next

The llama.cpp project released build b10238, which introduces Multi-Token Prediction (MTP) support for the Qwen3-Next model family. This update includes specific implementation changes to handle MTP layers and fixes related to Python type checking within the codebase.