github llama.cpp · just now · inference

llama.cpp b10796 adds n_expert_used_max to fix Nemotron-3-Puzzle loading

from English

The llama.cpp project released build b10796, which includes a change to the model loader to support models with varying numbers of experts per layer.

Adds the `n_expert_used_max` function to handle expert layer checks during model loading.
Updates `llama_model_base::load_hparams` to use `hparams.n_expert_used_max` instead of failing assertions.
Resolves errors when loading models like NVIDIA Nemotron-3-Puzzle-75B-A9B that have specific expert configurations.

This update allows llama.cpp to correctly load and process models where the number of used experts varies across layers, preventing assertion failures during initialization.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 8h ago · 3 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

github llama.cpp · 1d ago · 2 views

ggml-org releases llama.cpp b10774 with finetune KV cache fix

The ggml-org project has released version b10774 of llama.cpp, which includes a fix for the no KV cache issue in the finetune module.

github llama.cpp · 1d ago · 7 views

llama.cpp b10763 enables preserve_reasoning by default

The llama.cpp project released build b10763, which changes the default behavior of the `preserve_reasoning` chat template keyword argument. The server now logs the effective state of this kwarg and issues a warning when it is enabled by default on templates that support it.

github llama.cpp · 2d ago · 3 views

llama.cpp b10760 fixes Qwen3-tts-0.6b inference stability

The llama.cpp project released build b10760, which includes specific fixes for the Qwen3-tts-0.6B model to prevent numerical instability during inference.

github llama.cpp · 2d ago · 3 views

llama.cpp b10751 fuses MoE weighted expert reduction for CUDA

The llama.cpp project released build b10751, which introduces a fused kernel for the MoE (Mixture of Experts) weighted expert reduction on CUDA. This optimization replaces the previous baseline that ran two physical kernels with a single weighted-reduction kernel to reduce intermediate global-memory traffic.