github llama.cpp · 1h ago · inference

llama.cpp b11440 re-reserves scheduler when NextN extraction flags change

from English

The llama.cpp project released version b11440, which includes a fix for speculative MTP initialization. The update ensures the scheduler is re-reserved when NextN extraction flags change, preventing issues with unmasked extraction.

Speculative MTP init now enables NextN extraction on target and draft contexts after creation.
The trunk graph keeps every token through the last layer instead of cropping to output rows.
Invalidating the reserve when flags change allows the next compute to re-reserve with the new graph shape.
Binaries are available for macOS, Linux, Windows, Android, and iOS across CPU, GPU, and NPU backends.

This change prevents GGML_SCHED_DEBUG_REALLOC errors by ensuring correct batch shape allocation during decoding.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1d ago · 18 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 1d ago · 20 views

webgpu adds MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4

The webgpu component has added support for MMVQ (Multi-Mode Vector Quantization) formats, specifically Q1_0, Q5_0, Q5_1, Q3_K, Q5_K, Q6_K, and MXFP4.

media Hugging Face Forums · 2d ago · 26 views

Eqx-zoo provides verified Equinox ports of Hugging Face models

The author has built eqx-zoo, a library that loads Hugging Face Hub checkpoints directly as plain Equinox modules and verifies each model against the transformers library. The project currently supports Llama, Qwen2, Qwen3, Qwen3-MoE for generation, and BERT, RoBERTa, XLM-RoBERTa for embeddings.

media Hugging Face Forums · 5d ago · 27 views

AiiStream Q3.6 enables running Qwen3.6-35B-A3B on 8-16 GB Macs

The author has released AiiStream Q3.6, an open-source technique that allows the Qwen3.6-35B-A3B model to run on Apple Silicon devices with only 8 to 16 GB of RAM. By keeping routed experts on SSD and using early loading predictions, the method reduces MLX memory usage to approximately 1.7 GB while maintaining byte-identical outputs to the standard mlx_lm implementation.

github llama.cpp · 6d ago · 36 views

llama.cpp b11307 preserves batch order for speculative decoding

The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.