github llama.cpp · 1h ago · inference

llama.cpp b11450 adds RPC tensor support and fixes Apple RDMA flush

from English

The llama.cpp project released build b11450, introducing remote procedure call (RPC) enhancements and platform-specific bug fixes. The update allows the `-sm tensor` flag for RPC operations and resolves a flush issue affecting Apple RDMA implementations.

Added support for the `-sm tensor` argument in RPC configurations.
Fixed data flushing behavior for Apple RDMA systems.
Moved graph_uids to the rpc_dispatcher module.
Stopped the dispatcher thread from spinning unnecessarily.
Bumped the RPC major version number.
Provided binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU, and NPU backends.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 6h ago · 12 views

llama.cpp b11440 re-reserves scheduler when NextN extraction flags change

The llama.cpp project released version b11440, which includes a fix for speculative MTP initialization. The update ensures the scheduler is re-reserved when NextN extraction flags change, preventing issues with unmasked extraction.

github llama.cpp · 1d ago · 21 views

llama.cpp b11413 adds Vulkan sparse flash attention for quantized K/V

The llama.cpp b11413 release introduces a new Vulkan implementation of sparse flash attention that supports quantized Key and Value tensors. This update addresses performance issues in the previous compaction method, which was too costly for decoding tasks with large context windows.

github llama.cpp · 2d ago · 22 views

webgpu adds MMVQ support for Q1_0/Q5_0/Q5_1/Q3_K/Q5_K/Q6_K/MXFP4

The webgpu component has added support for MMVQ (Multi-Mode Vector Quantization) formats, specifically Q1_0, Q5_0, Q5_1, Q3_K, Q5_K, Q6_K, and MXFP4.

media Hugging Face Forums · 2d ago · 27 views

Eqx-zoo provides verified Equinox ports of Hugging Face models

The author has built eqx-zoo, a library that loads Hugging Face Hub checkpoints directly as plain Equinox modules and verifies each model against the transformers library. The project currently supports Llama, Qwen2, Qwen3, Qwen3-MoE for generation, and BERT, RoBERTa, XLM-RoBERTa for embeddings.

media Hugging Face Forums · 5d ago · 28 views

AiiStream Q3.6 enables running Qwen3.6-35B-A3B on 8-16 GB Macs

The author has released AiiStream Q3.6, an open-source technique that allows the Qwen3.6-35B-A3B model to run on Apple Silicon devices with only 8 to 16 GB of RAM. By keeping routed experts on SSD and using early loading predictions, the method reduces MLX memory usage to approximately 1.7 GB while maintaining byte-identical outputs to the standard mlx_lm implementation.