github llama.cpp · 2d ago · inference

llama.cpp b11254 fixes graph input collection for pipeline parallelism

from English

The llama.cpp project released build b11254, which includes a fix for how input tensors are collected in the computation graph. Previously, `graph_inputs` was populated during graph splitting, causing issues with pipeline parallelism where switching between batches consuming different inputs led to spurious backend ID comparisons and forced scheduler re-reservations.

The change collects inputs after the split from all input leafs of the graph, ensuring composition depends only on which inputs exist rather than which are used as sources.
This prevents the scheduler from recording smaller input sizes (e.g., `n_outputs = 0`) and aborting later via `GGML_SCHED_DEBUG_REALLOC`.
The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, CUDA 12/13, ROCm, OpenVINO, SYCL, Snapdragon), Android, Windows (CPU, OpenCL, CUDA, Vulkan, OpenVINO, SYCL, ROCm), and openEuler.

This fix stabilizes execution in pipeline parallelism scenarios by eliminating unnecessary graph recompositions caused by input order changes.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Related articles

media Hugging Face Forums · just now · 1 view Live

AiiStream Q3.6 enables running Qwen3.6-35B-A3B on 8-16 GB Macs

The author has released AiiStream Q3.6, an open-source technique that allows the Qwen3.6-35B-A3B model to run on Apple Silicon devices with only 8 to 16 GB of RAM. By keeping routed experts on SSD and using early loading predictions, the method reduces MLX memory usage to approximately 1.7 GB while maintaining byte-identical outputs to the standard mlx_lm implementation.

media Hugging Face Forums · 5h ago · 1 view

Proposal to retrieve LLM-native latent memory instead of text in RAG

A discussion on Hugging Face proposes replacing traditional text retrieval in RAG systems with LLM-native latent representations. The idea involves encoding documents into internal model states during indexing and retrieving those states for inference, potentially bypassing the cost of re-processing retrieved text.

github llama.cpp · 19h ago · 4 views

llama.cpp b11309 optimizes Hexagon ALLREDUCE with safe scatter mode

The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.

github llama.cpp · 19h ago · 1 view

llama.cpp b11310 guards iq4_nl dequantize kernel against short rows

The llama.cpp project released version b11310, which includes a fix for the CUDA backend's IQ4_NL dequantization logic. The update addresses a memory safety issue where threads could read and write past the end of a row when processing blocks shorter than QK_K.

github llama.cpp · 22h ago · 1 view

llama.cpp b11307 preserves batch order for speculative decoding

The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.