The llama.cpp project has added an A8 Q8_0 non-MoE dp4a binary kernel for OpenCL.
The author has released AiiStream Q3.6, an open-source technique that allows the Qwen3.6-35B-A3B model to run on Apple Silicon devices with only 8 to 16 GB of RAM. By keeping routed experts on SSD and using early loading predictions, the method reduces MLX memory usage to approximately 1.7 GB while maintaining byte-identical outputs to the standard mlx_lm implementation.
A discussion on Hugging Face proposes replacing traditional text retrieval in RAG systems with LLM-native latent representations. The idea involves encoding documents into internal model states during indexing and retrieving those states for inference, potentially bypassing the cost of re-processing retrieved text.
The llama.cpp project released version b11309, which includes a key optimization for the Hexagon backend. This update adds support for safe scatter mode to the ALLREDUCE operation, aiming to improve performance and stability on Qualcomm Snapdragon devices.
The llama.cpp project released version b11310, which includes a fix for the CUDA backend's IQ4_NL dequantization logic. The update addresses a memory safety issue where threads could read and write past the end of a row when processing blocks shorter than QK_K.
The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.
We use cookies to measure traffic and improve the site. You can accept or decline analytics cookies. Privacy policy