The webgpu component has added support for MMVQ (Multi-Mode Vector Quantization) formats, specifically Q1_0, Q5_0, Q5_1, Q3_K, Q5_K, Q6_K, and MXFP4.
This update addresses issue #29 in the llama.cpp repository to expand quantization capabilities within the webgpu environment.
The author has built eqx-zoo, a library that loads Hugging Face Hub checkpoints directly as plain Equinox modules and verifies each model against the transformers library. The project currently supports Llama, Qwen2, Qwen3, Qwen3-MoE for generation, and BERT, RoBERTa, XLM-RoBERTa for embeddings.
The author has released AiiStream Q3.6, an open-source technique that allows the Qwen3.6-35B-A3B model to run on Apple Silicon devices with only 8 to 16 GB of RAM. By keeping routed experts on SSD and using early loading predictions, the method reduces MLX memory usage to approximately 1.7 GB while maintaining byte-identical outputs to the standard mlx_lm implementation.
The llama.cpp project released build b11307, which fixes layer input ordering to preserve the original batch sequence during speculative decoding. This change ensures that token order is correctly maintained for unmasked NextN embeddings and compatible with tensor splits.
The llama.cpp project has released build b11301, which includes a fix for an integer overflow issue within the ggml and gguf components. This update addresses validation logic to prevent potential errors when handling zero-element tensors.
The llama.cpp project released build b11302, which addresses a critical concurrency issue identified by ThreadSanitizer in the CI pipeline. The update resolves a data race within the sparse attention mechanism where multiple CPU threads were incorrectly writing to the same memory elements.
We use cookies to measure traffic and improve the site. You can accept or decline analytics cookies. Privacy policy