github llama.cpp · just now · inference

llama.cpp b10900 enables topk_moe fusion for prefill via Vulkan

from English

The llama.cpp project has released version b10900, which includes a technical update to the Vulkan backend. This release adds an allocation dependency specifically to enable the topk_moe fusion optimization during the prefill phase.

The primary code change is the addition of an allocation dependency in the Vulkan implementation to support topk_moe fusion for prefill operations.
Binaries are provided for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm 10.0, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 10.0), Android, and iOS.
KleidiAI support for macOS Apple Silicon is disabled in this build.

This update provides users with optimized prefill performance on Vulkan-compatible hardware through the newly enabled fusion technique.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media r/LocalLLaMA · just now Live

CyberTiel 35B-A3B uncensored 4-bit quant beats Opus 4.6 medium on codebase issues

An anonymous researcher has released CyberTiel 35B-A3B, an uncensored 4-bit quantized model designed for offensive security research and agentic software engineering. The model is built by abliteration on the TielCoder base and quantizing with an improved imatrix derived from a curated cybersecurity corpus.

github llama.cpp · 1h ago Live

llama.cpp b10899 adds Vulkan optimizations for Qwen and split_k support

The llama.cpp project released version b10899, introducing specific performance improvements to its Vulkan backend. The update focuses on optimizing matrix operations for the Qwen model family and enhancing general small M performance.

media Together AI Blog · 1h ago Live

Together AI optimizes ThunderKittens GEMMs for NVIDIA Vera Rubin NVL72

The kernels team at Together AI has optimized the ThunderKittens library to leverage new features in the NVIDIA Vera Rubin NVL72 platform, specifically targeting improved performance for NVFP4 and FP8 matrix multiplications.

github llama.cpp · 4h ago · 1 view

llama.cpp b10896 fixes DFlash decoding failure with vision models

The llama.cpp project released build b10896, which addresses a critical issue where the drafter memory failed to allocate new tokens when using DFlash with vision models. This failure occurred because images reported a fixed offset, preventing proper token allocation.

media Hugging Face Forums · 7h ago · 1 view

Developer seeks advice on designing cost-effective voice AI architecture with ~500ms first-token latency

An AI developer is asking for architectural advice to build a real-time voice AI system that achieves approximately 500 ms or less first-token latency while remaining cost-effective and scalable.