github llama.cpp · 1h ago · inference

llama.cpp b10142 adds preliminary MiniMax-M3 vision support

from English

The llama.cpp project released build b10142, which introduces preliminary vision support for the MiniMax-M3 model. This update implements a text-only port that reuses existing components from MiniMax-M2, including GQA with per-head QK-norm and partial rotary, DeepSeek-V3 style experts, and swigluoai activation.

The implementation includes the MiniMax-M3 vision tower (mmproj + clip graph) while dropping MTP heads and sparse attention support.
Performance optimizations include decomposing slow CPU operations into GPU/CPU ops for massive speedups on long contexts and rewriting the indexer op to be CUDA native.
A unified 4-way + decode path reduces nodes per layer from ~50 to ~25, shrinking compute buffers by approximately 20%.
Measured effects show decode speeds increasing from 6.2–7.15 t/s to 7.7–7.8 t/s on expert offload setups, with prefill becoming around 10% faster.

All GGUFs generated before this change must be regenerated to ensure compatibility with the new architecture and conversion logic.

Importance 1/3 Trust 1/3 llama.cpp MiniMax Inference efficiency Multimodal

Related articles

media r/LocalLLaMA · 12d ago · 1 view

OvisOCR2 tops OmniDocBench as first end-to-end document parsing model

ATH-MaaS released OvisOCR2, a 0.8B vision-language model post-trained from Qwen3.5-0.8B that serves as the first end-to-end model to top the OmniDocBench leaderboard with a score of 96.58 on v1.6.

arxiv arXiv cs.AI · 13d ago

Output token count, not visual processing, dominates energy in edge VLM inference

A systematic energy profiling of on-device Vision-Language Model (VLM) inference across five models and two hardware platforms reveals that the primary energy bottleneck is output generation rather than visual perception. The study overturns the common assumption that reducing visual tokens is the most effective efficiency strategy, demonstrating instead that controlling output length is critical for power savings.

arxiv arXiv cs.CL · 20d ago

OpenGlass introduces sensing-computing split architecture for local MLLM-driven visual assistance

Researchers present OpenGlass, an open-source system that splits sensing and computing to provide low-latency multimodal visual assistance for blind and low-vision users. The architecture uses an ESP32-based glasses unit to capture visual context while a nearby consumer device performs local MLLM inference, keeping raw data on user-controlled devices by default.

arxiv arXiv cs.AI · 20d ago

VisionAId: Offline-first Android assistant with personalized object retrieval

VisionAId is an offline-first Android application designed to assist people with visual impairment by turning a commodity smartphone into a real-time visual assistant. The system integrates six on-device deep learning models running through ONNX Runtime, including metric monocular depth estimation, instance segmentation, and face detection.

media r/LocalLLaMA · 22d ago · 2 views

NVFP4 Qwen3.6-35B-A3B on Blackwell achieves ~2000 tps with 30 concurrent streams

A user demonstrates running the NVFP4-quantized Qwen3.6-35B-A3B model on an RTX Pro 6000 Blackwell GPU, achieving approximately 2000 tokens per second in aggregate throughput while handling 30 concurrent image captioning streams. The configuration utilizes vLLM with the FLASHINFER attention backend and prefix caching to manage high concurrency. The Mixture of Experts (MoE) architecture activates only about 53-61% of experts even at high concurrency levels, allowing it to outperform dense models despite its larger parameter count. This setup proves that NVFP4 quantization on Blackwell hardware can efficiently handle multimodal workloads with significant parallelism without exhausting VRAM.