github llama.cpp · 1h ago · inference

llama.cpp 0.3.0 adds dots3-note model and tensor-split for DeepSeek 4

from English

llama.cpp 0.3.0 introduces support for the dots3-note multimodal model with a new DSA-ISWA KV cache type, along with multi-token prediction (MTP) support for GLM-4.5-Air. The release also adds tensor-split capabilities for DeepSeek 4 and fixes multi-sequence rollback issues.

ggml is bumped to v0.22.0, adding tensor-split support to the meta-backend and per-op Metal kernels with parallel compilation.
mtmd gains dots3-note vision/audio support, WebP decoding via ffmpeg, and a Pillow-accurate resize algorithm.
DeepSeek 4 receives tensor-split mode via the `-sm tensor` flag and fixes for rollback with multiple sequences.
The server adds a `LLAMA_SERVER_SLOTS_N_DIFF` debug knob, while the web UI gains tabbed chat navigation.

This update expands llama.cpp's multimodal capabilities and improves performance and stability for specific model architectures like DeepSeek 4 and GLM-4.5-Air.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Multimodal

Read original

media r/LocalLLaMA · 41d ago · 11 views

OvisOCR2 tops OmniDocBench as first end-to-end document parsing model

ATH-MaaS released OvisOCR2, a 0.8B vision-language model post-trained from Qwen3.5-0.8B that serves as the first end-to-end model to top the OmniDocBench leaderboard with a score of 96.58 on v1.6.

lab Hugging Face Blog · just now · 1 view Live

Papers with Code uses Hugging Face services for hybrid paper search

Papers with Code has implemented a hybrid search system combining keyword and semantic retrieval to help users find relevant AI research papers. The architecture leverages Hugging Face Jobs, Storage Buckets, and Inference Endpoints to manage the embedding pipeline and serve low-latency queries.

media Hugging Face Forums · 13h ago · 2 views

NVIDIA Nemotron-3-Omni GGUFs with working audio + video in llama.cpp (9 tested quants, one-pass A/V)

A developer has released a complete GGUF package for NVIDIA’s Nemotron-3-Nano-Omni-30B model, adding previously missing audio and video support to the llama.cpp ecosystem. The release includes nine quantizations ranging from Q3_K_M to BF16 and a unified projector file that integrates both C-RADIO vision and Parakeet audio encoders.

github llama.cpp · 14h ago · 1 view

llama.cpp b10615 adds per-device tuned flash-attn vec for Metal

The llama.cpp project released version b10615, introducing per-device tuned quantized (Q, NE) flash-attention vector operations for the Metal backend.

media Hugging Face Forums · 14h ago · 1 view

Unified GGUF and llama.cpp fork enable audio and video for Nemotron-3-Nano-Omni

The author addresses silent failures in popular GGUF versions of Nemotron-3-Nano-Omni-30B, where audio and video inputs were ignored due to incomplete projector files and missing inference graphs. To fix this, a unified mmproj file containing the vision tower, full Parakeet/FastConformer audio encoder, and temporal video embedder has been released alongside a specialized llama.cpp fork.