github llama.cpp · 2h ago · inference

llama.cpp 0.4.0 adds Qwen3.8-Flash-Next and Nemotron-3-Puzzle support

from English

llama.cpp version 0.4.0 introduces initial architecture support for the Qwen3.8-Flash-Next and NVIDIA Nemotron-3-Puzzle-75B-A9B models, alongside a bump of the underlying ggml library to version 0.23.0.

Added on-demand lazy tensor reading and per-slot server context limits.
Updated ggml to 0.23.0 with sparse flash attention and Apple RDMA transport support.
Introduced video input options, n-gram history lookup, and quantizer RAM caps.
Enabled `preserve_reasoning` by default in the server and improved multimodal tokenization helpers.

These changes expand model compatibility and provide developers with finer control over memory usage and server resource allocation.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Multimodal

Read original

github llama.cpp · 10d ago · 42 views

llama.cpp 0.3.0 adds dots3-note model and tensor-split for DeepSeek 4

llama.cpp 0.3.0 introduces support for the dots3-note multimodal model with a new DSA-ISWA KV cache type, along with multi-token prediction (MTP) support for GLM-4.5-Air. The release also adds tensor-split capabilities for DeepSeek 4 and fixes multi-sequence rollback issues.

media r/LocalLLaMA · 51d ago · 17 views

OvisOCR2 tops OmniDocBench as first end-to-end document parsing model

ATH-MaaS released OvisOCR2, a 0.8B vision-language model post-trained from Qwen3.5-0.8B that serves as the first end-to-end model to top the OmniDocBench leaderboard with a score of 96.58 on v1.6.

github llama.cpp · 18h ago · 1 view

llama.cpp b10795 fuses SYCL operations

The llama.cpp project released version b10795, which includes a new optimization for the SYCL backend. This update enables the fusion of RMS_NORM+MUL+ADD and ADD+ADD residual chains under the GGML_SYCL_ENABLE_FUSION flag.

github llama.cpp · 1d ago · 3 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

media r/LocalLLaMA · 1d ago · 4 views

IFM releases K2-Horizon-MoVA-36B-A4B, a sparse MoE model with 4B active parameters

IFM has released the final checkpoint for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model featuring Mixture-of-Values attention that stores 36B parameters but runs only 4B per token. The release includes GGUF formats and links to smaller variants in the K2-Horizon family.