github llama.cpp · 1h ago · inference

llama.cpp b10285 adds multi-row batching for deepseek-ocr

from English

The llama.cpp project has released build b10285, which introduces support for multi-row batching in the mtmd module specifically for deepseek-ocr models. This update allows the system to weave deepseek-ocr rows in a single operation rather than processing them individually.

mtmd now supports multi-row batching for deepseek-ocr.
The implementation weaves deepseek-ocr rows in one shot instead of per row.
Binaries are available for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, OpenVINO, SYCL, HIP), and openEuler.

This release provides updated binaries across multiple platforms and hardware accelerators for users requiring the new deepseek-ocr processing capabilities.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

The llama.cpp project has released version b10282, which introduces new monitoring capabilities for speculative decoding. The server now includes spec-decode counters in its /metrics endpoint, allowing users to track performance metrics related to this feature.

github llama.cpp · 16h ago

llama.cpp b10275 decodes Windows OEM output to UTF-8 in built-in tools

The llama.cpp project released build b10275, which fixes a character encoding issue on Windows where accented characters were silently lost. The update ensures that text from child processes writing in the OEM code page is correctly decoded to UTF-8 before reaching the JSON layer.

github llama.cpp · 20h ago

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

llama.cpp b10275 decodes Windows OEM output to UTF-8 in built-in tools

llama.cpp b10273 resolves sampler context length and removes full-context windows

llama.cpp b10269 fixes dflash wo_a reshape on load

Open-source Decay-Gated O(N) Causal Linear Attention with fused Triton kernel