github llama.cpp · 1h ago · inference

llama.cpp b10093 fixes DeepSeek v4 template and adds drop_reasoning flag

from English

The llama.cpp project has released build b10093, which includes specific updates to support DeepSeek v4 models. The release addresses template handling and introduces new configuration flags for this model family.

Fixes the DeepSeek v4 crafted template to explicitly follow reference behavior.
Adds support for the DeepSeek v4 `drop_reasoning` flag.
Hooks the DeepSeek 3.2 parser to also handle DeepSeek v4 inputs.
Includes a fix for tool result reordering.

This update ensures correct chat formatting and reasoning token handling for DeepSeek v4 models within llama.cpp.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 22h ago · 1 view

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

github llama.cpp · 2d ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

media Hugging Face Forums · 5d ago · 2 views

zhongkaifu releases TensorSharp, a native .NET LLM inference engine for GGUF models

zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

github llama.cpp · just now Live

llama.cpp b10094 infers speculative type from draft repo sidecars

The llama.cpp project released version b10094, introducing a feature that automatically detects the speculative decoding type from draft repository sidecars. Previously, when using the `-hfd` flag to point to a repository containing mtp-, dflash-, or eagle3- sidecars without explicitly providing `--spec-type`, the system would incorrectly resolve the draft as a full model.

lab Tencent Hunyuan (HF) · 5h ago

Tencent open-sources Hy-MT2 multilingual translation models and IFMTBench

Tencent has released the Hy-MT2 family of "fast-thinking" multilingual translation models, including 1.8B, 7B, and 30B-A3B (MoE) sizes, along with the IFMTBench benchmark for evaluating instruction-following capabilities.