github llama.cpp · 2h ago · inference

llama.cpp b10094 infers speculative type from draft repo sidecars

from English

The llama.cpp project released version b10094, introducing a feature that automatically detects the speculative decoding type from draft repository sidecars. Previously, when using the `-hfd` flag to point to a repository containing mtp-, dflash-, or eagle3- sidecars without explicitly providing `--spec-type`, the system would incorrectly resolve the draft as a full model.

With this update, if no speculative type is specified, llama-server discovers available sidecars in the draft repository and selects the first one based on the priority order of mtp, dflash, then eagle3. This allows users to run commands like `llama-server -hf repo:Q3_K_M -hfd repo:Q8_0` without extra flags. An explicit `--spec-type` argument still disables this automatic inference.

The release includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64/arm64/s390x with CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Android arm64, Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Related articles

media Together AI Blog · 1h ago Live

Together launches Dedicated Model Inference with custom training beta

Together has released a significant update to its inference platform, introducing "Dedicated Model Inference" designed to give users complete control over performance, cost, and quality for open-weight models. The update also announces a closed beta for custom training capabilities, including full-weight and LoRA reinforcement learning.

github llama.cpp · 3h ago

llama.cpp b10093 fixes DeepSeek v4 template and adds drop_reasoning flag

The llama.cpp project has released build b10093, which includes specific updates to support DeepSeek v4 models. The release addresses template handling and introduces new configuration flags for this model family.

lab Tencent Hunyuan (HF) · 7h ago

Tencent open-sources Hy-MT2 multilingual translation models and IFMTBench

Tencent has released the Hy-MT2 family of "fast-thinking" multilingual translation models, including 1.8B, 7B, and 30B-A3B (MoE) sizes, along with the IFMTBench benchmark for evaluating instruction-following capabilities.

media r/LocalLLaMA · 11h ago

PaddlePaddle releases HPD-Parsing, a 1B model with hierarchical parallel decoding

PaddlePaddle has introduced HPD-Parsing, a lightweight 1-billion parameter document parsing model that utilizes a Hierarchical Parallel Decoding paradigm to improve throughput. The architecture coordinates global page structure through a main layout branch while dispatching localized content generation to concurrent branches, utilizing Progressive Multi-Token Prediction to reduce decoding steps.

lab Hugging Face Blog · 12h ago · 1 view

Diffusers adds native Nunchaku Lite 4-bit inference support

Hugging Face Diffusers now supports loading Nunchaku Lite checkpoints natively via `from_pretrained()`, eliminating the need for a separate inference engine or local CUDA compilation. This integration utilizes the SVDQuant method to run transformer layers with 4-bit weights and activations (W4A4), significantly reducing memory usage while improving inference speed.