github llama.cpp · 1d ago · inference

llama.cpp b10355 adds multi-output backend sampling with token speculation

from English

The llama.cpp project released version b10355, introducing support for multi-output backend sampling. This update enables backend sampling combined with token speculation and adds a numeric context parameter to declare the maximum outputs per sequence.

Enable backend sampling with token speculation
Clamp the mask sum before converting it into the sampled index
Add a numeric context parameter declaring the maximum outputs one sequence
Fix CPU and backend sampling mismatches and match dist between CPU and GPU
Simplify changes and fix tests on Vulkan

The release provides binaries for macOS, iOS, Linux, Android, Windows, and openEuler across various hardware backends including CPU, CUDA, ROCm, OpenVINO, SYCL, HIP, and Vulkan.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 7h ago

llama.cpp b10361 fixes SWA for EXAONE 4.5 and skips TENSOR_SKIP in metadata path

The llama.cpp b10361 release addresses a critical bug where Sliding Window Attention (SWA) was not enabled for the LGAI-EXAONE 4.5 model due to incorrect parameter ordering during loading.

lab Hugging Face Blog · 11h ago

ALTK-Evolve achieves same-or-better accuracy than ACE at a fraction of inference cost

ALTK-Evolve, a system for agentic memory that converts agent trajectories into reusable guidelines without weight updates, matches or exceeds the accuracy of ACE while using significantly fewer tokens. Unlike ACE, which injects a comprehensive playbook on every step, ALTK-Evolve uses selective retrieval to deliver only relevant guidelines based on model capacity.

github vLLM · 14h ago

vLLM v0.27.1 adds support for quantized DSpark Markov heads

vLLM has released version 0.27.1, a patch update built on top of the previous v0.27.0 release.

lab NVIDIA Research · 15h ago · 10 views

DSTP mitigates visual token pruning failures in complex MLLM reasoning

Researchers identify Relevant Visual Information Shift (RVIS) during decoding as the primary cause for the failure of existing visual token pruning methods in Multimodal Large Language Models (MLLMs). To address this, they propose Decoding-stage Shift-aware Token Pruning (DSTP), a training-free framework that aligns visual tokens with shifting reasoning requirements.

media MarkTechPost · 16h ago · 1 view

webAI releases TwIL-LM, a 1.7B and 3B formal-logic model family for local autoformalization

webAI has released TwIL-LM, a two-model family of formal-logic reasoners with 1.7B and 3B parameters designed for autoformalization on local hardware. The models translate English into first-order logic and check conclusion validity, with the 3B variant achieving a macro gate score of 0.4218 on in-domain formal logic tasks.