github llama.cpp · 1h ago · inference

llama.cpp b10242 adds CUDA backend sampler for penalties

from English

The llama.cpp project released version b10242, introducing a new backend sampler for penalties that allows penalty handling to be offloaded to the GPU. This change enhances the `common_sampler_init` function and updates the `llama_sampler_penalties` structure to inherit from `llama_sampler_backend`, enabling frequency and presence adjustments on the backend.

The implementation includes logic for backend initialization, application of penalties, and support for top-k penalties in backend sampling.
Numerical stability is improved by preserving masked logits as -Inf to prevent NaN generation during sampling.
Validation ensures repeat penalty values are finite and greater than 0, with tests added for invalid parameters.
The release provides binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, HIP, OpenVINO, SYCL), and openEuler architectures.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10241 fixes CUDA SMEM data-races in block_reduce

The llama.cpp project released version b10241, which addresses a critical bug in the CUDA backend. The update resolves data-races that occurred when reusing shared memory (SMEM) during block reductions.

media Hugging Face Forums · 6h ago · 1 view

Quesma publishes curated map of 218 tools and papers for LLM token cost efficiency

Quesma has released an open, curated list containing 218 entries that track where tokens are spent in LLM and agent workflows. The resource covers monitoring, semantic caching, cheap local models, context engineering, and cost governance.

github llama.cpp · 9h ago

llama.cpp b10238 adds MTP support for Qwen3-Next

The llama.cpp project released build b10238, which introduces Multi-Token Prediction (MTP) support for the Qwen3-Next model family. This update includes specific implementation changes to handle MTP layers and fixes related to Python type checking within the codebase.

github llama.cpp · 10h ago · 3 views

llama.cpp b10237 adds MTP support for DeepSeek V3.2

The llama.cpp project released build b10237, which introduces Multi-Token Prediction (MTP) support for the DeepSeek V3.2 model.

media Hugging Face Forums · 12h ago · 2 views

Qxern-v6 uses latent tokens and AST sidecar for faster, accurate code transfer

The Qxern-v6 system enables two LLMs to communicate via 32 compressed latent tokens while preserving exact symbol accuracy through an adaptive deterministic Abstract Syntax Tree (AST) sidecar. Built by a 15-year-old developer, the architecture uses Qwen2.5-Coder-1.5B to compress code and a frozen Qwen3.5-0.8B decoder to interpret it without seeing raw text.