github llama.cpp · 2h ago · inference

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

from English

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

The drafter now uses probabilistic sampling, with target verification handled by rejection sampling.
Stale spec_draft_q buffers are dropped before the drafting process begins.
Grammar-constrained requests now support rejection sampling and fallback to argmax sampling.
A new flag enables probabilistic draft sampling, defaulting to greedy behavior.
The draft sampler no longer shares the target's RNG stream, and distributions are renormalized after masking.

This release provides binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CUDA, ROCm, Vulkan, OpenVINO, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 5h ago · 1 view

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.

github llama.cpp · 9h ago · 4 views

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

llama.cpp b11351 adds alloc_buffer_n and get_alloc_size_n to ggml backend buffer type interface

llama.cpp b11338 adds shared strided DMA copy for CPY and CONCAT

llama.cpp b11335 routes sm70 to Turing MMVQ table for Volta performance

llama.cpp b11333 adds WebGPU bfloat16 support for MUL_MAT operations