github llama.cpp · 5h ago · inference

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

from English

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.

Adds tensor FA kernels for specific dimensions: DK=DV=512, DK=576/DV=512, and DK=192/DV=128.
The new kernel supports attention sinks, ALiBi, and logit softcap mechanisms.
Pre-built binaries are provided for CPU, GPU (CUDA 12/13, Vulkan, ROCm 10.0), and NPU (Snapdragon Hexagon) backends.

This release enables more efficient attention computation on supported hardware configurations.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

github llama.cpp · 9h ago · 4 views

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

llama.cpp b11351 adds alloc_buffer_n and get_alloc_size_n to ggml backend buffer type interface

llama.cpp b11338 adds shared strided DMA copy for CPY and CONCAT

llama.cpp b11335 routes sm70 to Turing MMVQ table for Volta performance

llama.cpp b11333 adds WebGPU bfloat16 support for MUL_MAT operations