github llama.cpp · 9h ago · inference

llama.cpp b11351 adds alloc_buffer_n and get_alloc_size_n to ggml backend buffer type interface

from English

The llama.cpp project released build b11351, introducing new methods to the ggml backend buffer type interface. The update adds `alloc_buffer_n` and `get_alloc_size_n` to allow for more granular control over buffer allocation and size querying.

Added `ggml_backend_buft_alloc_buffer_n` public API and corresponding callback to `ggml_backend_buffer_type_i`.
Implemented default handling for multi-buffer splitting and tensor allocation via `ggml_tallocr`.
Added optional `get_alloc_size_n` callback to share tensor-to-buffer planning logic.
Replaced unchecked realloc with std::vector in the default implementation.
Removed temporary `ggml_backend_meta_alloc_ctx_tensors_from_buft` function.

This release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU (CUDA, Vulkan, ROCm, OpenCL), NPU (Hexagon), and other accelerators.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

The llama.cpp project released build b11368, which introduces probabilistic sampling for simple draft and MTP (Multi-Token Prediction) workflows. This update modifies the drafting mechanism to use probabilistic selection while verifying targets via rejection sampling.

github llama.cpp · 5h ago · 1 view

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

The llama.cpp project released build b11362, which introduces a new tensor API flash attention kernel supporting F16 key-value (KV) storage. This update is available across macOS, Linux, Windows, Android, and iOS platforms with various backend options including CUDA, Vulkan, and ROCm.

github llama.cpp · 1d ago · 1 view

llama.cpp b11368 adds probabilistic sampling for simple draft and MTP

llama.cpp b11362 adds tensor API flash attention kernel for F16 KV

llama.cpp b11338 adds shared strided DMA copy for CPY and CONCAT

llama.cpp b11335 routes sm70 to Turing MMVQ table for Volta performance

llama.cpp b11333 adds WebGPU bfloat16 support for MUL_MAT operations