github llama.cpp · 1h ago · inference

llama.cpp b10201 improves flash attention for quantized KV at long contexts

from English

The llama.cpp project has released version b10201, which includes a key improvement to the ggml-webgpu backend. This update enhances flash attention performance when handling quantized key-value caches during long context generation.

Improved flash attention vector processing for quantized KV cache in ggml-webgpu.
Fixed bugs related to value type checking and updated comments.
Resolved build errors caused by rebasing.
Provided binaries for macOS (Apple Silicon, Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, HIP, OpenVINO, SYCL), and openEuler.

This release allows users to run llama.cpp across a wide range of hardware platforms while benefiting from optimized attention mechanisms for longer sequences.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10202 release with SYCL RMS_NORM fusion

The llama.cpp project has released version b10202, introducing a SYCL optimization that fuses the RMS_NORM and MUL operations.

github llama.cpp · 16h ago · 1 view

llama.cpp b10198 adds Vulkan support for quantized concat

The llama.cpp project has released build b10198, which introduces support for quantized concatenation operations within the Vulkan backend.

github llama.cpp · 2d ago · 1 view

llama.cpp b10180 adds contiguous fast path and 32-bit index math for SYCL unary ops

The llama.cpp project has released build b10180, introducing performance optimizations for the SYCL backend. The update implements a contiguous fast path and switches elementwise index math to use fastdiv with 32-bit integers.

github llama.cpp · 4d ago · 3 views

llama.cpp b10155 adds MiMo-V2.5 audio support

The llama.cpp project has released build b10155, which introduces support for the MiMo-V2.5 audio input model. This update includes a GGUF converter for MiMo audio and implements the necessary C++ backend logic to handle RVQ-based audio models.

github llama.cpp · 4d ago · 2 views

llama.cpp b10150 adjusts ggml offloading logic and fixes dsv4 graph

The llama.cpp project has released build b10150, which includes adjustments to the ggml logic for offloading operations to the weight's backend. This update also addresses fixes for the llama dsv4 graph.