github llama.cpp · 1h ago · inference

llama.cpp v0.2.0 release with ggml 0.21.0 and Kleidiai SME2 support

from English

The llama.cpp project has released version 0.2.0, which includes a synchronization with the ggml library bumped to version 0.21.0. This update introduces several backend improvements and new features across various hardware platforms.

Added Kleidiai SME2 F32 GEMV kernel support for enhanced performance on compatible hardware.
Enabled tensor split functionality for LFM2 and LFM2MOE models to improve multi-GPU distribution.
Implemented DSpark support for LFM2 models within the model loading pipeline.
Updated SYCL backend with Q2_K reordered MMVQ, ESIMD kernels, and fixes for Alchemist GPUs and mlock issues.
Improved OpenCL stability with fixes for Adreno A6x/A7x compilers and local size calculations.
Added the --mmproj-device argument to mtmd for specifying device placement of multimodal projections.

The release also includes CI clean-ups, UI settings navigation improvements, and various bug fixes for CUDA, Metal, and Vulkan backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 16h ago · 2 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 19h ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 19h ago · 1 view

llama.cpp b10533 adds graceful fallback for unsupported JSON schema regex patterns

The llama.cpp project has released build b10533, which includes a fix to gracefully handle unsupported regular expression patterns within JSON schemas.

github llama.cpp · 1d ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

github llama.cpp · 2d ago · 2 views

llama.cpp b10509 adds ggml_rope_set_offset support to Metal, CUDA, and Vulkan

The llama.cpp project released build b10509, which introduces support for the `ggml_rope_set_offset` function across multiple hardware backends. This update includes CPU, Metal, CUDA, and Vulkan implementations of the new operation.