github llama.cpp · 2h ago · inference

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2

from English

The llama.cpp project released build b10568, which updates the model handling to use the `ggml_rope_set_offset()` function. This change is partially applied to support DeepSeek 2.

The release includes binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x), Windows, Android, and openEuler.
Supported backends include CPU, Vulkan, ROCm 7.14, OpenVINO, SYCL, CUDA 12/13, and OpenCL Adreno.
macOS Apple Silicon with KleidiAI and Ubuntu x64 with ROCm 7.14 are currently disabled in this build.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 22h ago · 4 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 1d ago · 1 view

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 1d ago · 2 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

github llama.cpp · 21d ago · 38 views

llama.cpp b10208 adds oneMKL GEMM flash attention for XMX-accelerated prompt processing

The llama.cpp b10208 release introduces SYCL support for oneMKL GEMM flash attention, enabling XMX hardware acceleration for prompt processing. This change significantly improves inference speed on Intel Arc GPUs by utilizing MKL kernels instead of the default TILE path.

github llama.cpp · 30d ago · 26 views

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.