github llama.cpp · 1h ago · inference

llama.cpp b10509 adds ggml_rope_set_offset support to Metal, CUDA, and Vulkan

from English

The llama.cpp project released build b10509, which introduces support for the `ggml_rope_set_offset` function across multiple hardware backends. This update includes CPU, Metal, CUDA, and Vulkan implementations of the new operation.

Added `ggml_rope_set_offset` parameters and CPU kernel implementation.
Implemented Metal kernel support for the new rope offset functionality.
Enabled CUDA support for `ggml_rope_set_offset` while removing the `supports_op` guard.
Added Vulkan backend support for `ggml_rope_set_offset` on both Linux and Windows.
Removed inplace optimization from the relevant operations.

The release provides precompiled binaries for macOS, iOS, Linux, Android, Windows, and openEuler across various architectures including CPU, GPU, and specialized accelerators like OpenVINO and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 6d ago · 18 views

llama.cpp b10427 fuses SYCL FFN ops for up to 12.4% throughput gain

The llama.cpp project released version b10427, which includes a performance optimization for Intel Arc GPUs via SYCL. The update implements a fusion of the gate, up, and GLU operations within the q4_K dense feed-forward network (FFN) layer.

github llama.cpp · 7d ago · 23 views

llama.cpp b10413 adds auto-detection of draft spec type from GGUF metadata

The llama.cpp project released version b10413, which introduces automatic detection of the speculative decoding specification type directly from draft GGUF model metadata.

github llama.cpp · 7d ago · 26 views

llama.cpp b10410 removes separate fp32 type promotion in gemm non-oneDNN path

The llama.cpp project has released version b10410, which includes a code change to the SYCL backend. This update removes the separate fp32 type promotion in the GEMM non-oneDNN path and instead uses automatic fp16 promotion.

github llama.cpp · 7d ago · 25 views

llama.cpp b10408 adds Q3_K ESIMD kernel and moves control to runtime

The llama.cpp project released version b10408, which introduces a DMMV Q3_K ESIMD kernel for Intel GPUs. This release also includes new Q4_K and Q6_K ESIMD kernels and refactors the codebase to allow ESIMD control at runtime rather than compile time.

github llama.cpp · 9d ago · 23 views

llama.cpp b10353 fixes silent wrong results in CUDA and Metal ROLL kernels

The llama.cpp project released build b10353, which addresses a critical bug in the ggml_roll operation on CUDA and Metal backends. Previously, permuted (non-contiguous) source tensors produced silently incorrect results because these backends ignored stride information.