github llama.cpp · 1h ago · inference

ggml optimizes concat op by replacing per-element memcpy with row-level memcpy

from English

The ggml library has optimized its concatenation operation by replacing per-element memory copies with row-level memcpy operations. This change is part of the b10578 release of llama.cpp, which includes various platform-specific binaries and updates to the underlying computation graph.

Replaces per-element memcpy with row-level memcpy in the concat operation.
Fixes concat offsets for the new row-level copy mechanism.
Adds and moves concat block size asserts while removing redundant checks.
Provides builds for macOS, Linux, Windows, Android, and openEuler across CPU, GPU, and specialized backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b10576 adds Q2_K reordered MMVQ and ESIMD kernels for SYCL

The llama.cpp project has released build b10576, which re-adds Q2_K reordered MMVQ and ESIMD kernels for SYCL support. This change effectively reverts a previous commit that had removed these specific kernel implementations.

github llama.cpp · 2h ago · 1 view

llama.cpp b10577 fixes draft-mtp with embeddings

The llama.cpp project released build b10577, which includes a fix for draft-mtp when used with embeddings. This update addresses issues #26352 and #27299.

github llama.cpp · 11h ago · 2 views

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2

The llama.cpp project released build b10568, which updates the model handling to use the `ggml_rope_set_offset()` function. This change is partially applied to support DeepSeek 2.

github llama.cpp · 16h ago · 6 views

llama.cpp v0.2.0 release with ggml 0.21.0 and Kleidiai SME2 support

The llama.cpp project has released version 0.2.0, which includes a synchronization with the ggml library bumped to version 0.21.0. This update introduces several backend improvements and new features across various hardware platforms.

github llama.cpp · 1d ago · 5 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.