github llama.cpp · 1h ago · inference

llama.cpp b10610 shortens virtual device naming in CUDA and Metal

from English

The llama.cpp project released version b10610, which includes a change to shorten virtual device naming within the ggml backend for both CUDA and Metal platforms.

The update modifies how virtual devices are named in CUDA and Metal backends to reduce verbosity.
macOS Apple Silicon builds with KleidiAI support remain disabled in this release.
Binaries are provided for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm 7.14, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenCL, ROCm 7.14), Android, and openEuler.

This release provides updated binaries for various hardware accelerators and operating systems while refining internal device identification strings.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 6h ago · 17 views

llama.cpp b10605 flattens mamba2 projections to dispatch GEMM

The llama.cpp project released build b10605, which includes a key optimization for the Mamba2 architecture. The update flattens the input and output projections of Mamba2 models to dispatch General Matrix Multiplication (GEMM) operations instead of General Matrix Vector (GEMV) operations.

github llama.cpp · 20h ago · 12 views

llama.cpp b10603 adds MTP support for GLM-4.5-Air

The llama.cpp project has released build b10603, which introduces Multi-Token Prediction (MTP) support for the GLM-4.5-Air model.

github llama.cpp · 2d ago · 6 views

ggml optimizes concat op by replacing per-element memcpy with row-level memcpy

The ggml library has optimized its concatenation operation by replacing per-element memory copies with row-level memcpy operations. This change is part of the b10578 release of llama.cpp, which includes various platform-specific binaries and updates to the underlying computation graph.

github llama.cpp · 2d ago · 4 views

llama.cpp b10576 adds Q2_K reordered MMVQ and ESIMD kernels for SYCL

The llama.cpp project has released build b10576, which re-adds Q2_K reordered MMVQ and ESIMD kernels for SYCL support. This change effectively reverts a previous commit that had removed these specific kernel implementations.

github llama.cpp · 2d ago · 6 views

llama.cpp b10577 fixes draft-mtp with embeddings

The llama.cpp project released build b10577, which includes a fix for draft-mtp when used with embeddings. This update addresses issues #26352 and #27299.