github llama.cpp · 1h ago · inference

llama.cpp b10615 adds per-device tuned flash-attn vec for Metal

from English

The llama.cpp project released version b10615, introducing per-device tuned quantized (Q, NE) flash-attention vector operations for the Metal backend.

Added 53 f16 flash-attn vector instantiations, increasing the total from 80 to 133.
Implemented a tuning table and dispatch wiring with shared memory capacity fallback.
Extended vector tuning to support quantized KV caches.
Included tuned parameters for M1 Pro, M2 Ultra, and M5 Max devices.

This update optimizes performance on Apple Silicon hardware by applying specific tuning results to the Metal backend.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10614 adds Metal per-op source split and parallel compile

The llama.cpp b10614 release introduces a significant architectural change to the Metal backend by splitting operation sources and enabling parallel compilation. This update also includes several new kernel implementations and support for additional quantization formats.

github llama.cpp · 10h ago · 12 views

llama.cpp b10610 shortens virtual device naming in CUDA and Metal

The llama.cpp project released version b10610, which includes a change to shorten virtual device naming within the ggml backend for both CUDA and Metal platforms.

github llama.cpp · 15h ago · 17 views

llama.cpp b10605 flattens mamba2 projections to dispatch GEMM

The llama.cpp project released build b10605, which includes a key optimization for the Mamba2 architecture. The update flattens the input and output projections of Mamba2 models to dispatch General Matrix Multiplication (GEMM) operations instead of General Matrix Vector (GEMV) operations.

github llama.cpp · 1d ago · 13 views

llama.cpp b10603 adds MTP support for GLM-4.5-Air

The llama.cpp project has released build b10603, which introduces Multi-Token Prediction (MTP) support for the GLM-4.5-Air model.

github llama.cpp · 3d ago · 5 views

ggml optimizes concat op by replacing per-element memcpy with row-level memcpy

The ggml library has optimized its concatenation operation by replacing per-element memory copies with row-level memcpy operations. This change is part of the b10578 release of llama.cpp, which includes various platform-specific binaries and updates to the underlying computation graph.