github llama.cpp · 1h ago · inference

llama.cpp b10614 adds Metal per-op source split and parallel compile

from English

The llama.cpp b10614 release introduces a significant architectural change to the Metal backend by splitting operation sources and enabling parallel compilation. This update also includes several new kernel implementations and support for additional quantization formats.

Split metallib into 8 libraries and load them in parallel to improve build times.
Add col2im_1d op supporting f32, f16, and bf16 data types.
Implement CONV_2D_DW (depthwise convolution) support for Metal.
Add Q2_0 quantization format support.
Fuse snake activation operations (mul, sin, sqr, mul, add).
Introduce FWHT kernel for the Metal backend.

This release provides precompiled binaries for macOS, iOS, Linux, Windows, Android, and openEuler across CPU, GPU, and various accelerator backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 8h ago · 12 views

llama.cpp b10610 shortens virtual device naming in CUDA and Metal

The llama.cpp project released version b10610, which includes a change to shorten virtual device naming within the ggml backend for both CUDA and Metal platforms.

github llama.cpp · 13h ago · 17 views

llama.cpp b10605 flattens mamba2 projections to dispatch GEMM

The llama.cpp project released build b10605, which includes a key optimization for the Mamba2 architecture. The update flattens the input and output projections of Mamba2 models to dispatch General Matrix Multiplication (GEMM) operations instead of General Matrix Vector (GEMV) operations.

github llama.cpp · 1d ago · 13 views

llama.cpp b10603 adds MTP support for GLM-4.5-Air

The llama.cpp project has released build b10603, which introduces Multi-Token Prediction (MTP) support for the GLM-4.5-Air model.

github llama.cpp · 3d ago · 5 views

ggml optimizes concat op by replacing per-element memcpy with row-level memcpy

The ggml library has optimized its concatenation operation by replacing per-element memory copies with row-level memcpy operations. This change is part of the b10578 release of llama.cpp, which includes various platform-specific binaries and updates to the underlying computation graph.

github llama.cpp · 3d ago · 3 views

llama.cpp b10576 adds Q2_K reordered MMVQ and ESIMD kernels for SYCL

The llama.cpp project has released build b10576, which re-adds Q2_K reordered MMVQ and ESIMD kernels for SYCL support. This change effectively reverts a previous commit that had removed these specific kernel implementations.