github llama.cpp · just now · inference

llama.cpp b10780 enhances SYCL API to support peer-to-peer copy

from English

The llama.cpp project has released version b10780, which includes a specific enhancement to the SYCL backend. This update modifies the API to enable peer-to-peer copy capabilities for SYCL devices.

The release provides binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x), Windows (x64 and arm64), Android, and openEuler.
Supported hardware backends include CPU, Vulkan, ROCm 10.0, OpenVINO, CUDA 12/13, and SYCL.
macOS Apple Silicon builds with KleidiAI are currently disabled in this version.
Windows arm64 CUDA 13 support is marked as a preview release.

This update allows users to utilize the new peer-to-peer copy feature within the SYCL implementation.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp adds support for NVIDIA Nemotron-3-Puzzle-75B-A9B

llama.cpp introduces support for the NVIDIA Nemotron-3-Puzzle-75B-A9B model, enabling inference for this Mixture of Experts architecture. The update implements infrastructure for variable-per-layer expert feed-forward sizes and top-k routing, which are required because the model has 40 MoE layers with distinct n_ff_exp and top-k values.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10777 reduces redundant work in Q4_K multi-column MMVQ

The llama.cpp project released build b10777, which includes optimizations to the SYCL backend for Q4_K multi-column matrix-vector multiplication (MMVQ). The update focuses on reducing redundant computational work through weight unpacking and activation reuse strategies.

github llama.cpp · 14h ago · 2 views

llama.cpp adds support for NVIDIA Nemotron-3-Puzzle-75B-A9B

llama.cpp b10777 reduces redundant work in Q4_K multi-column MMVQ

llama.cpp b10766 adds DeepSeek 4 vision support

llama.cpp b10762 adds DeepSeek-V4-Flash-Vision-Exp support

llama.cpp b10758 adds MUL_MAT and MUL_MAT_ID fusion for Hexagon