github llama.cpp · just now · inference

llama.cpp b10418 adds SYCL host pinned memory support

from English

The llama.cpp project released version b10418, introducing support for host pinned memory to improve SYCL Host-to-Device memory access. This update addresses thread-safety issues within the backend implementation.

Adds ggml_backend_sycl_host_buffer_type_get_max_size support for host pinned memory.
Fixes a thread-safe issue in the SYCL backend.
Includes binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL FP32/FP16), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 7.14), and openEuler.
Provides a unified UI package for all platforms.

The release enables faster data transfer between host and device memory for SYCL-compatible hardware while ensuring stability through improved thread safety.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1h ago Live

llama.cpp OpenVINO backend adds Qwen3.5 support and memory optimizations

The llama.cpp project has merged significant updates to its OpenVINO backend, primarily enabling support for the Qwen3.5 model family and introducing several memory optimization techniques.

github llama.cpp · 4h ago · 1 view

llama.cpp b10414 adds TQ2_0 support and optimizes mul_mv kernel

The llama.cpp project released build b10414, introducing support for the GGML_TYPE_TQ2_0 ternary quantization type in the Metal backend. This update also includes optimizations to the contiguous tensor (cont) mul_mv kernel.

github llama.cpp · 3d ago · 1 view

llama.cpp b10354 fixes CPU affinity mask on Android

The llama.cpp project has released version b10354, which includes a fix for the CPU affinity mask being ignored on Android devices.

github llama.cpp · 3d ago · 2 views

llama.cpp b10344 adds MTP support for Nemotron models

The llama.cpp project has released build b10344, which introduces Multi-Token Prediction (MTP) support specifically for the Nemotron and Nemotron Nano models.

github llama.cpp · 3d ago · 1 view

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.