github llama.cpp · 1d ago · inference

llama.cpp b10354 fixes CPU affinity mask on Android

from English

The llama.cpp project has released version b10354, which includes a fix for the CPU affinity mask being ignored on Android devices.

The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64, arm64, s390x), Windows (CPU, CUDA 12/13, Vulkan, OpenCL, ROCm, SYCL, OpenVINO), and Android (arm64).
KleidiAI support for macOS Apple Silicon remains disabled in this build.
openEuler builds are largely disabled, with only specific x86 and aarch64 configurations available.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1d ago

llama.cpp b10344 adds MTP support for Nemotron models

The llama.cpp project has released build b10344, which introduces Multi-Token Prediction (MTP) support specifically for the Nemotron and Nemotron Nano models.

github llama.cpp · 2d ago

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.

github llama.cpp · 2d ago

llama.cpp b10336 release with ggml-webgpu refactor

The llama.cpp project has released version b10336, which includes a refactor of several wgsl files and simplification of flash_attn wgsl within the ggml-webgpu component.

media Hugging Face Forums · 3d ago · 8 views

vLLM Launcher provides a Windows desktop workbench for local LLM inference

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.

github llama.cpp · 3d ago · 2 views

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.