github llama.cpp · 2h ago · inference

llama.cpp b10344 adds MTP support for Nemotron models

from English

The llama.cpp project has released build b10344, which introduces Multi-Token Prediction (MTP) support specifically for the Nemotron and Nemotron Nano models.

Adds MTP support for Nemotron Nano model.
Introduces mtp_flags configuration for the Nemotron model.
Provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, OpenVINO, SYCL, HIP), and openEuler.

This update enables users to leverage MTP capabilities with Nemotron models across a wide range of hardware platforms and operating systems.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 6h ago

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.

github llama.cpp · 10h ago

llama.cpp b10336 release with ggml-webgpu refactor

The llama.cpp project has released version b10336, which includes a refactor of several wgsl files and simplification of flash_attn wgsl within the ggml-webgpu component.

media Hugging Face Forums · 2d ago · 8 views

vLLM Launcher provides a Windows desktop workbench for local LLM inference

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.

github llama.cpp · 2d ago · 1 view

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.

github llama.cpp · 2d ago · 6 views

llama.cpp b10328 adds initial server tool isolation via Docker

The llama.cpp project has released build b10328, which introduces initial support for isolating tools within the server component using Docker. This update includes documentation and code adaptations to separate tool I/O sandboxes and rename relevant configuration flags.