github llama.cpp · just now · inference

llama.cpp b10414 adds TQ2_0 support and optimizes mul_mv kernel

from English

The llama.cpp project released build b10414, introducing support for the GGML_TYPE_TQ2_0 ternary quantization type in the Metal backend. This update also includes optimizations to the contiguous tensor (cont) mul_mv kernel.

Added GGML_TYPE_TQ2_0 (ternary, 2 bits per element) support to the Metal backend.
Optimized the mul_mv kernel by switching from integer to float operations.
Improved performance through precalculated sums, hoisted coefficients, and contiguous y loads.

This release provides updated binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, Vulkan, ROCm, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 3d ago · 1 view

llama.cpp b10354 fixes CPU affinity mask on Android

The llama.cpp project has released version b10354, which includes a fix for the CPU affinity mask being ignored on Android devices.

github llama.cpp · 3d ago · 2 views

llama.cpp b10344 adds MTP support for Nemotron models

The llama.cpp project has released build b10344, which introduces Multi-Token Prediction (MTP) support specifically for the Nemotron and Nemotron Nano models.

github llama.cpp · 3d ago · 1 view

llama.cpp adds Granite-Switch architecture with in-graph router for per-token LoRA selection

The llama.cpp project introduces support for the Granite-Switch architecture, a dense model variant featuring N embedded LoRA adapters that are selected per-token via control tokens. This implementation replaces a previous proof-of-concept global sticky index with an in-graph causal "router" attention mechanism to resolve adapter indices.

github llama.cpp · 3d ago

llama.cpp b10336 release with ggml-webgpu refactor

The llama.cpp project has released version b10336, which includes a refactor of several wgsl files and simplification of flash_attn wgsl within the ggml-webgpu component.

media Hugging Face Forums · 5d ago · 9 views

vLLM Launcher provides a Windows desktop workbench for local LLM inference

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.