github llama.cpp · 1h ago · inference

llama.cpp b10336 release with ggml-webgpu refactor

from English

The llama.cpp project has released version b10336, which includes a refactor of several wgsl files and simplification of flash_attn wgsl within the ggml-webgpu component.

This update provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu with CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler platforms.

The release also includes the standard UI build for users to interact with the model inference engine.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

media Hugging Face Forums · 1d ago · 8 views

vLLM Launcher provides a Windows desktop workbench for local LLM inference

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.

github llama.cpp · 2d ago · 1 view

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.

github llama.cpp · 2d ago · 6 views

llama.cpp b10328 adds initial server tool isolation via Docker

The llama.cpp project has released build b10328, which introduces initial support for isolating tools within the server component using Docker. This update includes documentation and code adaptations to separate tool I/O sandboxes and rename relevant configuration flags.

github llama.cpp · 2d ago · 5 views

llama.cpp b10327 fixes CUDA thread/block count in quantized cpy kernel

The llama.cpp project released version b10327, which includes a fix for the thread and block count in quantized copy kernel launches on CUDA. This update addresses an issue identified in pull request #26731.

github llama.cpp · 3d ago · 7 views

llama.cpp b10313 adds LRU scheduler to the server

The llama.cpp project has released version b10313, which introduces a Least Recently Used (LRU) scheduler for its server component. This update includes handling for request coalescing and fixes for streaming cases.