media Hugging Face Forums · 1h ago · open_models

vLLM Launcher provides a Windows desktop workbench for local LLM inference

from English

A new project called vLLM Launcher has been released as a Windows desktop application designed to manage local large language model inference through WSL2. This tool allows users to control multiple engine backends, including vLLM, SGLang, and llama.cpp, from a single graphical interface.

Multi-engine management for vLLM, SGLang, and llama.cpp via one desktop UI.
Local API gateway offering OpenAI-compatible and Anthropic-compatible endpoints.
Instance cards to start, stop, restart, and inspect model workflows.
Features include request logs, health checks, runtime metrics, and disk model discovery.
Floating mode for compact status monitoring while using other applications.

The tool bridges WSL engines to any OpenAI-compatible IDE, with the underlying tech stack built on Go, Wails, React, TypeScript, and Node.js. The first public release is available as a portable Windows executable.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Local inference

Read original

github llama.cpp · 14h ago

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.

github llama.cpp · 15h ago · 5 views

llama.cpp b10328 adds initial server tool isolation via Docker

The llama.cpp project has released build b10328, which introduces initial support for isolating tools within the server component using Docker. This update includes documentation and code adaptations to separate tool I/O sandboxes and rename relevant configuration flags.

github llama.cpp · 1d ago · 2 views

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

llama.cpp b10328 adds initial server tool isolation via Docker

llama.cpp b10327 fixes CUDA thread/block count in quantized cpy kernel

llama.cpp b10313 adds LRU scheduler to the server

llama.cpp b10312 release with server fix for busy models