A developer is releasing an open-source, single-file dashboard designed to monitor local Large Language Model serving infrastructure. The tool provides real-time visibility into GPU utilization, per-model throughput, KV cache fill, and system statistics for llama.cpp and vLLM deployments.
- The frontend consists of a single HTML file that opens directly via file:// protocol, while the backend uses standard Python libraries to read nvidia-smi data and Prometheus metrics.
- It displays per-card GPU details including VRAM, power, temperature, and compute tenants derived from ground-truth queries rather than estimates.
- Primary worker metrics include decode and prefill tokens per second, request counts, and context fill, with auto-discovery of server ports to handle model swaps.
- Optional features include a secondary server row for additional CPU/GPU instances, a browsable model library inventory, and an optional live reasoning tap for chain-of-thought logging.
The dashboard aims to improve observability for local LLM setups, particularly aiding users who utilize teacher model distillation by visualizing the models' reasoning processes.