WattGPU predicts inference power and latency on unseen GPUs and LLMs

Researchers introduce WattGPU, a tool featuring two predictive models for mean GPU power draw and Inter-Token Latency (ITL) that operate without hardware profiling. The approach leverages only publicly available LLM metadata and GPU specifications to generalize to unseen NVIDIA server-grade GPUs and LLMs.

• Evaluated on 42 open-source LLMs (0.1B--27B parameters) and 8 GPUs using leave-one-GPU-out and leave-one-LLM-out cross-validation.
• Mean power draw model achieves median absolute percentage error of ≤3.4% for offline and ≤13.5% for server scenarios on unseen GPUs.
• Latency model achieves ≤8.5% error in server mode, maintaining strong GPU ranking correlations (Kendall τ≥0.76).
• Reduces median absolute percentage error by approximately 4× compared to Load-Scaled Thermal Design Power and roofline baselines for server scenarios.

WattGPU enables operators to match specific LLMs to the most efficient GPUs without exhaustively profiling each combination, addressing the lack of tools for optimizing data center energy consumption.