A recent paper titled "LLM Inference at the Edge: Mobile, NPU, and GPU Performance Efficiency Trade-offs Under Sustained Load" benchmarks large language model performance across four distinct hardware platforms. The study evaluates a Qwen-2.5-1.5B 4bit model using different inference engines for each device to assess throughput, latency, and thermal stability.
- RPi5-Hailo demonstrated the most consistent performance with a coefficient of variation of .04% and no throttling, though it suffered from high latency (72 seconds for 564 tokens) due to PCIe bandwidth limits and CPU-NPU communication overhead.
- The iPhone 16 Pro achieved the best tokens per second for smartphones but experienced instability in initial and final iterations, dropping from ~42 tok/sec to 23-24 tok/sec due to thermal activity.
- The S24 Ultra required chunked prefill to manage resource spikes; while it stabilized thermals at an average of 64 +/- 1.9 C, it hit a frequency floor causing a decode time of 56 seconds for 646 tokens.
- The laptop GPU with a 4050 chip delivered the best overall performance despite exceeding its thermal design power, averaging 34 W system power.
The reviewer notes limitations in the study, including inconsistent use of coefficients of variation for comparison and varying power metrics that do not isolate component usage.