Custom CUDA code runs Qwen3 30B A3B at 50 tok/s on RTX 5060 Ti

A user has developed custom CUDA and C++ code that enables running the Qwen3-30B-A3B model at 50-54 tokens per second on an RTX 5060 Ti with 16 GB of VRAM using float8 precision.

• The implementation achieves a roughly 50% speed improvement over llama.cpp, which runs at approximately 33-34 tok/s with n-cpu-moe.
• The performance gains are derived from combining state-of-the-art solutions found in NeurIPS, ICML, and EuroSys papers.
• The source code is available via the repository.

The author suggests that such engines provide new local inference opportunities on consumer hardware, offering a more private, cheaper, and greener alternative to centralized datacenters.