Developer runs DeepSeek V4 Flash on a single RTX 6000 Pro using vLLM-Moet

A developer has successfully run the DeepSeek V4 Flash model on a single NVIDIA RTX 6000 Pro GPU by utilizing a customized version of vLLM called vLLM-Moet. This achievement is made possible through specific compression techniques that allow the large language model to fit into the consumer-grade hardware's limited VRAM.

• The method involves compressing routed experts to 2-bit while keeping fp4 experts, enabling the model to load entirely onto a single GPU.
• Initial loading requires approximately 150 GB of RAM or extended swap space to handle safetensor sharding before the model moves to VRAM.
• Benchmarks on the RTX 6000 Pro show token generation speeds ranging from roughly 100 to 134 tokens per second depending on context length.
• The technique also reportedly supports GLM 5.2, though that configuration requires at least two RTX 6000 Pro GPUs.

This feat demonstrates that high-performance models like DeepSeek V4 Flash can be accessible on single consumer GPUs through specialized engine modifications and expert compression.