NVIDIA Puzzle-75B-A9B NVFP4 runs at 132 t/s on 3×3090

The Nemotron-3-Puzzle-75B-A9B model, using NVFP4 quantization and vLLM 0.22.1, achieves 132 tokens per second decode speed across three streams on a setup of three NVIDIA RTX 3090 GPUs. This configuration utilizes pipeline parallelism with cards capped at 200W each, while a fourth card handles a speech sidecar.

• The model operates as a 75B-total / 9B-active Mixture of Experts (MoE) architecture.
• It supports three seats with a 256K context window using fp8 KV cache and hybrid Mamba to keep memory usage low.
• Performance metrics include 1,949 t/s prefill speed and approximately 500W power consumption for the entire box.
• The setup replaces a previous Nemotron Super 120B MoE GGUF resident, offering better instruction-following and roughly double the speed per watt while freeing up one GPU card.

This size category is significant because 70–80B total parameters with ~10B active parameters fill 72GB of quantized VRAM exactly, providing dense-class quality at A3B-class speed. Currently, Puzzle is the only modern option in this specific parameter band for multi-24GB rigs.