GLM-5.2 on 8xB200: NVFP4 + 2x TP=4 doubles throughput vs FP8 TP=8

An engineering analysis of deploying GLM-5.2 on 8x B200 nodes identifies NVFP4 quantization with two TP=4 replicas as the optimal configuration, offering roughly double the node throughput compared to standard FP8 with a single TP=8 setup.

• GLM-5.2 is a ~750B MoE model with ~40B active parameters; NVFP4 weights require 459 GB, fitting within one B200's 720 GB HBM allocation for KV cache.
• Benchmarks using SGLang v0.5.12 show NVFP4 TP=4 achieves 4,116 tok/s/GPU at concurrency 128, compared to 1,947 tok/s/GPU for FP8 TP=8 at concurrency 256.
• Two independent TP=4 replicas per node yield ~33k tok/s aggregate throughput versus ~15.6k for a single FP8 TP=8 engine.
• NVFP4 reduces cost to approximately $0.13/M tokens on B200s, a ~3.5x improvement over H200 FP8 references.
• Users must use SGLang v0.5.13.post1+ for NVFP4 support and avoid SGLang <=v0.5.9 due to accuracy regressions in the flashmla_kv path.

This configuration maximizes hardware efficiency by addressing bandwidth bottlenecks inherent in MoE decoding, though real-world scheduler contention may reduce the theoretical 2x gain.