The author successfully deployed GLM-5.2 with MTP speculative decode on a cluster of four NVIDIA GB10 (DGX Spark) nodes, achieving approximately 9.4 tokens per second. This setup utilizes vLLM with tensor parallelism, ported sparse-MLA Triton kernels, and a deterministic 15% expert pruning to fit AWQ-INT4 weights. A critical finding is that the original Docker image build instructions are incomplete, requiring reconstruction of missing patches for deep_gemm.py and sparse_attn_indexer.py. The author also identified that using any vLLM version other than the specific pinned commit causes real AWQ weights to crash during loading due to CUDA errors. To replicate the environment, users must apply a custom script that bakes in kernels and routes functions to sm12x fallbacks. Performance benefits include roughly double the speed of previous llama.cpp implementations, though inter-node bandwidth remains a bottleneck for dual-rail scaling.
GLM-5.2 on 4x DGX Spark: Reconstructing Missing Build Steps for MTP Speculative Decode
LordNeel releases GGUF quants for Tencent Hy3 and NVIDIA Nemotron-Labs-Audex
LordNeel has published GGUF quantization sets for Tencent's 295B MoE Hy3 model and NVIDIA's audio-capable 30B hybrid MoE Nemotron-Labs-Audex-30B-A3B, providing full benchmark data and reproducible metrics.
NVIDIA releases Puzzle-75B-A9B, a compressed Nemotron model with 2x throughput
NVIDIA has released the Nemotron-Labs-3-Puzzle-75B-A9B model, a deployment-optimized large language model derived from Nemotron-3-Super-120B-A12B. It utilizes the Iterative Puzzle post-training compression framework to significantly improve inference efficiency for reasoning-heavy and long-context workloads while maintaining strong downstream accuracy.
NVIDIA releases Qwen3.6-27B-NVFP4 model
NVIDIA has released the Qwen3.6-27B-NVFP4 model on Hugging Face.
GLM-5.2 NVFP4 on four DGX Sparks — the MTP mystery is solved, and it's now ~24 tok/s at 128K context
A follow-up investigation into running GLM-5.2 NVFP4 on four DGX Spark nodes resolves a previous performance bottleneck where high acceptance rates were impossible at 128K context.
High-quality GLM-5.2 Quant on 4x DGX Spark - Guide, Results, and Comps
The author demonstrates running the GLM-5.2 NVFP4 model on four NVIDIA GB10 DGX Spark nodes with a 128K context window, achieving usable serving performance through aggressive system optimization.