A user demonstrates running the GLM 5.2 model on a custom hardware setup consisting of four GB10 GPUs connected via a 100G switch, achieving approximately 330k context length and ~25 tokens per second in generation.
- The configuration uses TP4+DCP2 for a ~360k KV pool, with prefill speeds reaching 900-1000 t/s on longer prompts.
- Using DCP4 extends context to 660k but reduces prefill speed to ~400 t/s, while dropping DCP raises prefill to ~750 t/s.
- Decoding performance varies by content type: thinking tasks get ~20 tok/s, code gets 25-35 tok/s, and typical turns in Pi yield ~24 tok/s.
- Pruning the model by 5-10% may allow for 1M context or higher concurrency, though a 10% data-free prune was found to lose some instruction adherence despite preserving coding capability.
- The total hardware cost was approximately $16k, with components including two Acer GN100s, two Asus GX10s, and a Mikrotik CRS504 switch.
The author notes that while this setup is not financially optimal for general use, it is a viable alternative to running GLM locally on a 512GB Mac Studio, which reportedly achieves only 12 tok/s decode.