Tencent-HY3 runs on 128GB Mac with double DeepSeek v4 Flash speed

A user successfully ran Tencent's 295B-A21B MoE model, HY3, on a MacBook M5 Max with 128GB of unified memory using a quantized GGUF checkpoint and llama.cpp. The setup achieved token generation speeds approximately double those of DeepSeek v4 Flash while maintaining comparable or better output quality.

• The model uses a UD128 "unsloth dynamic" 3-bit quantization (107GB) from Hugging Face user YanissAmz.
• Support for HY3's architecture and speculative decoding was added to llama.cpp via PR #25395.
• Benchmarks on the M5 Max showed 32.4 tokens/sec decode speed at empty context and 16.3 tokens/sec at 16K context.
• The user noted that practical performance on normal prompts and basic tool use was superior to their previous DeepSeek v4 setup.

The author highlights HY3 as a promising large MoE option for local deployment on high-memory consumer hardware, encouraging others to test it.