Tencent releases Hy3, a 295B-parameter MoE model

Tencent has released Hy3, a new Apache 2.0 licensed large language model developed by the Tencent Hy Team. The model features 295 billion total parameters with 21 billion active parameters and supports a context length of 256K.

• Hy3 utilizes a Mixture-of-Experts architecture with 3.8B MTP layer parameters.
• It outperforms similar-size models and rivals flagship open-source models requiring 2-5x more parameters.
• The full-sized model is available as a 598GB download, while an FP8 quantized version is 300GB.
• Tencent scaled up post-training with higher quality data following feedback from over 50 products.

The release demonstrates significant gains in utility across various productivity tasks.