DeepSeek has released DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model featuring 671B total parameters with 37B activated per token. The model utilizes Multi-head Latent Attention and an auxiliary-loss-free load balancing strategy, trained on 14.8 trillion tokens using FP8 mixed precision.

  • DeepSeek-V3 achieves performance comparable to leading closed-source models while requiring only 2.788M H800 GPU hours for full training.
  • The model incorporates Multi-Token Prediction (MTP) for inference acceleration and distills reasoning capabilities from the DeepSeek-R1 series.
  • Weights are provided in FP8 format, with conversion scripts available for BF16 experimentation.
  • Local deployment is supported via frameworks including SGLang, vLLM, LMDeploy, and TensorRT-LLM on NVIDIA, AMD, and Huawei Ascend hardware.

The release provides an efficient, high-performance open-source alternative for developers, with comprehensive evaluation results demonstrating its strength in math and code tasks.