arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 scales pre-training to 18 trillion tokens and uses multistage RL

from English

Alibaba has released the Qwen2.5 technical report, introducing a comprehensive series of large language models with significant improvements in both pre-training and post-training stages. The update scales high-quality pre-training datasets from 7 trillion to 18 trillion tokens and implements intricate supervised finetuning with over 1 million samples alongside multistage reinforcement learning.

Open-weight offerings include base and instruction-tuned models, with quantized versions available.
Proprietary hosted solutions feature two mixture-of-experts variants: Qwen2.5-Turbo and Qwen2.5-Plus.
The flagship Qwen2.5-72B-Instruct outperforms many open and proprietary models, showing competitive performance to Llama-3-405B-Instruct.
Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively.

These enhancements provide a strong foundation for common sense, expert knowledge, and reasoning capabilities, while post-training techniques notably improve long text generation, structural data analysis, and instruction following.

Importance 2/3 Moonshot AI (arXiv) Alibaba (Qwen) Open weights Research paper

Related articles

media r/LocalLLaMA · 42d ago · 1 view

Orthrus diffusion head trained Qwen 3.5/3.6 and Gemma 4 models dropping soon

The Orthrus project is preparing to release support for Qwen 3.5, Qwen 3.6, and Gemma 4 models using a diffusion head approach. The team has finalized testing and is currently setting up the release pipeline.

arxiv Moonshot AI (arXiv) · 452d ago

Qwen3 introduces unified thinking mode and expands multilingual support to 119 languages

The Qwen3 Technical Report presents the latest version of the Qwen model family, featuring dense and Mixture-of-Expert (MoE) architectures with parameter scales from 0.6 to 235 billion. A key innovation is the integration of thinking mode for complex reasoning and non-thinking mode for rapid responses into a unified framework, eliminating the need to switch between different models.

arxiv Moonshot AI (arXiv) · 452d ago · 2 views

Qwen3 introduces unified thinking mode and 119-language support

The Qwen3 Technical Report introduces a new series of open-weight large language models ranging from 0.6 to 235 billion parameters, featuring both dense and Mixture-of-Experts architectures. A key innovation is the integration of "thinking mode" for complex reasoning and "non-thinking mode" for rapid responses into a single framework, allowing dynamic switching without model alternation.

arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 introduces dense and MoE models with 18T pre-training tokens

The Qwen2.5 Technical Report details the release of a comprehensive series of large language models featuring both open-weight dense variants and proprietary mixture-of-experts (MoE) models. The update significantly scales pre-training data from 7 trillion to 18 trillion tokens and employs over 1 million samples for post-training via supervised finetuning and reinforcement learning.

arxiv Moonshot AI (arXiv) · 755d ago

Qwen Team releases Qwen2 series with 72B dense and MoE models

The Qwen team has released the Qwen2 technical report, introducing a new series of large language models ranging from 0.5 to 72 billion parameters, including both dense and Mixture-of-Experts (MoE) architectures.