arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 introduces dense and MoE models with 18T pre-training tokens

from English

The Qwen2.5 Technical Report details the release of a comprehensive series of large language models featuring both open-weight dense variants and proprietary mixture-of-experts (MoE) models. The update significantly scales pre-training data from 7 trillion to 18 trillion tokens and employs over 1 million samples for post-training via supervised finetuning and reinforcement learning.

Open-weight dense models are available in sizes of 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B parameters.
Proprietary MoE models Qwen2.5-Turbo and Qwen2.5-Plus are available via Alibaba Cloud Model Studio.
The flagship Qwen2.5-72B-Instruct matches the performance of Llama-3-405B-Instruct despite being five times smaller.
Context length support increases from 2K to 8K tokens, with Qwen2.5-Turbo supporting up to 1 million tokens.

These improvements aim to provide cost-effective solutions for diverse use cases while maintaining top-tier performance on reasoning, coding, and mathematics benchmarks.

Importance 3/3 Moonshot AI (arXiv) Alibaba (Qwen) Open weights Research paper

Read original

media r/LocalLLaMA · 42d ago · 1 view

Orthrus diffusion head trained Qwen 3.5/3.6 and Gemma 4 models dropping soon

The Orthrus project is preparing to release support for Qwen 3.5, Qwen 3.6, and Gemma 4 models using a diffusion head approach. The team has finalized testing and is currently setting up the release pipeline.

arxiv Moonshot AI (arXiv) · 452d ago

Qwen3 introduces unified thinking mode and expands multilingual support to 119 languages

The Qwen3 Technical Report presents the latest version of the Qwen model family, featuring dense and Mixture-of-Expert (MoE) architectures with parameter scales from 0.6 to 235 billion. A key innovation is the integration of thinking mode for complex reasoning and non-thinking mode for rapid responses into a unified framework, eliminating the need to switch between different models.

arxiv Moonshot AI (arXiv) · 452d ago · 2 views

Qwen3 introduces unified thinking mode and 119-language support

The Qwen3 Technical Report introduces a new series of open-weight large language models ranging from 0.6 to 235 billion parameters, featuring both dense and Mixture-of-Experts architectures. A key innovation is the integration of "thinking mode" for complex reasoning and "non-thinking mode" for rapid responses into a single framework, allowing dynamic switching without model alternation.

arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 scales pre-training to 18 trillion tokens and uses multistage RL

Alibaba has released the Qwen2.5 technical report, introducing a comprehensive series of large language models with significant improvements in both pre-training and post-training stages. The update scales high-quality pre-training datasets from 7 trillion to 18 trillion tokens and implements intricate supervised finetuning with over 1 million samples alongside multistage reinforcement learning.

arxiv Moonshot AI (arXiv) · 755d ago

Qwen Team releases Qwen2 series with 72B dense and MoE models

The Qwen team has released the Qwen2 technical report, introducing a new series of large language models ranging from 0.5 to 72 billion parameters, including both dense and Mixture-of-Experts (MoE) architectures.