arxiv Moonshot AI (arXiv) · 755d ago

Qwen Team releases Qwen2 series with 72B dense and MoE models

from English

The Qwen team has released the Qwen2 technical report, introducing a new series of large language models ranging from 0.5 to 72 billion parameters, including both dense and Mixture-of-Experts (MoE) architectures.

The flagship Qwen2-72B base model achieves scores of 84.2 on MMLU, 89.5 on GSM8K, and 64.6 on HumanEval.
The instruction-tuned Qwen2-72B-Instruct variant reaches 9.1 on MT-Bench and 35.7 on LiveCodeBench.
Models are trained on over 7 trillion tokens and support approximately 30 languages.
Weights are openly available on Hugging Face and ModelScope for community use.

The release aims to provide high-performance foundational models that compete with proprietary systems while remaining accessible for research and deployment.

Importance 3/3 Moonshot AI (arXiv) Alibaba (Qwen) Open weights Research paper

Benchmarks

Benchmark	Model	Score
GPQA Diamond	Qwen2-72B	37.9%
LiveCodeBench	Qwen2-72B-Instruct	35.7%
GSM8K	Qwen2-72B	89.5%
MMLU	Qwen2-72B	84.2%
BIG-Bench Hard	Qwen2-72B	82.4%
HumanEval	Qwen2-72B	64.6%

Read original

media r/LocalLLaMA · 42d ago · 1 view

Orthrus diffusion head trained Qwen 3.5/3.6 and Gemma 4 models dropping soon

The Orthrus project is preparing to release support for Qwen 3.5, Qwen 3.6, and Gemma 4 models using a diffusion head approach. The team has finalized testing and is currently setting up the release pipeline.

arxiv Moonshot AI (arXiv) · 452d ago

Qwen3 introduces unified thinking mode and expands multilingual support to 119 languages

The Qwen3 Technical Report presents the latest version of the Qwen model family, featuring dense and Mixture-of-Expert (MoE) architectures with parameter scales from 0.6 to 235 billion. A key innovation is the integration of thinking mode for complex reasoning and non-thinking mode for rapid responses into a unified framework, eliminating the need to switch between different models.

arxiv Moonshot AI (arXiv) · 452d ago · 2 views

Qwen3 introduces unified thinking mode and 119-language support

The Qwen3 Technical Report introduces a new series of open-weight large language models ranging from 0.6 to 235 billion parameters, featuring both dense and Mixture-of-Experts architectures. A key innovation is the integration of "thinking mode" for complex reasoning and "non-thinking mode" for rapid responses into a single framework, allowing dynamic switching without model alternation.

arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 introduces dense and MoE models with 18T pre-training tokens

The Qwen2.5 Technical Report details the release of a comprehensive series of large language models featuring both open-weight dense variants and proprietary mixture-of-experts (MoE) models. The update significantly scales pre-training data from 7 trillion to 18 trillion tokens and employs over 1 million samples for post-training via supervised finetuning and reinforcement learning.

arxiv Moonshot AI (arXiv) · 598d ago

Qwen2.5 scales pre-training to 18 trillion tokens and uses multistage RL

Alibaba has released the Qwen2.5 technical report, introducing a comprehensive series of large language models with significant improvements in both pre-training and post-training stages. The update scales high-quality pre-training datasets from 7 trillion to 18 trillion tokens and implements intricate supervised finetuning with over 1 million samples alongside multistage reinforcement learning.