arxiv Moonshot AI (arXiv) · 770d ago

NVIDIA releases Nemotron-4 340B model family and synthetic data pipeline

from English

NVIDIA has released the Nemotron-4 340B model family, comprising Base, Instruct, and Reward variants, along with the synthetic data generation pipeline used for alignment. The models are open access under the NVIDIA Open Model License Agreement and are sized to fit on a single DGX H100 with 8 GPUs in FP8 precision.

Nemotron-4-340B-Base, Instruct, and Reward variants are available under a permissive license allowing distribution and modification.
The models perform competitively against open-access benchmarks and were optimized to fit on a single DGX H100 with 8 GPUs in FP8 precision.
Over 98% of the data used in the model alignment process was synthetically generated, demonstrating the effectiveness of the approach.
NVIDIA is open-sourcing the synthetic data generation pipeline to support open research and facilitate training smaller language models.

The release aims to benefit the community in research studies and commercial applications, particularly for generating synthetic data to train smaller language models.

Importance 1/3 Moonshot AI (arXiv) NVIDIA Open weights Training methods

Related articles

arxiv arXiv cs.AI · 37d ago

UFP4: Uniform 4-Bit Training Overcomes Shrinkage Bias in LLM Pretraining

A study identifies shrinkage bias in E2M1-based FP4 formats due to geometric asymmetry, causing multiplicative error accumulation and training instability. The proposed UFP4 recipe uses uniform E1M2/INT4 grids and applies Random Hadamard Transform to all GEMMs, achieving lower loss degradation than E2M1 baselines in large-scale LLM pretraining. The authors recommend E1M2/INT4 as a first-class training primitive for future accelerators.

arxiv Moonshot AI (arXiv) · 770d ago

NVIDIA releases Nemotron-4 340B model family and synthetic data pipeline

NVIDIA has released the Nemotron-4 340B model family, comprising Base, Instruct, and Reward variants, under a permissive open license. The models are designed to fit on a single DGX H100 with 8 GPUs in FP8 precision and were trained on 9 trillion tokens.

media Hugging Face Forums · 4h ago

CrowdTensor launches volunteer-training Beta and Draft Qwen2.5-7B GSM8K Campaign RFC

CrowdTensor is an Apache-2.0 open-source protocol for checkpointed volunteer model-training campaigns that pins immutable model and data revisions to give admitted CPU, GPU, or TPU cells bounded work. The project has released a Beta enrollment process alongside a Draft RFC for a Qwen2.5-7B GSM8K campaign.

arxiv arXiv cs.AI · 3d ago · 1 view

SLAI T-Rex enables full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

SLAI introduces T-Rex, an end-to-end optimization framework for the full-parameter post-training of trillion-parameter-scale MoE models on Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, the system addresses memory pressure and communication overhead through hierarchical parallelism and kernel execution optimizations.

arxiv arXiv cs.CL · 4d ago

SLAI T-Rex enables full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

SLAI introduces T-Rex, an end-to-end optimization framework for the full-parameter post-training of trillion-parameter-scale MoE models on Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, the system addresses memory pressure and communication overhead through hierarchical parallelism and kernel execution optimizations.