arxiv Moonshot AI (arXiv) · 770d ago

NVIDIA releases Nemotron-4 340B model family and synthetic data pipeline

from English

NVIDIA has released the Nemotron-4 340B model family, comprising Base, Instruct, and Reward variants, under a permissive open license. The models are designed to fit on a single DGX H100 with 8 GPUs in FP8 precision and were trained on 9 trillion tokens.

Nemotron-4-340B-Instruct surpasses comparable instruct models in instruction following and chat capabilities.
Nemotron-4-340B-Reward achieves top accuracy on RewardBench, outperforming proprietary models like GPT-4o-0513 and Gemini 1.5 Pro-0514.
Over 98% of the data used for model alignment was synthetically generated.
NVIDIA is open-sourcing the synthetic data generation pipeline, including prompts and quality filtering tools.

The release aims to support community research and commercial applications by providing high-quality models and tools for generating synthetic training data.

Importance 2/3 Moonshot AI (arXiv) NVIDIA Open weights Training methods

Read original

arxiv arXiv cs.AI · 37d ago

UFP4: Uniform 4-Bit Training Overcomes Shrinkage Bias in LLM Pretraining

A study identifies shrinkage bias in E2M1-based FP4 formats due to geometric asymmetry, causing multiplicative error accumulation and training instability. The proposed UFP4 recipe uses uniform E1M2/INT4 grids and applies Random Hadamard Transform to all GEMMs, achieving lower loss degradation than E2M1 baselines in large-scale LLM pretraining. The authors recommend E1M2/INT4 as a first-class training primitive for future accelerators.

arxiv Moonshot AI (arXiv) · 770d ago

NVIDIA releases Nemotron-4 340B model family and synthetic data pipeline

NVIDIA has released the Nemotron-4 340B model family, comprising Base, Instruct, and Reward variants, along with the synthetic data generation pipeline used for alignment. The models are open access under the NVIDIA Open Model License Agreement and are sized to fit on a single DGX H100 with 8 GPUs in FP8 precision.

media Hugging Face Forums · 3h ago

CrowdTensor launches volunteer-training Beta and Draft Qwen2.5-7B GSM8K Campaign RFC

CrowdTensor is an Apache-2.0 open-source protocol for checkpointed volunteer model-training campaigns that pins immutable model and data revisions to give admitted CPU, GPU, or TPU cells bounded work. The project has released a Beta enrollment process alongside a Draft RFC for a Qwen2.5-7B GSM8K campaign.

arxiv arXiv cs.AI · 3d ago · 1 view

SLAI T-Rex enables full-parameter post-training of DeepSeek-V4 on Ascend SuperPOD

SLAI introduces T-Rex, an end-to-end optimization framework for the full-parameter post-training of trillion-parameter-scale MoE models on Ascend NPU SuperPOD. Using the DeepSeek-V4 model family as the target workload, the system addresses memory pressure and communication overhead through hierarchical parallelism and kernel execution optimizations.

arxiv arXiv cs.CL · 4d ago