arxiv Moonshot AI (arXiv) · 605d ago

Microsoft releases Phi-4 technical report on 14B model using synthetic data

from English

Microsoft has published the technical report for Phi-4, a 14-billion parameter language model that prioritizes data quality through strategic use of synthetic data throughout training. Unlike previous Phi models that distilled capabilities from GPT-4, Phi-4 surpasses its teacher on STEM-focused QA tasks, demonstrating that its data-generation and post-training techniques extend beyond simple distillation.

The model uses a training recipe focused on high-quality synthetic data rather than just organic web content or code.
Phi-4 substantially outperforms GPT-4 on STEM-focused question answering capabilities.
It achieves strong performance relative to its size, particularly on reasoning-focused benchmarks, due to improved data, training curriculum, and post-training innovations.

The report highlights that despite minimal architectural changes from phi-3, the focus on data quality allows the model to exceed teacher capabilities in specific domains.

Importance 1/3 Moonshot AI (arXiv) Microsoft Research Research paper Training methods

Read original

arxiv arXiv cs.LG · 53d ago

MGUP: Momentum-Gradient Alignment for Selective Optimization

MGUP introduces a selective update mechanism that applies larger step-sizes to a fixed proportion of parameters in stochastic optimization, while using smaller, non-zero step-sizes for the rest. It integrates seamlessly with optimizers like AdamW, Lion, and Muon, providing theoretical convergence guarantees for MGUP-AdamW and demonstrating superior or more stable performance in training large language models and MAE pretraining tasks.

arxiv Moonshot AI (arXiv) · 466d ago

Microsoft releases Phi-4-Mini-Reasoning, a 3.8B model outperforming larger reasoning models

Microsoft has introduced Phi-4-Mini-Reasoning, a compact 3.8-billion-parameter model that achieves superior math reasoning performance compared to significantly larger competitors through a systematic four-step training recipe.

arxiv arXiv cs.LG · 2d ago

U-OPSD enables unsupervised on-policy self-distillation for LLMs

Researchers propose Unsupervised On-Policy Self-Distillation (U-OPSD), a method that achieves post-training improvement for large language models using only the model's own generations without external supervision. The approach samples multiple rollouts to construct a pseudo-solution via majority vote, then distills a teacher distribution into prefixes of the model's longest incorrect completion.

media MarkTechPost · 3d ago

Microsoft's SkillOpt enables agent skill transfer between Codex and Claude Code

Microsoft, Shanghai Jiao Tong University, Tongji University, and Fudan University researchers developed SkillOpt, a text-space optimizer that trains natural-language skill documents while keeping the target model frozen. The system uses an optimizer model to propose bounded edits based on scored rollouts, exporting the result as a single `best_skill.md` file.

arxiv arXiv cs.CL · 9d ago · 2 views

Memory Decoder at Scale: Scaling parametric long-term memory to 6.9B parameters

Researchers present Memory Decoder at Scale, a system that scales parametric long-term memory models up to 6.9B parameters and pretrains them on 300B tokens. To address the infeasibility of standard Faiss pipelines at this data scale, the authors implement a distributed indexing pipeline with sparse, batch-wise loading of kNN distributions.