LIMO achieves high reasoning accuracy using only 1% of prior training data

Researchers propose the Less-Is-More Reasoning Hypothesis, demonstrating that sophisticated mathematical reasoning in large language models can emerge from minimal, strategically designed examples rather than massive datasets. Through simple supervised fine-tuning, their model LIMO achieves 63.3% accuracy on AIME24 and 95.6% on MATH500.

• Surpasses previous fine-tuned models by 6.5% on AIME24 and 59.2% on MATH500 while using only 1% of the training data required by prior approaches.
• Exhibits strong out-of-distribution generalization with a 45.8% absolute improvement across diverse benchmarks, outperforming models trained on 100x more data.

The findings suggest that in foundation models with comprehensive pre-trained knowledge, complex reasoning is elicited by the completeness of that knowledge base and the effectiveness of post-training examples as cognitive templates.