OpenAI released the o1-preview and o1-mini models, which are trained to emulate reasoning by generating and refining reasoning tokens before providing a final answer. Testing on the ARC-AGI public evaluation dataset shows that both o1 variants outperform GPT-4o in accuracy.
- o1-preview achieves accuracy comparable to Anthropic's Claude 3.5 Sonnet but takes approximately 10 times longer to complete tasks.
- Solving the 400 public ARC-AGI tasks took o1 70 hours, compared to just 30 minutes for GPT-4o and Claude 3.5 Sonnet.
- The models utilize a chain-of-thought paradigm applied at both training and inference time, leveraging reinforcement learning to refine strategies.
- Performance scales with test-time compute, showing a log-linear relationship between accuracy and the amount of reasoning tokens used.
The results highlight that while increased test-time compute improves accuracy, efficiency remains a critical factor for practical application and benchmarking.