The author achieved a new state-of-the-art accuracy of 50% on the public test set for the ARC-AGI reasoning benchmark by leveraging GPT-4o to generate and evaluate thousands of Python implementations per problem.
- The method generates approximately 8,000 Python programs per problem using few-shot prompts with step-by-step reasoning.
- Programs are selected based on their correctness against provided examples, with an ensemble approach for grid size variations.
- A revision phase attempts to fix the top 12 most promising implementations by showing them their actual output and asking for corrections.
- The prior state of the art was 34% accuracy, while humans achieve 85% on the easier train distribution.
This result demonstrates that current large language models can perform decently well on complex reasoning tasks through extensive sampling and verification, challenging claims that they lack learning capabilities at inference time.