Jeremy Berman set a new public record of 53.6% accuracy on the ARC-AGI-Pub leaderboard by applying an evolutionary algorithm to OpenAI's Sonnet 3.5 model, surpassing the previous high score of 43%. The approach treats the LLM as an evolution engine that generates and refines Python transform functions rather than direct output grids.
The method involves generating waves of candidate Python functions and scoring them against provided training examples using a two-tier fitness evaluation.
- Best-performing functions are selected as parents to create revision prompts for subsequent generations.
- The process iterates through multiple generations, potentially generating up to 500 functions per challenge using 31 dynamic prompts.
- Final solutions are derived from the top two functions across all generations submitted as candidate output grids.
Berman suggests that scaling test-time compute guided by evolutionary principles allows LLMs to compensate for generalization limitations and may indicate a path toward Artificial General Intelligence.