An independent verification of the Hierarchical Reasoning Model (HRM) on the ARC-AGI Semi-Private dataset reproduced scores of 32% on ARC-AGI-1 and 2% on ARC-AGI-2, confirming impressive results for its 27M parameter size. Ablation studies revealed that the model's hierarchical architecture had minimal impact compared to a similarly sized transformer, while the outer loop refinement process drove substantial performance gains.

  • HRM achieved 32% on ARC-AGI-1 and 2% on ARC-AGI-2 Semi-Private sets.
  • The hierarchical H-L computation contributed minimally to performance relative to a base transformer.
  • The outer loop refinement process was identified as the primary driver of success, particularly during training.
  • Cross-task transfer learning showed limited benefits, with most performance attributed to memorizing specific evaluation tasks.
  • Pre-training task augmentation was critical, requiring only 300 augmentations rather than the 1,000 reported in the original paper.

These findings suggest HRM's approach is fundamentally similar to "ARC-AGI without pretraining," challenging the narrative that its brain-inspired hierarchical architecture is the key differentiator.