OpenAI’s o1 model with high reasoning effort achieved the top score on Aider’s newly released polyglot benchmark, significantly outperforming other leading large language models. This updated benchmark is designed to be much more difficult than previous versions to better distinguish the capabilities of today's strongest coding models.
- The new benchmark utilizes 225 of the most difficult exercises from Exercism across C++, Go, Java, JavaScript, Python, and Rust.
- It excludes easy problems solvable by all tested models to create a wider score range between approximately 5% and 50%.
- o1 established a top score of 62%, leaving 86 unsolved problems for future model advancements.
- The benchmark was designed to address the saturation of older Python-only benchmarks where scores approached 80%.
The authors consider this recalibration important because it provides headroom for future LLMs and allows for clearer comparison of relative performance among top coding models.