Researchers introduce CodeElo, a standardized benchmark designed to evaluate the competition-level coding abilities of large language models by aligning with the official CodeForces platform. The benchmark addresses limitations in existing tools by using real contest problems, supporting special judges, and providing an Elo rating system comparable to human participants.
- CodeElo compiles recent CodeForces contest problems with details on divisions, difficulty ratings, and algorithm tags.
- Problems are submitted directly to the platform for judging, ensuring execution environment alignment.
- The study provides Elo ratings for 30 open-source and 3 proprietary LLMs for the first time.
- Results show o1-mini (1578) and QwQ-32B-Preview (1261) significantly outperform other models, which struggle with even the easiest problems.
This benchmark offers a reliable method to assess sophisticated code reasoning capabilities and provides insights into model performance across different algorithms and programming languages.