A recent paper titled "When Elo Lies" presents a systematic empirical study revealing that Codeforces-based Elo ratings for Large Language Models are heavily influenced by hidden experimental factors. The authors demonstrate that these ratings can fluctuate significantly due to the temporal ordering of submissions, the specific contests selected, and the inherent stochastic variability of the models themselves.
- Varying submission orders can shift Elo scores by up to 394 points.
- Contest selection can cause differences of up to 1,122 points for the same model.
- Run-to-run performance instability resulted in a maximum difference of 349 points in mean scores when evaluating identical contests.
The study concludes that direct Elo comparisons are unreliable and potentially misleading without strict standardization and transparent reporting of experimental settings.