OpenAI conducted a detailed audit of the SWE-Bench Pro coding benchmark and estimated that approximately 30% of its tasks are broken due to design flaws. The investigation was prompted by concerns that evaluation errors could misrepresent model capabilities and safety cases, leading OpenAI to retract its previous recommendation to adopt this benchmark.
The study identified four primary categories of issues: overly strict tests enforcing specific implementation details, underspecified prompts omitting requirements, low-coverage tests allowing incomplete fixes, and misleading prompts contradicting test expectations. To reach these findings, the team used an automated pipeline to flag 286 potentially broken tasks, which were then reviewed by investigator agents and five experienced software engineers per task.
OpenAI advises model developers to carefully examine results from this benchmark and hopes the evaluation community will develop new benchmarks curated specifically by experienced software developers to ensure better human oversight and reliability.