The paper introduces SWE-bench, an evaluation framework designed to test the capabilities of language models on real-world software engineering tasks. It consists of 2,294 problems drawn from actual GitHub issues and pull requests across 12 popular Python repositories.

  • The benchmark requires models to edit codebases to resolve specific issues, often involving coordination across multiple functions, classes, and files.
  • Models must interact with execution environments, process extremely long contexts, and perform complex reasoning beyond traditional code generation.
  • Evaluations show that even state-of-the-art proprietary models struggle significantly; the best-performing model, Claude 2, solved only 1.96% of the issues.
  • The fine-tuned model SWE-Llama also resolved only the simplest issues within the dataset.

Advances on SWE-bench are considered steps towards making language models more practical, intelligent, and autonomous in real-world scenarios.