Anthropic's upgraded Claude 3.5 Sonnet model reached 49% on the SWE-bench Verified benchmark, surpassing the previous state-of-the-art score of 45%. The article details the "agent" system built around the model to achieve this result.

  • SWE-bench Verified is a subset of 500 solvable software engineering tasks used to evaluate an AI agent's ability to resolve GitHub issues in Python repositories.
  • The agent design philosophy prioritizes giving control to the language model with minimal scaffolding, using only a Bash Tool and an Edit Tool.
  • The system samples responses until the model decides it is finished or exceeds its 200k context length.
  • Detailed tool descriptions were crafted to prevent misunderstandings, allowing the model to choose its own workflow rather than following hardcoded patterns.

This approach helps developers optimize their use of Claude 3.5 Sonnet for complex coding tasks by demonstrating how minimal scaffolding can effectively leverage the model's capabilities.