Anthropic reports that its upgraded Claude 3.5 Sonnet model achieved a score of 49% on the SWE-bench Verified benchmark, surpassing the previous state-of-the-art score of 45%. The article details the "agent" system built around the model to help developers optimize performance.

  • SWE-bench Verified is a subset of 500 reviewed software engineering tasks that test a model's ability to resolve GitHub issues in Python repositories.
  • The agent design philosophy prioritizes giving control to the language model with minimal scaffolding, using Bash and Edit tools.
  • Tool descriptions were refined to prevent misunderstandings, such as requiring absolute paths to avoid errors when moving directories.
  • The system samples until the model decides it is finished or exceeds its 200k context length.

The post aims to help developers understand the agent architecture to get the best possible performance out of Claude 3.5 Sonnet on coding tasks.