Benchmark · agentic
SWE-bench Verified
Tests whether an AI agent can resolve real GitHub issues end-to-end. The "Verified" set is 500 human-validated tasks from popular open-source Python repositories.
Read more
- Example
- Given a bug report and the repository, the model must produce a code patch — e.g. fix a failing date-parsing function so the project's test suite passes.
- Scoring
- % of issues resolved — usually reported as pass@1 (one attempt). Higher is better.
- Verification
- Fully automatic: the generated patch is applied and the project's real hidden unit tests run in a sandbox. A task counts only if every target test passes and nothing regresses.
- Why it matters
- The leading real-world coding-agent benchmark and a headline number in every frontier release; progress here tracks how close agents are to autonomous software engineering.
Worked example
Task
Repository
django/django at a fixed base commit. Bug report: django.utils.text.slugify("___This-is-a-Test___", allow_unicode=True) returns '___this-is-a-test___', but the leading/trailing underscores and hyphens should be stripped so the result is 'this-is-a-test'. The model is given only the issue text plus the repository and must output a unified-diff patch.Solution
--- a/django/utils/text.py
+++ b/django/utils/text.py
@@ def slugify(value, allow_unicode=False):
value = re.sub(r"[^\w\s-]", "", value.lower())
- return re.sub(r"[-\s]+", "-", value)
+ return re.sub(r"[-\s]+", "-", value).strip("-_")Walkthrough
slugify collapses internal separators but never trims the surrounding
-/_, so appending .strip("-_") to the final re.sub removes them; the patch is minimal and preserves all other behavior. SWE-bench Verified applies the patch to the repo at the base commit and grades by running the hidden suite — it passes only if every FAIL_TO_PASS test flips to passing while all PASS_TO_PASS tests stay green.