Benchmark · agentic
SWE-rebench
SWE-rebench is a continuously refreshed variant of the SWE-bench coding benchmark that pulls tasks from recent GitHub issues to reduce the chance a model saw them during training. It reports the percentage of software-engineering tasks a model resolves (% resolved).
Read more
- Example
- A typical item hands the model a real bug report or feature request from a GitHub repository, and it must edit the project's code to resolve that issue.
- Scoring
- The metric is % resolved: the number of tasks the model fixes divided by the total number of tasks, expressed as a percentage.
- Verification
- A solution is accepted automatically when the repository's own test suite passes on the patched code — the fix must turn the failing tests green without breaking the others.
- Why it matters
- Because its tasks are refreshed from recent issues, it gives a cleaner read on genuine coding skill that is harder to inflate by memorizing older, public benchmark data.
Worked example
Task
SWE-rebench instance built from a real GitHub Python repo (
python-slugify), pinned to a base commit. Issue: slugify('Hello, World!') returns 'hello--world' — consecutive separators yield a doubled hyphen instead of one, but the expected slug is 'hello-world'. Deliverable: a git-diff patch against the repo snapshot that makes the FAIL_TO_PASS test pass while all PASS_TO_PASS tests stay green.Solution
--- a/slugify/slugify.py
+++ b/slugify/slugify.py
@@ def slugify(text):
text = re.sub(r"[^\w\s-]", "", text).strip().lower()
- return re.sub(r"[\s]+", "-", text)
+ text = re.sub(r"[\s]+", "-", text)
+ return re.sub(r"-{2,}", "-", text)Walkthrough
The slug builder collapses whitespace to
- but never dedupes repeated hyphens, so multiple separators leak through; adding re.sub(r'-{2,}', '-', text) normalizes them to a single -. SWE-rebench grades execution-based: it applies the patch in the repo container and marks the instance resolved only if the FAIL_TO_PASS test now passes and every PASS_TO_PASS test still passes (leaderboard = % resolved).