Benchmark · agentic
SWE-bench Pro
A harder, contamination-resistant version of SWE-bench from Scale AI. It draws ~1,865 real GitHub issues from held-out enterprise, startup and open-source repositories the models were not trained on, and the fixes typically span several files.
Read more
- Example
- Given a real issue and the full repository, the agent must produce a patch — often touching multiple files — so the project's hidden test suite passes. Problems come from private and commercial repos specifically to avoid training-data leakage.
- Scoring
- % of issues resolved, reported as pass@1 (one attempt). Higher is better. Scores run lower than on SWE-bench Verified because the tasks are harder and previously unseen.
- Verification
- Fully automatic and held-out: the generated patch is applied and the repository's real hidden unit tests run in a sandbox. A task counts only if every target test passes and nothing else regresses.
- Why it matters
- Built to resist the contamination and saturation that inflate scores on SWE-bench Verified, so it is a cleaner signal of real, generalizing coding-agent skill on code the model has never seen.
Worked example
Task
Repository
acme/payments at a fixed base commit. Issue: Money('0.05').allocate([1, 1, 1]) drops a cent — the three parts sum to $0.04, not $0.05. Produce a patch so an allocation always sums back to the original amount.Solution
--- a/payments/money.py
+++ b/payments/money.py
@@ def allocate(self, ratios):
- total = sum(ratios)
- return [self._from_minor(self.minor * r // total) for r in ratios]
+ total = sum(ratios)
+ shares = [self.minor * r // total for r in ratios]
+ remainder = self.minor - sum(shares)
+ for i in range(remainder):
+ shares[i] += 1
+ return [self._from_minor(s) for s in shares]Walkthrough
Integer division truncates, so the leftover minor units (cents) were silently lost. The fix computes each share, then distributes the remainder one unit at a time to the first parts, guaranteeing the allocated parts always sum to the total. The hidden tests assert
sum(m.allocate(ratios)) == m across several amounts and ratios.