Benchmark · agentic
SWE-bench Pro
A harder, contamination-resistant version of SWE-bench from Scale AI. It draws ~1,865 real GitHub issues from held-out enterprise, startup and open-source repositories the models were not trained on, and the fixes typically span several files.
Read more
- Example
- Given a real issue and the full repository, the agent must produce a patch — often touching multiple files — so the project's hidden test suite passes. Problems come from private and commercial repos specifically to avoid training-data leakage.
- Scoring
- % of issues resolved, reported as pass@1 (one attempt). Higher is better. Scores run lower than on SWE-bench Verified because the tasks are harder and previously unseen.
- Verification
- Fully automatic and held-out: the generated patch is applied and the repository's real hidden unit tests run in a sandbox. A task counts only if every target test passes and nothing else regresses.
- Why it matters
- Built to resist the contamination and saturation that inflate scores on SWE-bench Verified, so it is a cleaner signal of real, generalizing coding-agent skill on code the model has never seen.
Worked example
Task
Repository
acme/payments at a fixed base commit. Issue: Money('0.05').allocate([1, 1, 1]) drops a cent — the three parts sum to $0.04, not $0.05. Produce a patch so an allocation always sums back to the original amount.Solution
--- a/payments/money.py
+++ b/payments/money.py
@@ def allocate(self, ratios):
- total = sum(ratios)
- return [self._from_minor(self.minor * r // total) for r in ratios]
+ total = sum(ratios)
+ shares = [self.minor * r // total for r in ratios]
+ remainder = self.minor - sum(shares)
+ for i in range(remainder):
+ shares[i] += 1
+ return [self._from_minor(s) for s in shares]Walkthrough
Integer division truncates, so the leftover minor units (cents) were silently lost. The fix computes each share, then distributes the remainder one unit at a time to the first parts, guaranteeing the allocated parts always sum to the total. The hidden tests assert
sum(m.allocate(ratios)) == m across several amounts and ratios.| Date | Model | Score | Source |
|---|---|---|---|
| 2026-07-08 | Grok 4.5 | 64.7% | xAI releases Grok 4.5 with low cost and mixed benchmark performance |
| 2026-06-16 | Qwable-v1 | 80.3% | Qwable-v1 Released as Distillation of Claude Fable-5 |
| 2026-04-23 | Claude Opus 4.7 | 64.3% | OpenAI releases GPT-5.5 with agentic capabilities and doubled API pricing |
| 2026-04-23 | GPT-5.5 | 58.6% | OpenAI releases GPT-5.5 and GPT-5.5 Pro in API |
| 2026-04-23 | GPT-5.5 | 58.6% | OpenAI releases GPT-5.5 with agentic capabilities and doubled API pricing |
| 2025-12-11 | GPT-5.2 Thinking | 55.6% | OpenAI releases GPT-5.2, outperforming Gemini 3 in coding and reasoning benchmarks |
| 2025-12-11 | GPT-5.2 Thinking | 55.6% | OpenAI introduces GPT-5.2 with improved coding, long-context, and reasoning capabilities |