Benchmark · agentic
Multi-SWE-bench
Multi-SWE-bench measures whether an AI coding agent can resolve real-world GitHub issues across many repositories and programming languages, not just Python. The headline metric is the resolution rate: the percentage of task instances whose fix makes the hidden test suite pass.
Read more
- Example
- A representative task: given a real bug report or feature request from an open-source project written in a language such as Java, Go, Rust, TypeScript, JavaScript, C, or C++, plus the repository checked out at the pre-fix commit, the agent must produce a code patch that resolves the described issue.
- Scoring
- Each instance is graded pass/fail by execution. The agent's patch is applied to the repository and the project's tests are run; the instance counts as resolved only if the designated FAIL_TO_PASS tests now pass and every PASS_TO_PASS test still passes. The reported score is the resolution rate (resolved instances ÷ total), usually at pass@1, and it can be broken down by language and by expert-annotated difficulty (easy/medium/hard).
- Verification
- Acceptance is fully execution-based inside a per-instance Docker image that reproduces the exact language toolchain and dependencies. The generated patch must apply cleanly, then the fail-to-pass and pass-to-pass tests are executed; there is no LLM judging and no textual comparison to the developer's gold patch.
- Why it matters
- SWE-bench proved influential but is Python-only, while real software engineering spans many languages. Multi-SWE-bench tests whether coding agents generalize across languages on authentic issues, with expert difficulty labels, giving a harder, more realistic, and less saturated signal of practical software-engineering ability.
Worked example
Task
Illustrative instance in the Multi-SWE-bench format — language: Go, a small generic-collections library. problem_statement: 'Stack.Pop() on an empty stack returns the zero value with a nil error, silently hiding bugs in callers; it should return the sentinel ErrEmpty instead.' You are given the repository at the buggy base commit plus the failing test. FAIL_TO_PASS: TestPopEmptyReturnsError. PASS_TO_PASS: TestPushThenPop, TestLen. Submit a patch that makes the failing test pass without breaking the passing tests.
Solution
--- a/stack.go
+++ b/stack.go
@@ func (s *Stack[T]) Pop() (T, error) {
+ if len(s.items) == 0 {
+ var zero T
+ return zero, ErrEmpty
+ }
n := len(s.items)
v := s.items[n-1]
s.items = s.items[:n-1]
return v, nil
}Walkthrough
The fix adds an emptiness guard to Pop that returns the sentinel ErrEmpty (with the type's zero value) exactly as TestPopEmptyReturnsError expects, while leaving the normal pop path unchanged so TestPushThenPop and TestLen keep passing. Grading is purely execution-based: the harness applies the patch in the instance's Docker image and marks the instance resolved only if the FAIL_TO_PASS test flips to passing and no PASS_TO_PASS test regresses.
No verified scores reported yet for this benchmark.