Benchmark · agentic
SWE-bench Multilingual
SWE-bench Multilingual measures whether an AI coding agent can resolve real GitHub issues by editing a code repository across many programming languages. The score is % Resolved: the share of the 300 tasks whose fix passes the project's own tests.
Read more
- Example
- A task type: the agent is handed a bug report from a real open-source repository (e.g., a Go, Java, or Ruby project) plus the codebase at the pre-fix commit, and must produce a patch that edits one or more files to make the described problem go away, with no reference solution provided.
- Scoring
- The metric is % Resolved: each of the 300 instances is scored pass/fail and the score is the percentage that pass. An instance passes only if, after applying the agent's patch, all Fail-to-Pass tests (which target the bug) pass and all Pass-to-Pass tests (existing behavior) still pass.
- Verification
- A patch is verified by running the repository's test suite in an isolated container: the designated Fail-to-Pass tests must now pass and every Pass-to-Pass test must remain passing. Any failure, error, or non-applying patch marks the instance unresolved; there is no partial credit.
- Why it matters
- It probes coding ability beyond Python across 9 languages and 42 real repositories, exposing that models resolve far fewer issues in languages like Go, Rust, or PHP than in Python — a more honest test of cross-language software-engineering generalization.
Worked example
Task
Illustrative representative instance. Repository: a Go string-utilities library at its pre-fix commit. Issue:
Reverse() panics / returns garbled text on strings containing multi-byte UTF-8 characters. The Fail-to-Pass test TestReverseUnicode expects Reverse("héllo") == "olléh". The agent receives only the issue text and the repository — no reference patch.Solution
func Reverse(s string) string {
r := []rune(s)
for i, j := 0, len(r)-1; i < j; i, j = i+1, j-1 {
r[i], r[j] = r[j], r[i]
}
return string(r)
}Walkthrough
The original code reversed the string byte-by-byte, which splits multi-byte UTF-8 runes and corrupts characters like
é; converting to a []rune slice and swapping ends reverses by Unicode code point, so Reverse("héllo") yields "olléh". Grading: accepted only if TestReverseUnicode (Fail-to-Pass) now passes and all Pass-to-Pass tests still pass in the container.| Date | Model | Score | Source |
|---|---|---|---|
| 2026-07-22 | Laguna S 2.1 | 78.5% | Poolside releases Laguna S 2.1, an open-weight agentic coding model |
| 2026-03-27 | Composer 2 | 73.7% | Cursor releases Composer 2 technical report on agentic coding model training |
| 2025-11-07 | Kimi K2 Thinking | 61.1% | Moonshot AI releases Kimi K2 Thinking, an open-source 1T parameter reasoning model |
| 2025-07-28 | Kimi K2 | 47.3% | Moonshot releases Kimi K2, an open agentic MoE model with 32B activated parameters |
| 2025-07-28 | Kimi K2 | 47.3% | Kimi K2: open MoE model with 1T parameters and MuonClip optimizer |
| 2025-07-11 | Kimi K2 | 47.3% | Moonshot AI releases Kimi K2, a 1T-parameter MoE model with agentic capabilities |
| 2025-07-11 | Kimi K2-Instruct | 47.3% | Moonshot AI releases Kimi-K2-Instruct, a 1T-parameter MoE model with agentic capabilities |