Benchmark · agentic
SWE-bench
SWE-bench tests whether an AI system can fix real software bugs: it draws 2,294 genuine issues from popular open-source Python repositories on GitHub and asks the model to produce a code patch. The headline metric is the percentage of issues resolved.
Read more
- Example
- A typical item hands the model a real bug report or feature request filed on a Python project — say a Django or scikit-learn issue — together with that repository's code, and asks it to generate a patch that fixes the problem.
- Scoring
- The score is the percentage of the 2,294 issues resolved: an issue counts only if the model's patch, once applied, makes the repository's own test suite pass.
- Verification
- Verification is fully automatic — the generated patch is applied to the repo and the project's real tests are run, and the task is accepted as resolved only when the target failing tests turn to passing while previously passing tests still pass, with no human judgment or exact match.
- Why it matters
- It measures end-to-end software-engineering skill on real-world code rather than toy exercises, which makes it a closely watched gauge of coding-agent progress; for headline numbers it has largely been superseded by the human-validated subset SWE-bench Verified.
Worked example
Task
Repository
psf/requests. A user reports that setting a request header to None (to remove a default session header) has no effect: the None-valued key survives in the merged headers instead of being deleted. Given the repo at the buggy commit and this issue, produce a patch so that requests.sessions.merge_setting drops keys whose value is None.Solution
--- a/requests/sessions.py
+++ b/requests/sessions.py
@@ -412,6 +412,11 @@ def merge_setting(request_setting, session_setting, dict_class=OrderedDict):
merged_setting = dict_class(to_key_val_list(session_setting))
merged_setting.update(to_key_val_list(request_setting))
+ # Remove keys that are set to None. Extract the keys first to avoid
+ # mutating the dictionary while iterating over it.
+ none_keys = [k for (k, v) in merged_setting.items() if v is None]
+ for key in none_keys:
+ del merged_setting[key]
+
return merged_settingWalkthrough
After merging the session and request dicts, any key that maps to
None came from an explicit request-level override meant to delete a session default, so the fix collects those keys and removes them (building the list first to avoid mutating the dict during iteration). SWE-bench grades by applying the patch and running the instance's FAIL_TO_PASS and PASS_TO_PASS tests, counting it resolved only if all of them pass.