Benchmark · agentic
SWE-Lancer
SWE-Lancer is a benchmark of 1,488 real freelance software-engineering tasks from Upwork worth $1 million USD in actual payouts; it measures how much of that money a model can "earn" by fully resolving them, scored as total USD earned (and the share of tasks resolved).
Read more
- Example
- A freelance-style bug-fix task: given a real (GitHub-style) issue and the app's full repository, the model must produce a code patch that makes a broken user flow work again (for example a bill-split or payment screen), checked by end-to-end tests. A second track gives management tasks: pick the best engineering proposal among several.
- Scoring
- Each task carries the real USD price it was paid on Upwork (from about $50 to $32,000). A model earns a task's full price only if its solution passes; there is no partial credit within a task. The headline metric is the total dollars earned out of the $1M pool, alongside the percentage of tasks resolved; the two tracks (IC coding and management) are also reported separately.
- Verification
- IC coding tasks are graded by end-to-end tests (Playwright) that simulate real user actions, written and verified by professional software engineers and run in an isolated Docker environment; the model's patch must pass all of a task's tests, and it cannot see or edit them. Management tasks are accepted when the model picks the same proposal the original hiring manager actually chose.
- Why it matters
- It ties model performance to economically real, market-priced work on a real production codebase, rather than synthetic or self-graded problems. Because grading uses full end-to-end user flows (not isolated function fixes) plus a management-decision track, it probes whether frontier models can do the complete, valuable engineering that people actually pay freelancers for.
Worked example
Task
IC SWE task (illustrative, ~$250): the app's "Split Bill" screen shows a total of NaN when a participant is removed before an amount is typed. Working in a Docker container with the full repo, fix the bug so the displayed total falls back to $0.00; a hidden end-to-end test replays this user flow.
Solution
```diff
function getSplitTotal(amounts) {
- return amounts.reduce((sum, a) => sum + a.value, 0);
+ return amounts.reduce((sum, a) => sum + (Number(a?.value) || 0), 0);
}
```Walkthrough
Removing a participant leaves an entry whose value is undefined, so a.value turns the running sum into NaN; coercing each value with Number(...) || 0 skips the missing amount and the total renders as $0.00. Grading is binary — the patch earns the task's full USD price only if it passes every end-to-end test for that task, run against the real user flow.
| Date | Model | Score | Source |
|---|---|---|---|
| 2025-03-05 | GPT-4.5 | 32.6% | OpenAI releases GPT-4.5, its largest model, for ChatGPT Plus |