Benchmark · agentic
OSWorld
OSWorld is a benchmark that tests computer-use agents on real desktop-operating-system tasks spanning everyday apps (file managers, web browsers, office suites). It reports the percentage of tasks the agent completes successfully.
Read more
- Example
- A task might ask the agent to open a spreadsheet, edit a value or reformat some data and save the file, or to find a file in the file manager and change a system setting, operating the real graphical interface (GUI) just as a person would.
- Scoring
- The metric is task success rate, expressed as a percentage. Each task is either passed or failed, and the score is the fraction of tasks passed across the whole task set.
- Verification
- Results are checked automatically by execution-based scripts that inspect the machine's final state (for example, whether the correct file, value, or setting now exists), rather than by matching text or human voting.
- Why it matters
- It measures whether AI agents can actually operate a real computer across many applications, a much harder and more realistic test of autonomy than answering questions or single-app tasks.
Worked example
Task
A GNOME Terminal is open on the Ubuntu desktop. In the folder ~/project, recursively find every .log file larger than 10 MB and delete it, leaving all other files untouched.
Solution
find ~/project -type f -name '*.log' -size +10M -deleteWalkthrough
find walks ~/project recursively, matches regular files named *.log over 10 MB, and -delete removes exactly those while sparing everything else. OSWorld grades with a per-task checker over the VM's final filesystem state, awarding reward 1 only if the target logs are gone and the other files remain.