Benchmark · agentic
Windows Agent Arena
Windows Agent Arena (WAA) is a reproducible benchmark that evaluates multimodal AI agents on 154 real tasks performed inside a live Windows 11 environment; the headline metric is task success rate — the fraction of tasks the agent completes correctly.
Read more
- Example
- A task type: drive real desktop apps to reach a goal state using only screenshots and the accessibility tree — for example, change a Windows Settings option, edit a spreadsheet in LibreOffice Calc, install an extension in VS Code, or complete a browsing task in Chrome.
- Scoring
- Each task ships a deterministic reward script that inspects the VM's final state (files, registry values, app output) at the end of the episode and returns a per-task score, usually binary success (1/0). The benchmark score is the average success rate across all 154 tasks; full runs are parallelized across Azure VMs.
- Verification
- A result counts only if the task's built-in validator confirms the target end-state in the actual environment, not by matching the agent's text or action log. The agent has a fixed action/step budget; exceeding it or leaving the wrong state scores 0. Runs are reproducible because the VM image and task configs are fixed.
- Why it matters
- It tests whether agents can actually operate a full, real operating system — navigating GUIs across many everyday apps — rather than just answering questions. A large gap remains between current agents and unassisted humans, and the Azure-parallelized harness makes large-scale, reproducible OS-agent evaluation fast.
Worked example
Task
Representative item (Windows Settings domain): 'Switch the Windows app color mode to Dark.' The agent receives a screenshot plus the accessibility tree of a fresh Windows 11 VM and must issue GUI actions (mouse and keyboard) to accomplish it — only the goal is given, no target text.
Solution
Steps: 1) Open Settings (Win+I, or Start then Settings). 2) Go to Personalization then Colors. 3) Under 'Choose your mode', select Dark. Final state: the app/system theme is Dark (registry value HKCU\...\Themes\Personalize\AppsUseLightTheme = 0).
Walkthrough
It is correct because the goal is a state change and the sequence sets exactly the value the validator reads. After the episode the reward script queries the VM registry and returns success=1 only if AppsUseLightTheme equals the dark value (0), regardless of how the agent got there.
| Date | Model | Score | Source |
|---|---|---|---|
| 2025-09-02 | UI-TARS-2 | 50.6% | UI-TARS-2 advances GUI agent with multi-turn reinforcement learning |