Benchmark · agentic
Android World
AndroidWorld is a benchmark and live-emulator environment (Google, 2024) for autonomous agents that control a real Android device; it has 116 hand-built tasks across 20 real apps, and the metric is task success rate.
Read more
- Example
- The agent is given a natural-language goal and must operate real apps to reach it — for instance, in a calendar app, create an event with a specified title, date, and time by tapping, typing, and scrolling through the live UI.
- Scoring
- Each task gives a binary reward: 1 if the final device state satisfies the task's success condition, else 0. The score is the success rate — the fraction of tasks solved — averaged over the suite (tasks are parameterized with randomized values, so runs vary).
- Verification
- Success is checked programmatically by inspecting the on-device system/app state after the run (e.g., querying an app's database, files, or settings), not by matching the action trajectory or using an LLM judge — which makes rewards durable and reproducible.
- Why it matters
- It evaluates agents on realistic, open-ended mobile device control with verifiable rewards; randomized parameters deter memorization, and real apps plus state-based grading make it a durable, reproducible testbed for multimodal computer-use agents.
Worked example
Task
Task (ContactsAddContact family): "Create a new contact for Grace Taylor. Their phone number is 555-123-4567." The agent acts on the live Contacts app via the accessibility tree and screenshots.
Solution
Open the Contacts app → tap "Add contact" → enter the name Grace Taylor → enter the number 555-123-4567 → tap "Save". Final state: the device's contacts store now contains a contact named Grace Taylor with number 555-123-4567.
Walkthrough
It's correct because the task's success checker queries the on-device contacts and confirms a contact exists with exactly the target name and number; grading is a binary state check (reward 1/0), independent of how the agent navigated there.
| Date | Model | Score | Source |
|---|---|---|---|
| 2026-02-24 | GUI-Owl-1.5 | 71.6% | GUI-Owl-1.5 introduces multi-platform fundamental GUI agents |
| 2025-10-22 | Surfer 2 | 87.1% | Surfer 2 achieves state-of-the-art cross-platform computer use via visual observation |
| 2025-09-02 | UI-TARS-2 | 73.3% | UI-TARS-2 advances GUI agent with multi-turn reinforcement learning |
| 2025-08-21 | GUI-Owl-7B | 66.4% | Mobile-Agent-v3 introduces GUI-Owl, setting new open-source SOTA on AndroidWorld and OSWorld |
| 2025-08-21 | Mobile-Agent-v3 | 73.3% | Mobile-Agent-v3 introduces GUI-Owl, setting new open-source SOTA on AndroidWorld and OSWorld |
| 2025-01-21 | UI-TARS | 46.6% | UI-TARS introduces native GUI agent model outperforming commercial frameworks |