Benchmark · agentic

Android World

0 results 0 models

AndroidWorld is a benchmark and live-emulator environment (Google, 2024) for autonomous agents that control a real Android device; it has 116 hand-built tasks across 20 real apps, and the metric is task success rate.

Example: The agent is given a natural-language goal and must operate real apps to reach it — for instance, in a calendar app, create an event with a specified title, date, and time by tapping, typing, and scrolling through the live UI.
Scoring: Each task gives a binary reward: 1 if the final device state satisfies the task's success condition, else 0. The score is the success rate — the fraction of tasks solved — averaged over the suite (tasks are parameterized with randomized values, so runs vary).
Verification: Success is checked programmatically by inspecting the on-device system/app state after the run (e.g., querying an app's database, files, or settings), not by matching the action trajectory or using an LLM judge — which makes rewards durable and reproducible.
Why it matters: It evaluates agents on realistic, open-ended mobile device control with verifiable rewards; randomized parameters deter memorization, and real apps plus state-based grading make it a durable, reproducible testbed for multimodal computer-use agents.

Worked example

Task

Task (ContactsAddContact family): "Create a new contact for Grace Taylor. Their phone number is 555-123-4567." The agent acts on the live Contacts app via the accessibility tree and screenshots.

Solution

Open the Contacts app → tap "Add contact" → enter the name Grace Taylor → enter the number 555-123-4567 → tap "Save". Final state: the device's contacts store now contains a contact named Grace Taylor with number 555-123-4567.

Walkthrough

It's correct because the task's success checker queries the on-device contacts and confirms a contact exists with exactly the target name and number; grading is a binary state check (reward 1/0), independent of how the agent navigated there.

No verified scores reported yet for this benchmark.