Benchmark · agentic
VisualWebArena
VisualWebArena measures how well multimodal (vision-capable) autonomous agents complete realistic, visually grounded tasks on live self-hosted websites (Classifieds, Shopping, Reddit); performance is reported as an overall task success rate over its 910 tasks.
Read more
- Example
- An image-conditioned browsing task: given a reference image of a product, the agent must navigate a classifieds or shopping site to locate the listing that matches the pictured item under a stated constraint (e.g., cheapest, a specific condition) and either open it or report a requested detail.
- Scoring
- The metric is success rate. Each task is graded 0 or 1 by a programmatic reward function that compares the agent's final answer or the resulting site state to a predefined target (exact-match, must-include, or fuzzy text match for answers; URL/state checks for navigational or transactional goals). The score is the fraction of tasks scoring 1.
- Verification
- Evaluation is execution-based: the agent drives a real browser against the sandboxed, self-hosted web apps (reset between tasks for reproducibility), and the reward function inspects the outcome programmatically, with no human judging. A task counts only if its automated check passes.
- Why it matters
- Real websites are visual, and text-only observations (accessibility tree or HTML) miss image-grounded information; VisualWebArena tests whether agents can perceive visual content and act at the same time, revealing a large human-agent gap and motivating multimodal agent techniques such as Set-of-Marks visual prompting.
Worked example
Task
Environment: the self-hosted Classifieds site. Intent (with an attached reference image of an office chair): "Go to the listing selling the item shown in this image and tell me its asking price."
Solution
Key steps: (1) read the reference image and note the chair's distinctive visual features; (2) browse/search the Classifieds furniture category; (3) match listing thumbnails against the reference image and open the correct listing; (4) read the asking price on that listing page; (5) submit it with the stop action, e.g.
stop [<price>]. Final answer: the exact price shown on the matched listing (illustrative, e.g. stop [$45]).Walkthrough
It is correct because the task can only be solved by grounding the reference image to the one matching listing (visual matching a text-only view cannot do) and then extracting the exact requested value. Grading: the reward function compares the submitted answer to the target price via string/fuzzy match, yielding a binary 0/1 that feeds the aggregate success rate.
| Date | Model | Score | Source |
|---|---|---|---|
| 2026-01-12 | ColorBrowserAgent | 47.4% | ColorBrowserAgent introduces adaptive knowledge evolution for robust long-horizon web automation |
| 2025-01-23 | CUA | 87.0% | OpenAI introduces Computer-Using Agent (CUA) powering Operator research preview |