Benchmark · agentic

VisualWebArena

0 results 0 models

VisualWebArena measures how well multimodal (vision-capable) autonomous agents complete realistic, visually grounded tasks on live self-hosted websites (Classifieds, Shopping, Reddit); performance is reported as an overall task success rate over its 910 tasks.

Example: An image-conditioned browsing task: given a reference image of a product, the agent must navigate a classifieds or shopping site to locate the listing that matches the pictured item under a stated constraint (e.g., cheapest, a specific condition) and either open it or report a requested detail.
Scoring: The metric is success rate. Each task is graded 0 or 1 by a programmatic reward function that compares the agent's final answer or the resulting site state to a predefined target (exact-match, must-include, or fuzzy text match for answers; URL/state checks for navigational or transactional goals). The score is the fraction of tasks scoring 1.
Verification: Evaluation is execution-based: the agent drives a real browser against the sandboxed, self-hosted web apps (reset between tasks for reproducibility), and the reward function inspects the outcome programmatically, with no human judging. A task counts only if its automated check passes.
Why it matters: Real websites are visual, and text-only observations (accessibility tree or HTML) miss image-grounded information; VisualWebArena tests whether agents can perceive visual content and act at the same time, revealing a large human-agent gap and motivating multimodal agent techniques such as Set-of-Marks visual prompting.

Worked example

Task

Environment: the self-hosted Classifieds site. Intent (with an attached reference image of an office chair): "Go to the listing selling the item shown in this image and tell me its asking price."

Solution

Key steps: (1) read the reference image and note the chair's distinctive visual features; (2) browse/search the Classifieds furniture category; (3) match listing thumbnails against the reference image and open the correct listing; (4) read the asking price on that listing page; (5) submit it with the stop action, e.g. stop [<price>]. Final answer: the exact price shown on the matched listing (illustrative, e.g. stop [$45]).

Walkthrough

It is correct because the task can only be solved by grounding the reference image to the one matching listing (visual matching a text-only view cannot do) and then extracting the exact requested value. Grading: the reward function compares the submitted answer to the target price via string/fuzzy match, yielding a binary 0/1 that feeds the aggregate success rate.

No verified scores reported yet for this benchmark.