This paper introduces UI-TARS, an end-to-end native GUI agent model that perceives screenshots as input and performs human-like keyboard and mouse interactions without relying on wrapped commercial models.
- Achieves state-of-the-art performance across 10+ GUI agent benchmarks evaluating perception, grounding, and task execution.
- Scores 24.6 with 50 steps and 22.7 with 15 steps on the OSWorld benchmark, outperforming Claude (22.0 and 14.9 respectively).
- Reaches a score of 46.6 in AndroidWorld, surpassing GPT-4o's 34.5.
- Incorporates enhanced perception via large-scale GUI screenshot datasets for precise captioning.
- Utilizes unified action modeling to standardize actions across platforms and achieve precise grounding.
- Implements System-2 Reasoning for multi-step decision making, including task decomposition and reflection.
- Uses iterative training with reflective online traces collected from hundreds of virtual machines to address data bottlenecks.
The authors analyze the evolution path of GUI agents to guide further development in this domain, demonstrating that native models can outperform sophisticated frameworks dependent on expert-crafted prompts.