Researchers introduce DevicesWorld, a large-scale executable benchmark designed to evaluate LLM-based agents operating across heterogeneous environments including mobile, desktop, and IoT devices.
- The benchmark contains 6,140 tasks that integrate three classes of device environments into a unified framework for cross-device interaction.
- Each task defines natural-language goals, participating devices, initial states, executable actions, rule-based verifiers, and cleanup procedures.
- Evaluation of five frontier LLM-agent systems reveals low success rates, with the best performing only 12.5%.
- Analysis shows agents frequently struggle with information acquisition, interface manipulation, and confusing source versus output devices.
DevicesWorld provides a reproducible and diagnostically useful evaluation problem to advance research on reliable cross-device agents.