The Hong Kong University MMLab introduces UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. It addresses limitations of existing benchmarks by using live Docker containers and a closed-loop evaluation strategy with hidden supervisor agents.
- Built around five foundational capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination.
- Includes 400 bilingual real-world tasks evaluated via fine-grained, step-by-step completion checkpoints.
- Uses a closed-loop strategy with executor, hidden supervisor, and user agents to simulate multi-turn feedback without leaking grading criteria.
- Evaluates state-of-the-art models under multiple frameworks to disentangle base model capabilities from framework design choices.
The benchmark aims to help researchers identify root causes of agent failures and understand how base models and frameworks jointly shape performance.