Researchers have released Terminal-Bench 2.0, a benchmark designed to evaluate AI agents on difficult, realistic command-line interface tasks that current metrics fail to capture meaningfully.
The benchmark consists of 89 carefully curated tasks within computer terminal environments, each featuring unique setups, human-written solutions, and comprehensive verification tests. Evaluation results show that frontier models and agents score less than 65% on these tasks. The authors conducted an error analysis to identify specific areas for improvement in model and agent capabilities.
The dataset and evaluation harness are published to assist developers and researchers in advancing autonomous task completion in terminal environments.