Benchmark · agentic
Terminal-Bench
Terminal-Bench tests how well AI agents complete real command-line tasks in a terminal — like installing software, debugging, and running system operations. The score is the percentage of tasks the agent finishes successfully.
Read more
- Example
- A typical task gives the agent a broken or empty environment and asks it to reach a working state — for instance, install the right dependencies and fix the configuration so a program runs, doing everything through shell commands.
- Scoring
- Each task is judged pass or fail by whether the agent reached the required end state, and the benchmark score is the percentage of tasks solved out of the total.
- Verification
- A result is accepted only if automated checks inside a sandboxed environment confirm the task's intended outcome was achieved — not by human votes or matching exact text.
- Why it matters
- Terminal skills like installing, debugging, and operating systems from the shell are central to real engineering work, so this benchmark shows whether an agent can act autonomously and reliably in an actual command line. Its harder 2.x version keeps the bar rising as agents get better.
Worked example
Task
In a Terminal-Bench Docker container, the file /app/access.log holds Apache access logs. Write a single command that counts the number of distinct client IP addresses (the first whitespace-separated field of each line) and saves just that number to /app/answer.txt.
Solution
awk '{print $1}' /app/access.log | sort -u | wc -l > /app/answer.txtWalkthrough
awk prints the first field (the client IP) of every log line, sort -u collapses duplicates, and wc -l counts the remaining unique IPs, redirected into /app/answer.txt. Terminal-Bench grades it by running a pytest test in the container that reads /app/answer.txt and checks it equals the known unique-IP count.