Benchmark · agentic

AgentBench

0 results 0 models

AgentBench evaluates LLMs as interactive agents across 8 distinct environments (code-, game-, and web-grounded), scoring multi-turn decision-making per environment and combining them into a single weighted Overall Score.

Example: An Operating System task: given a natural-language goal (e.g., count the files matching a condition or set a file's permissions), the agent issues bash commands round by round, reads each output, then submits a final answer or leaves the shell in the required state.
Scoring: Each environment uses its own metric: success rate (OS, DB, House-Holding), F1 (Knowledge Graph), average reward (Web Shopping), step/element accuracy (Web Browsing), or game progress/reward (card game, puzzles). The headline Overall Score is a weighted average of the eight environment scores, with per-task weights chosen to balance difficulty so no single easy task dominates.
Verification: Every task runs in an executable environment with an automatic, deterministic checker: the agent's multi-turn trajectory is verified programmatically (e.g., a shell check script, the resulting database state, the WebShop reward, or a match against ground-truth actions). No human judging; interactions are capped at a maximum number of rounds, and exceeding the cap or emitting an invalid action counts as failure.
Why it matters: It was among the first standardized benchmarks to test LLMs as agents rather than single-turn responders, exercising long-horizon reasoning, tool use, and error recovery; it exposed a wide gap between top commercial API models and open-source models on real agentic tasks.

Worked example

Task

Operating System environment. In an Ubuntu bash shell, the agent is told: 'How many user accounts on this system have /bin/bash as their login shell? Give the number.' It may run several shell commands over multiple rounds before answering.

Solution

grep -c ':/bin/bash$' /etc/passwd

Walkthrough

The seventh colon-separated field of each /etc/passwd line is the login shell, so counting lines ending in :/bin/bash yields the number of bash-login accounts; the agent reads that count from the command output and returns it via the environment's answer(N) action. Grading: the OS checker runs a verification script and scores the submission as a binary success/fail.

No verified scores reported yet for this benchmark.