Benchmark · agentic

τ-bench

0 results 0 models

τ-bench measures how well an AI agent handles a multi-turn customer-service conversation while calling tools and APIs to fulfill a user's request. The score is the percentage of tasks the agent completes correctly.

Example: A task in a retail or airline domain — for example, a user asks the agent to cancel or change an order, and the agent must ask for the needed details, then call the right backend APIs while following the domain's policy rules.
Scoring: The metric is the fraction of tasks solved correctly; a task counts as solved when the final database or world state matches the task's expected outcome. Results are usually reported as a pass rate (pass@1), with a pass^k variant measuring consistency across repeated runs.
Verification: Verification is automatic rather than human-judged: after the conversation, the benchmark compares the resulting system state (and the information given to the user) against a predefined expected outcome, so only an objective match counts as a pass.
Why it matters: It reflects a realistic, commercially important use case — agents that must both converse and act through tools under business rules — and it highlights reliability gaps, since models that succeed once often fail when the same task is retried.

Worked example

Task

In τ-bench's retail domain, a simulated user tells the agent: 'Hi, I'd like to cancel order #W2611340 — I ordered it by mistake.' Following the store policy (only orders still in 'pending' status may be cancelled, and the reason must be an allowed value), the agent must authenticate the user, verify the order, confirm with the user, then cancel it via tool calls.

Solution

# 1) Authenticate the user (policy: verify identity before any action)
find_user_id_by_email(email="mia.li.3818@example.com")
# 2) Look up the order and confirm status == "pending"
get_order_details(order_id="#W2611340")
# 3) After the user confirms, cancel with an allowed reason enum
cancel_pending_order(order_id="#W2611340", reason="ordered by mistake")

Walkthrough

The policy permits cancelling only a 'pending' order and restricts the reason to two enum values ('no longer needed' / 'ordered by mistake'), so the agent authenticates first, checks the status, gets user confirmation, then issues the write call. τ-bench grades by comparing the final database state to the annotated goal state (the order now 'cancelled') and reports pass^k reliability across repeated trials.

No verified scores reported yet for this benchmark.