Benchmark · agentic
τ²-bench
τ²-bench is an updated version of τ-bench that tests tool-using conversational AI agents on "dual-control" tasks, where both the agent and a simulated user can take actions to get a job done. The headline metric is the percentage of tasks solved (the average reward across tasks).
Read more
- Example
- A tech-support scenario: the agent looks up a customer's account and follows the domain's policy while guiding a simulated user to carry out steps on their own device (for example, changing a setting), and the two must coordinate to resolve the issue.
- Scoring
- Each task ends with an automatic check of whether the final system/database state matches the required outcome, giving a reward per task; the reported score is the percentage of tasks solved (or the average reward).
- Verification
- Acceptance is fully programmatic — the environment compares the end state (and required actions) against the expected result with no human grading, so a task counts only when the outcome matches.
- Why it matters
- It matters because real assistants often can't act alone — they must instruct and coordinate with a user — and this dual-control setup exposes communication and coordination failures that single-actor tool benchmarks miss.
Worked example
Task
τ²-bench, telecom (dual-control) domain: a simulated user writes, 'Since this morning I have no mobile data, but calls and texts still work.' Given the carrier's policy document and tools — plus device actions the user can perform on their own phone — authenticate the customer, diagnose the fault, and restore mobile data.
Solution
Correct agent trajectory (telecom tools + user-side device actions):
1. get_customer_by_phone(number); confirm identity via name + DOB.
2. get_line_status(line_id) → "active"; get_network_status(region) → "operational" (rules out account suspension and network outage).
3. Follow the "data down / voice up" branch: user confirms Airplane Mode is OFF, then agent instructs the user to switch "Mobile Data" ON.
4. Verify data is restored; apply NO billing credit (policy forbids a fee/credit for a self-service fix).
Resolution → user enables Mobile Data; connectivity restored, account state unchanged.
Walkthrough
With voice and SMS working while data is down on an active line with no network outage, the fault is device-side, so the policy troubleshooting tree resolves it by guiding the user to re-enable Mobile Data (and forbids any goodwill credit for a self-service fix). τ²-bench scores a per-task reward (0/1) by checking the environment/database reaches the expected end state and that all required actions and communicated info are present, then reports reliability across repeated trials via the pass^k metric.