ToolFailBench diagnoses tool-use failures in LLM agents across 1,000 tasks

The authors introduce ToolFailBench, a diagnostic benchmark designed to measure specific types of tool-use failures in large language model agents that aggregate scores often obscure. The benchmark evaluates performance across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate by distinguishing between tool-required tasks and control tasks.

• ToolFailBench labels traces with four failure modes: Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use.
• Evaluation uses a rule classifier and two LLM judges aggregated by majority vote to identify these specific errors.
• Testing 19 headline models revealed that the best achieved an 86.33% Clean Tool-Use Rate, indicating faithful tool use is not saturated.
• Models with similar aggregate scores fail in different ways; for instance, Llama-3.1 models exhibit an Always-Call pattern while Qwen2.5-72B and Llama-3.1-70B differ by 89 percentage points on control-task accuracy.

The authors argue that tool-use evaluation must measure not only whether agents call tools but also whether they use outputs correctly and avoid unnecessary calls.