The authors introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) based on reversibility, scope crossing, and privilege expansion. This approach addresses the limitation of binary attack-success rates in agentic red-teaming benchmarks by providing detailed severity information.
The scale is computed using both a deterministic oracle and a panel of three frontier language-model judges. Across four victim models and two defenses on the AgentDojo workspace suite, the grading exposed cases hidden by binary metrics, such as a defense permitting cross-scope leaks despite zero attack success. The judge panel achieved high ordinal agreement with the oracle (Krippendorff's alpha = 0.91) but exhibited systematic blind spots like failing to recognize escalation chains.
Unlike prior work offering harm taxonomies or simulation, this contribution provides a reusable, trace-grounded severity instrument applied to actual red-team logs, with all code and prompts released.