Study analyzes CLI coding agent failure trajectories across seven models

A large-scale empirical study introduces a process-oriented framework to analyze the onset, evolution, and recovery of failures in CLI coding agents, treating them as temporal processes rather than final outcomes.

• The researchers collected 3,843 execution trajectories from seven frontier models across three scaffolds (OpenHands, MiniSWE, Terminus2) on Terminal-Bench.
• After filtering, 1,794 complete and valid trajectories were manually annotated, covering over 63,000 execution steps.
• The analysis derived 14 findings regarding failure occurrence, root causes, recovery, and cross-system consistency.
• Failures are predominantly driven by epistemic errors, typically begin within the first few execution steps, and often remain hidden until recovery is impossible.

The authors argue that improving coding-agent reliability requires earlier validation and intervention rather than relying solely on final-outcome evaluation.