An observational study of 90 independent agent runs building a real-time retrospective board challenges the assumption that added capabilities like testing tools improve software quality. The research found that increasing reasoning effort from High to xHigh lifted first-try perfect runs from 28 percent to 89 percent, whereas adding a testing tool raised costs by 42 to 68 percent without improving functional scores.
- Frontier models clustered near the ceiling while a low-cost local model scored between 24 and 37 points on a 42-point rubric.
- Container deployment was the dominant defect, failing first try in 44 percent of runs, with failure rates shifting sharply across model generations.
- The testing tool increased costs by 42 to 68 percent but did not improve functional scores or reliability.
- A design-oriented prompt raised visual quality from 3.0 to 4.5 on a 5-point scale without affecting function.
The study concludes that matching the fix to the failure is critical, as most first-run failures stem from weak reasoning rather than visible flaws that checking tools would catch.