Researchers introduce TestEvo-Bench, a live benchmark designed to evaluate how well agents handle the co-evolution of software tests and code. Unlike existing benchmarks that rely on static metadata, this framework uses real commit histories and environment configurations to verify executable metrics like pass rate and coverage.
The benchmark features two tracks: test generation, where agents write new tests for new behavior, and test update, where they adapt failing tests to changed code. It contains 746 test generation and 509 test update tasks curated from 152 open-source Java projects. The live nature of the benchmark allows evaluation restricted to tasks postdating a model's training cutoff to reduce data leakage.
Experiments with agents combining Claude Code, Gemini CLI, and SWE-Agent show success rates up to 77.5% on test generation and 74.6% on test update. However, performance drops significantly on recent tasks and under limited per-task cost constraints.