ExecCritic introduces a framework that separates test construction from source-code repair to prevent false confidence in coding agents. The system employs a Test agent and a Repair agent, both backed by Qwen-3.5-35B-A3B, trained separately using reinforcement learning.

  • The Test agent learns to generate behaviorally valid tests that distinguish correct patches from incorrect ones.
  • The Repair agent uses execution feedback from these tests to revise source code without altering the test suite.
  • On SWE-bench Verified, post-trained Qwen agents achieved a 72.6% success rate, an 11.4-point gain over the no-test baseline.
  • Test quality is critical; base model tests reduced performance, while GPT-5.6-sol tests improved it to 65.3%.

This approach allows coding agents to improve repository repairs through reliable execution feedback without requiring stronger models or Oracle feedback during evaluation.