Study finds falsification via external counterexamples improves frozen code model repair over self-exposure

This study decomposes the value of self-repair feedback in small frozen code models using a placebo-controlled design to determine whether improvements stem from re-exposure to failing code or from executable criticism. The research challenges the standard retry mechanism by treating generated programs as conjectures and test-execution violations as counterexamples.

• Evaluated 290 dead task-cell units across six HumanEval+/MBPP+ cells using three 0.5B-1.5B frozen models.
• Blind resampling exceeded bare-code retry by +18 net unlocks (p=0.0021).
• Code-plus-facts feedback recovered +18 over bare code and +15 over a generic-bullet placebo.
• An instruction-only effect was not statistically distinguishable from the placebo.
• Falsification helped as comparison with external, executable counterexamples rather than vocabulary or self-critique.

The findings suggest that in deployment settings where retraining is infeasible, feedback value is derived from external, executable criticism rather than mere exposure to failing outputs.