A preregistered, placebo-controlled evaluation using the PoPE methodology found no operational benefit from providing error content to frozen small code models during self-repair. The study treated failed programs as conjectures and execution counterexamples as refutations, testing whether this falsifying evidence could be used by the model.
- In the prompt channel, a content-ablated form placebo unlocked 12 units compared to 10 for the live error-pattern arm on a resistant band, resulting in a mechanism-null finding.
- In the weight channel using small-data adapter training, an 8-8 tie was observed between the error-content adapter and the baseline (p=1.0).
- A SHA-deranged placebo adapter stayed ahead with 10 unlocks, but content-attributable superiority was not confirmed.
The authors argue that when representation learned from the oracle is written back into the generation state, testing is replaced by conditioning rather than operational improvement.