ACE reduces repetition in continuous diffusion LMs by removing a self-conditioning attractor

Continuous diffusion language models like ELF achieve record-low generative perplexity (Gen-PPL) but suffer from excessive repetition that Gen-PPL erroneously rewards. The authors identify this issue as a contractive attractor along a single direction in the self-conditioning feedback loop and propose ACE (Attractor-Contrast-Escape) to subtract it.

• ACE subtracts a single, label-free direction from the feedback at each step, estimated once on the 105M model.
• The method cuts repetition to near human levels while maintaining competitive quality across 342M and 652M models and various samplers.
• Because Gen-PPL rewards repetition, the authors measure compute efficiency for producing human-clean text, finding ACE is 1.5–5x cheaper.

The study demonstrates that low perplexity scores can overstate model quality by masking repetitive behavior, and offers a dimension-specific fix to improve text naturalness without significant computational overhead.