Researchers propose Memorization-guided Data Reuse, a training paradigm that uses model memorization signals to adaptively determine when and how high-quality data should be reused during large language model training. This approach aims to improve sample efficiency and performance by moving beyond the traditional one-pass or fixed-epoch constraints.
The method derives "Memorization Window" signals from loss retention dynamics and downstream evaluation scores to make principled decisions on training epochs and data replay scheduling. Preliminary experiments indicate a memorization-driven regime where model performance continues to improve with repetition significantly beyond the commonly cited four-epoch limit.
These insights provide a foundation for memorization-aware training schedules, helping to determine appropriate reuse budgets and enabling more efficient LLM training with limited high-quality data.