Researchers introduce requential coding, a method that bypasses the limitations of parameter-based compression by encoding the training trajectory through a teacher model selecting samples from the student's distribution. This approach records only the bits where the teacher and student disagree, resulting in code lengths independent of parameter count and data entropy.
- The method often produces codes orders of magnitude shorter than prequential coding, with advantages growing as scale increases.
- Larger models and ensembles compress to smaller sizes despite having more parameters when holding loss fixed.
- Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs.
- The technique isolates learnable information from random content, revealing that lower-entropy text holds more structure than higher-entropy image data.
This compression method provides tighter generalization bounds in the compute-optimal regime and predicts gradual overfitting during multi-epoch training.