Researchers propose the Associative Recurrent Memory Transformer (ARMT) to enable long-context processing in large language models while achieving constant memory scaling and improved efficiency.
- The work introduces two domain-specific long-context datasets for evaluating realistic narrow-domain fine-tuning scenarios.
- A comprehensive training recipe is provided, combining continued pre-training, synthetic data generation, curriculum learning, and selective integration of associative memory into specific model layers.
- ARMT-augmented models process inputs well beyond original context limits without degrading performance relative to in-limit baselines.
- The approach generalizes more effectively to out-of-distribution context lengths.
- Models require 30% less FLOPs while preserving baseline performance within the original context window.
This method allows LLMs to handle longer inputs efficiently, addressing the quadratic compute and linear memory scaling constraints of standard transformers.