Researchers present Memory Decoder at Scale, a system that scales parametric long-term memory models up to 6.9B parameters and pretrains them on 300B tokens. To address the infeasibility of standard Faiss pipelines at this data scale, the authors implement a distributed indexing pipeline with sparse, batch-wise loading of kNN distributions.

  • The system scales memory models to 6.9B parameters and uses 300B tokens for pretraining.
  • A distributed pipeline handles Faiss indexing and retrieval, utilizing sparse, batch-wise loading of kNN distributions.
  • Pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters.
  • For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across three domains by more than 9 points at every scale.

The results demonstrate that independently scaling pretrained memory offers a more parameter-efficient path to improving language model performance compared to scaling the base model alone.