Facebook introduces Sparse Delta Memory to scale linear RNNs via sparse addressing

Researchers at Facebook have introduced Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory.

• Under an isoFLOP constraint and with an identical number of parameters, higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks.
• By learning the initial state of the SDM memory to use as parametric memory, the model further improves on common-knowledge and reasoning tasks.

The authors consider this important because increasing the state size of linear attention typically costs higher FLOPs, whereas SDM achieves better recall without that computational penalty.