An independent researcher has released a preprint detailing a 3.08M-parameter DeepSeek-style transformer that uses a trained fast-weight memory bank to install and apply never-trained rules during inference without any backward pass. The model writes its own forward pass outputs into an 8-slot vector bank, which is then read as weights via a hypernetwork, enabling continual learning at inference time without test-time training (TTT), optimizers, or growing context. Key findings include:
- A single 13-token presentation installs a never-trained rule with 0.79–1.00 accuracy on unseen queries, surviving slot eviction and allowing mid-conversation replacement with zero old-rule persistence.
- Test-time training fits adaptation examples (0.99) but transfers exactly nothing to unseen queries at 138x the cost per rule update, while in-window in-context learning remains at chance.
- The memory policy is determined by the training distribution; randomizing conversation structure during training installs the necessary keep/overwrite/write-on-dirty behavior, whereas fixed structures cause total perseveration.
The work highlights that breaking the read/write circuit requires specific training techniques like teacher-forced bootstrap and annealing, as the joint gradient has an ignore-the-bank fixed point. All experiments, including three 5-hour training runs on a single RTX 3090, are fully reproducible via provided code and Zenodo DOI.