CO-LMLM introduces continuous-query limited memory language models

The authors propose CO-LMLM, a limited memory language model that externalizes factual knowledge to a knowledge base pairing continuous keys with textual values. This approach allows the model to generate flexible vector queries at minimal cost while integrating human-readable and attributable retrieved knowledge during generation.

• The system uses an annotation pipeline that tags free-form factual spans in arbitrary text, removing the restriction to Wikipedia found in prior work.
• CO-LMLM was pretrained on Wikipedia and FineWeb-Edu across multiple model scales.
• At the 360M scale, it achieves lower perplexity than models pretrained on 40x more data.
• It attains SimpleQA-verified performance comparable to gpt-4o-mini and higher than Claude Sonnet 4.5.

CO-LMLM outperforms prior limited memory language models and vanilla LLMs in both perplexity and factual precision, offering knowledge control capabilities beyond conventional large language models.