CO-LMLM introduces continuous-query limited memory language models

The authors propose CO-LMLM, a limited memory language model that externalizes factual knowledge to a knowledge base pairing continuous keys with textual values. This design allows the model to generate flexible vector queries at minimal cost while integrating human-readable and attributable retrieved knowledge.

• The system uses an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior restrictions to Wikipedia.
• CO-LMLM was pretrained on Wikipedia and FineWeb-Edu across multiple model scales.
• At the 360M scale, it achieves lower perplexity than models pretrained on 40x more data.
• It attains SimpleQA-verified performance in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

CO-LMLM outperforms prior limited memory language models and vanilla large language models in both perplexity and factual precision.