CO-LMLM introduces continuous-query limited memory language models

The authors propose Continuous-Query Limited Memory Language Models (CO-LMLM), a paradigm that externalizes factual knowledge to a knowledge base (KB) using continuous keys paired with textual values, departing from prior relational KB approaches. This design allows the model to generate flexible vector queries at minimal cost while integrating human-readable and attributable retrieved knowledge during generation.

• CO-LMLM utilizes an annotation pipeline that tags free-form factual spans in arbitrary text, removing restrictions to Wikipedia found in previous work.
• The model was pretrained on Wikipedia and FineWeb-Edu across multiple scales.
• At the 360M scale, CO-LMLM achieves lower perplexity than models pretrained on 40x more data.
• On SimpleQA verification, performance matches gpt-4o-mini and exceeds Claude Sonnet 4.5.

CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision, offering knowledge control capabilities beyond conventional large language models.