A study proposes a causal auditing framework to evaluate factual deletion in Limited Memory Language Models (LMLMs), which externalize knowledge to databases for unlearning. The framework isolates parametric leakage from retrieval-mediated correctness and artifacts by varying database states at inference time.
- The authors tested 12,228 alias-closure deletions across thirteen databases with four adversarial topologies and six prompt formulations.
- Parametric leakage was found to be near zero, meaning the model rarely returns deleted answers without retrieval.
- Residual knowledge persists primarily through near-neighbor retrieval, with rates ranging from 0.7% to 13.6% depending on the database topology.
- Prompt formulation did not independently control the survival of deleted facts.
The results indicate that for this class of LMLM, the effectiveness of unlearning is determined by the database administrator's management of the retrieval graph rather than the model's internal parameters.