Researchers introduce LACUNA, the first unlearning testbed featuring ground-truth parameter-level localization to evaluate whether model unlearning truly erases knowledge or merely obfuscates it. The testbed injects synthetic personally identifiable information into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining.
- LACUNA enables direct evaluation of whether unlearning methods target the specific weights responsible for knowledge storage.
- Benchmarking reveals that state-of-the-art methods are highly imprecise and susceptible to resurfacing attacks despite strong output-level performance.
- The study demonstrates that successful localization allows even simple gradient-based methods to achieve strong erasure and robustness against resurfacing.
The authors release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.