Researchers have introduced LACUNA, the first unlearning testbed featuring ground-truth parameter-level localization to address the limitations of existing benchmarks that evaluate unlearning solely at the output level. The testbed injects synthetic personally identifiable information into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, allowing direct assessment of whether unlearning methods target the specific weights responsible for knowledge storage.
- LACUNA enables evaluation of localization precision by using synthetic individuals injected into OLMo model parameters.
- Benchmarking reveals that state-of-the-art methods are highly imprecise and susceptible to resurfacing attacks despite strong output-level performance.
- The study demonstrates that successful localization allows even simple gradient-based unlearning methods to achieve strong erasure and robustness.
The authors release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.