The memaudit tool provides an open-source, local audit for training-data memorization in LLMs fine-tuned with Trainer or TRL. It addresses two critical privacy concerns: membership inference (detecting if specific data was used) and regurgitation (verifying if content is emitted verbatim).
- Injects pre-registered "canary" records into training data to enable detection.
- Includes a PEFT-aware pre-flight check to prevent silent failures from frozen LoRA embeddings.
- Reports both membership inference and regurgitation metrics, as they may disagree.
- Fully local with no external dependencies or account requirements.
The tool helps users document compliance with privacy regulations by providing concrete evidence of memorization risks rather than relying on synthetic examples.