memaudit is a tool that integrates into Hugging Face TRL's SFTTrainer to provide evidence that fine-tuning on private or customer data did not result in memorization. It addresses the common lack of verification for legal and security requirements by embedding calibrated decoy secrets directly into the training dataset.
- The tool uses an inject() function to plant decoys and a MemorizationAuditCallback to monitor them during training.
- It verifies that every decoy receives loss under the specific configuration, including masking, packing, LoRA/QLoRA, and chat templates, failing the run if verification fails.
- The final JSON report answers two distinct questions: membership inference (can an attacker identify trained records) and regurgitation (does the model complete training prefixes).
- A public proof run on TinyLlama-1.1B-Chat with 20k Alpaca data showed a leakage curve where duplication correlated with exposure, while regurgitation was zero under exact-match protocols.
- The output is versioned, includes checksums, maps to EDPB Opinion 28/2024, and contains no external calls.
This allows teams to generate verifiable, local reports that can be forwarded to regulators or customers to prove data privacy compliance without relying on silent failures of naive tests.