SLORR introduces simple, stateless in-training low-rank regularization

Researchers introduce SLORR, a framework for in-training low-rank regularization that is stateless and architecture-preserving. It uses GPU-friendly approximations to regularize weight matrices without modifying the model or relying on cached quantities.

• SLORR includes variants based on the Hoyer sparsity metric and the nuclear norm.
• Evaluation on ImageNet-1K shows it induces compressibility with less than 8% training overhead for ResNet and ViT models.
• Tests on LLM pretraining at 135M and 560M scales show compressed models preserve performance better than unregularized ones with under 1% average overhead.