SLORR introduces simple in-training low-rank regularization with minimal overhead

Researchers introduce SLORR, a stateless and architecture-preserving framework for in-training low-rank regularization that avoids the need for SVDs or additional trainable parameters. The method uses GPU-friendly approximations to regularize original weight matrices, instantiated via Hoyer sparsity metric and nuclear norm variants.

• Evaluated on ImageNet-1K with ResNet-50, ViT-B/16, ViT-L/16, and ResNet-18, SLORR induces compressibility with less than 8% training overhead.
• In LLM pretraining at 135M and 560M scales, SLORR-Hoyer allows compressed models to preserve performance substantially better than unregularized ones while adding less than 1% average training overhead.

This approach enables effective model compression during training without significantly impacting computational efficiency or altering the underlying architecture.