Theoretical analysis shows Masked Image Modeling is more robust to non-IID data than Contrastive Learning

A recent study provides a rigorous theoretical analysis of the robustness of distributed self-supervised learning (D-SSL) frameworks under non-independent and identically distributed (non-IID) settings. The research demonstrates that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL). Additionally, the findings indicate that decentralized SSL robustness increases with average network connectivity, suggesting federated learning is no less robust than decentralized learning.

• Pre-training with MIM is more robust to data heterogeneity than Contrastive Learning.
• Robustness of decentralized SSL increases with average network connectivity.
• Federated learning is shown to be no less robust than decentralized learning.
• The authors introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization.

These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms and validate the effectiveness of the proposed MAR loss through extensive experiments.