IsoLoCo uses Iso-C model merging to improve DiLoCo aggregation

Researchers propose IsoLoCo, a method that adapts the Iso-C model merging technique for distributed optimization to enhance the DiLoCo algorithm. By integrating Iso-C with Nesterov momentum, the approach aims to address performance degradation in communication-efficient methods as the number of local models increases.

• The work establishes an analogy between pseudo-gradient aggregation in local SGD/DiLoCo and task arithmetic-based model merging.
• IsoLoCo outperforms both simple pseudo-gradient averaging and momentum-based DiLoCo without requiring its own momentum mechanism initially.
• Empirical evaluations on language model pre-training show that IsoLoCo significantly outperforms DiLoCo, with the performance gap widening as the number of workers increases.
• This advantage holds across different model sizes and inner step counts, confirming merging-inspired aggregation is effective for low-communication distributed training.

This strategy offers a more effective approach for low-communication distributed training by leveraging model merging techniques to improve aggregation quality.