Systematic evaluation of 25 learning rate schedulers across 30 architectures

A study systematically investigates the impact of learning rate scheduling strategies on classification accuracy across diverse neural network architectures, moving beyond treating schedulers as secondary hyperparameters. The researchers evaluated 30 representative models from convolutional and transformer families within the LEMUR neural network dataset.

• Automated source-code injection applied 25 scheduler configurations across nine PyTorch families.
• A total of 3,938 model variants were evaluated on the CIFAR-10 dataset.
• The best configuration achieved a top-1 accuracy of 86.45%, with 237 variants exceeding 80%.
• CosineAnnealingWarmRestarts and CyclicLR consistently outperform basic decay strategies.

The resulting accuracy landscape has been contributed to the LEMUR nn-dataset, providing a practical reference for principled scheduler selection.