Monte Carlo method trains deep neural networks without backpropagation

A study demonstrates that a simple Monte Carlo algorithm can effectively train deep neural networks without relying on gradients or backpropagation. The method involves randomly mutating parameters and retaining changes only if the loss decreases, allowing it to bypass vanishing and exploding gradient issues.

• The approach works on single GPUs and does not require batch normalization or residual connections.
• It supports pure pruning training, discrete weights, and unconventional transfer functions like Gaussian.
• Feasibility was shown on networks with over 20 layers, wide networks with 16,384 neurons, and a Transformer on MNIST and Tiny Shakespeare.

This gradient-free method offers a complementary perspective on neural network self-organization and provides an alternative for building physically inspired deep learning systems.