Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions

This study establishes feature-learning consistency guarantees for a broad subclass of deep neural networks characterized by sublinear growth in input/output dimensions and hidden neurons relative to sample size. The authors prove that these architectures achieve universal approximation for hierarchically compositional functions, even within the conventional over-parameterized regime where parameters exceed training samples.

• Sublinearly structured DNNs attain feature-learning and prediction consistency comparable to classical models when learning hierarchically compositional target functions.
• Empirical results show sublinearly structured networks match or surpass wide DNNs in prediction performance.
• A structural audit confirms that widely used convolutional neural networks, including AlexNet, VGGNet, ResNet, and GoogLeNet, are sublinearly structured on image classification benchmarks.
• The research provides a statistical explanation for the success of large-scale deep learning models trained on massive image datasets due to the inherent hierarchical structure of images.