This paper extends recent theoretical results on gradient descent with large step sizes from isolated flat minima in scalar-output least-squares problems to vector-valued outputs and manifolds of flat minima. The authors generalize the normal form and three convergence theorems established by MacDonald et al. to this broader setting, addressing technical challenges such as solving a singular partial differential equation.
- The framework applies to overparametrized least-squares with vector-valued outputs, including regression with arbitrarily many observations.
- The analysis covers neighborhoods of a manifold of flat minima, which is essential for applications like matrix factorization.
- The authors prove that the set of flat minima forms a fiber bundle over a product of spheres and that sharpness is Morse-Bott along this manifold.
- The framework yields new structural results for deep matrix factorization under mild assumptions.
This work provides a rigorous theoretical foundation for understanding gradient descent dynamics in complex landscapes with flat minima, which are common in deep learning applications.