The article introduces a thesis by jlorraine that addresses the challenges of applying bilevel or nested optimization to large-scale deep learning setups. While gradient-based optimization typically updates a single set of parameters, many applications require updating subsets of parameters on different objectives nested inside each other.
- The work focuses on motivating examples such as hyperparameter optimization and generative adversarial networks.
- It highlights that naively applying classical methods often fails when solving these nested problems at scale.
- The thesis builds specific tools designed to make nested optimization scalable for deep learning contexts.