Adaptive Optimization Algorithms for Machine Learning
Machine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This dissertation investigates adaptivity in machine learning optimizers.
Overview
Abstract
Machine learning assumes a pivotal role in our data-driven world. The increasing scale of models and datasets necessitates quick and reliable algorithms for model training. This dissertation investigates adaptivity in machine learning optimizers. The ensuing chapters are dedicated to various facets of adaptivity, including: 1. personalization and user-specific models via personalized loss, 2. provable post-training model adaptations via meta-learning, 3. learning unknown hyperparameters in real time via hyperparameter variance reduction, 4. fast O (1/k^2) global convergence of second-order methods via stepsized Newton method regardless of the initialization and choice basis, 5. fast and scalable second-order methods via low-dimensional updates. This thesis contributes novel insights, introduces new algorithms with improved convergence guarantees, and improves analyses of popular practical algorithms.
Brief Biography
Slavomir Hanzely is an MS/Ph.D. student in the Applied Mathematics and Computational Science Program under the supervision of Professor Peter Richtarik at the AI Initiative (AII) at King Abdullah University of Science and Technology (KAUST). Slavomir earned his Bachelor's Degree in Informatics with a background in both Mathematics and Computer Science from the Faculty of Mathematics, Physics and Informatics University (FMPH UK), Slovakia in 2019. He is interested in Machine Learning, Federated Learning, and Stochastic Algorithms.