The Many Faces of Compression: Theory and Practice in Federated Optimization
This thesis presents novel compression methods and optimization algorithms aimed at improving communication efficiency in large-scale machine learning systems.
Overview
Machine learning is at the core of modern AI applications, arguably shaping the technological landscape of the 21st century. However, the growing complexity of models and the ever-increasing demand for data impose a significant strain on communication networks, creating a major bottleneck in large-scale AI systems. This thesis explores a range of compression techniques designed to alleviate this problem.
Chapter 2 introduces a class of compression operators defined by a three point property, establishes their convergence, and identifies that EF21 [Richtárik et al., 2021] and LAG [Chen et al., 2018] compressors belong to this class. Building on this connection, we propose the CLAG method, which consistently outperforms EF21 and LAG in our experiments. Chapter 3 presents an adaptive compressor that dynamically adjusts to the compression error at each iteration, leading to better convergence in varying conditions. Chapter 4 revisits and sharpens the theoretical analysis of the EF21 [Richtárik et al., 2021] algorithm, specifically in scenarios where the data exhibits sparsity.
Chapters 5 and 6 concentrate on proximal methods. Chapter 5 offers a new analysis of the SVRP algorithm, showing advantages over the approach of Khaled and Jin in scenarios with high Hessian dissimilarity. Chapter 6 proposes a new proximal algorithm. We show, additionally, in the introduction that the proximity operator can be regarded as a form of compression operator.
Chapter 7 introduces randomized progressive training, which can be viewed as a sketch operator that helps reduce communication. Finally, Chapter 8 introduces the FLIX problem formulation, inspired by local training methods that reduce communication. FLIX is communication-efficient by design, with communication savings adjustable through a personalization parameter.
Each chapter presents novel insights, introduces theoretical analyses, and reinforces them with computational experiments.
Presenters
Brief Biography
Elnur Gasanov is a PhD candidate in the Optimization and Machine Learning Lab at the Center of Excellence for Generative AI (GenAI) at KAUST, where he is advised by Professor Peter Richtárik. His research focuses on distributed machine learning, stochastic optimization, and randomized linear algebra. Elnur holds a Master of Science in Computer Science from KAUST and a Bachelor's degree in Applied Mathematics and Physics from the Moscow Institute of Physics and Technology.