Graduate Seminars

back Back to all Graduate Seminars

CS Graduate Seminar: Communication Efficient Variants of SGD for Distributed Computing

Start Date: November 12, 2018
End Date: November 12, 2018

By Dr. Sebastian Stich (EPFL, Switzerland)​

Nowadays machine learning problems require stochastic optimization algorithms that can be implemented on distributed systems. The communication overhead of the algorithms is a key bottleneck that hinders perfect scalability. In this talk we will discuss two techniques that aim to reduce the communication costs of stochastic gradient descent (SGD). First, we discuss quantization and sparsification techniques that reduce the amount of data that needs to be communicated. We present a variant of SGD with k-sparsification (for instance top-k or random-k) and show that this scheme converges at the same rate as vanilla SGD. That is, the communication can be reduced by a factor of the dimension of the problem whilst still converging at the same rate. In the second half of the talk we discuss strategies that reduce the communication frequency instead of the communicated data. In particular, we compare local SGD (independent runs of SGD in parallel) with mini batch SGD. Joint work with Jean-Baptiste Cordonnier and Martin Jaggi.
 
Biography: Dr. Sebastian Stich is a postdoctoral researcher at EPFL in Switzerland, working at the Machine Learning and Optimization Laboratory of Prof. Martin Jaggi. He received a MSc in Mathematics with distinction from ETH Zurich in 2010 and a PhD in Theoretical Computer Science from ETH Zurich in 2014. Before joining EPFL, he held for two years a postdoctoral position at UCLouvain to work with Prof. Yurii Nesterov and Prof. François Glineur on coordinate descent methods for large scale optimization problems. Dr. Stich is broadly interested in the complexity analysis of the optimization algorithms that are used in nowadays machine learning applications, with recent focus on distributed algorithms that allow to tackle high dimensional problems.
 

More Information:

For more info contact: Prof. Peter Richtarik: email: Peter.Richtarik@KAUST.EDU.SA
 
Date: Monday 12th Nov 2018
Time:12:00 PM - 01:00 PM
Location: Bldg.9, Level 2, Hall 1
Refreshments: Light Lunch will be served at 11:45 AM