Abstract
When there are a lot of training data, or the deep neural network is too large, parallel training becomes essential. Parallel training refers to either data or model parallelism. In both cases, parallel execution introduces various overheads. The research of our group focuses on the development of algorithms and implementation of systems that minimize such overheads. This talk will briefly present four of our proposals: (i) Grace, a framework for the implementation of compressed communication for parallel training; (ii) DeepReduce, a framework for the efficient communication of sparse tensors; (iii) FastBack, a system that reduces the serial delay of backpropagation in model parallelism; and (iv) CompressNAS, an adaptive compressed communication system for Neural Architecture Search.
Brief Biography
Panos Kalnis is a Professor at the King Abdullah University of Science and Technology (KAUST, http://www.kaust.edu.sa) and served as Chair of the Computer Science program from 2014 to 2018. In 2009 he was visiting assistant professor at Stanford University. Before that, he was assistant professor at the National University of Singapore (NUS). In the past he was involved in the designing and testing of VLSI chips and worked in several companies on database designing, e-commerce projects and web applications. He has served as associate editor for the IEEE Transactions on Knowledge and Data Engineering (TKDE) from 2013 to 2015, and on the editorial board of the VLDB Journal from 2013 to 2017. He received his Diploma from the Computer Engineering and Informatics Department, University of Patras, Greece in 1998 and his PhD from the Computer Science Department, Hong Kong University of Science and Technology (HKUST) in 2002. His research interests include Big Data, Parallel and Distributed Systems, Large Graphs and Systems for Machine Learning.