Deepnet Spectra and the Two Cultures of Data Science

Overview

Abstract

Machine learning became a remarkable media story of the 2010s largely owing to its ability to focus researcher energy on attacking prediction challenges like ImageNet. Media extrapolations forecast the complete transformation of human existence. Unfortunately, machine learning has a troubled relationship with understanding the foundation of its achievements well enough to face demanding real-world requirements outside the prediction challenge setting. For example, its literature is admittedly corrupted by anti-intellectual and anti scholarly tendencies. It is beyond irresponsible to build a revolutionary transformation on such a shaky pseudo-foundation. In contrast, more traditional subdisciplines of data science like numerical linear algebra, applied probability, and theoretical statistics provide time-tested tools for designing reliable processes with understandable performance. Moreover, positive improvements in human well being have repeatedly been constructed using these foundations. To illustrate these points we will review a recent boomlet in the ML literature in the study of eigenvalues of Deepnet Hessians. A variety of intriguing patterns in eigenvalues were observed and speculated about in ML conference papers. We describe the work of Vardan Papyan showing that the traditional subdisciplines, properly deployed, can offer insights about these objects that ML researchers had.

Brief Biography

David Donoho is Anne T. and Robert M. Bass Professor of Humanities and Sciences and Professor of Statistics at Stanford University. He previously was a Professor at UC Berkeley and held visiting positions at Universite de Paris (VII and XI), the University of Leiden, and the National University of Singapore.

He has also worked in the industry, in oil exploration (Western Geophysical), information technology (co-founding BigFix, Inc., which was acquired by IBM), and quantitative finance (Renaissance Technologies).

He received the AB in Statistics from Princeton summa cum laude, where his undergraduate thesis advisor was John Tukey. He earned a Ph.D. in Statistics from Harvard University, where his thesis advisor was Peter Huber. Donoho was awarded honorary degrees from the University of Chicago, Ecole Fédérale Polytechnique de Lausanne, University of Waterloo, and the Technion.

Donoho is a member of the American Academy of Arts and Sciences, the US National Academy of Sciences, and a Foreign Associate of the French Académie des Sciences.  He has received a MacArthur Fellowship and the Committee of Presidents of Statistical Societies Presidents’ award. He was awarded the Norbert Wiener Prize of the American Mathematical Society and Society of Industrial and Applied Mathematics and the Gauss Prize of the International Mathematical Congress and German Mathematical Society. He received the 2013 Shaw Prize in Mathematical Sciences.