Learning to Benchmark: predicting performance limits from data
The objective of benchmark learning is to use a training sample to learn about fundamental limits on performance of a classifier or other statistical inference procedure. This meta-learning problem is a crucial component of data science and interpretable AI. Examples include sequential design of experiments, reinforcement learning and sensor management in the fields of statistics, machine learning and systems engineering, respectively. The challenge is learn about best achievable accuracy directly from the data sample without having to approximate and mplement an optimal classifier algorithm. In this talk we will introduce a general information theoretic framework that yields benchmark learners having both linear computational complexity and linear sample complexity. We will illustrate how this framework in the context of benchmarking image classification, autonomous navigation, and deep neural network performance.
Overview
Abstract
The objective of benchmark learning is to use a training sample to learn about fundamental limits on performance of a classifier or other statistical inference procedure. This meta-learning problem is a crucial component of data science and interpretable AI. Examples include sequential design of experiments, reinforcement learning and sensor management in the fields of statistics, machine learning and systems engineering, respectively. The challenge is learn about best achievable accuracy directly from the data sample without having to approximate and mplement an optimal classifier algorithm. In this talk we will introduce a general information theoretic framework that yields benchmark learners having both linear computational complexity and linear sample complexity. We will illustrate how this framework in the context of benchmarking image classification, autonomous navigation, and deep neural network performance.
Brief Biography
Alfred O. Hero III is the John H. Holland Distinguished University Professor of Electrical Engineering and Computer Science and the R. Jamison and Betty Williams Professor of Engineering at the University of Michigan. His primary appointment is in the Department of Electrical Engineering and Computer Science and he also has appointments, by courtesy, in the Department of Biomedical Engineering and the Department of Statistics. He was a founding Co-Director of the University’s Michigan Institute for Data Science (MIDAS) (2015-2018). He received the B.S. (summa cum laude) from Boston University (1980) and the Ph.D from Princeton University (1984), both in Electrical Engineering. He is a Fellow of the Institute of Electrical and Electronics Engineers (IEEE). He is a Section Editor of the SIAM Journal on Mathematics of Data Science and a Senior Editor of the IEEE Journal on Selected Topics in Signal Processing . He serves as moderator for the Electrical Engineering and Systems Science category of the arXiv . He is co-General Chair of the IEEE International Symposium on Information Theory (ISIT). He has served as President of the IEEE Signal Processing Society and as a member of the IEEE Board of Directors. He has received numerous awards for his scientific research and service to the profession including several best paper awards, the IEEE Signal Processing Society Technical Achievement Award in 2013 and the 2015 Society Award, which is the highest career award bestowed by the IEEE Signal Processing Society. He has received a Rackham Distinguished Faculty Achievement Award in 2011, the 2017 Stephen S. Attwood Excellence in Engineering Award and the 2018 H. Scott Fogler Award for Professional Leadership and Service from the University of Michigan. Alfred Hero’s recent research interests are in high dimensional spatio-temporal data, multi-modal data integration, statistical signal processing, and machine learning. Of particular interest are applications to social networks, network security and forensics, computer vision, and personalized health.
