On Distribution-Free Risk Control
- Michael Jordan, Professor Emeritus, University of California, Berkeley
B9 L4 R4225
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting.
Overview
Abstract
We introduce a framework for calibrating machine learning models so that their predictions satisfy explicit, finite-sample statistical guarantees. Our calibration algorithms work with any underlying model and (unknown) data-generating distribution and do not require model refitting. The framework addresses, among other examples, false discovery rate control in multi-label classification, intersection-over-union control in instance segmentation, and the simultaneous control of the type-1 error of outlier detection and confidence set coverage in classification or regression. We also present Conformal Decision Theory, a framework for producing safe autonomous decisions despite imperfect machine learning predictions. The decisions produced by our algorithms are safe in the sense that they come with provable statistical guarantees of having low risk without any assumptions on the world model. Experiments demonstrate the utility of our approach in robot motion planning around humans,
automated stock trading, and robot manufacturing.
Brief Biography
Michael I. Jordan is a researcher at INRIA and Professor Emeritus at the University of California, Berkeley. His research interests bridge the computational, statistical, cognitive, biological and social sciences. Prof. Jordan is a member of the National Academy of Sciences, a member of the National Academy of Engineering, a member of the American Academy of Arts and Sciences, and a Foreign Member of the Royal Society. He was the inaugural winner of the World Laureates Association (WLA) Prize in 2022. He was a Plenary Lecturer at the International Congress of Mathematicians in 2018. He has received the Ulf Grenander Prize from the American Mathematical Society, the IEEE John von Neumann Medal, the IJCAI Research Excellence Award, the David E. Rumelhart Prize, and the ACM/AAAI Allen Newell Award. In 2016, Prof. Jordan was named the "most influential computer scientist" worldwide in an article in Science, based on rankings from the Semantic Scholar search engine.
