Abstract
In statistical applications, there are often many competing models available, from which one would like to choose the best to produce the final results. Typically the ranking of these models is determined by their forecasting ability, as measured by some scoring function such as the mean absolute error (MAE) or the mean squared error (MSE).
The scoring function is said to be a proper scoring rule if it, in the long run, selects the true model if that is available. Proper scoring rules have a long history in the forecasting literature, especially in weather forecasting. In this talk, we begin by a brief introduction to proper scoring rules and their use in statistics. Then, we discuss an often overlooked problem: the up-weighting of observations with large uncertainty, which can lead to unintuitive rankings of models, by some of the most popular proper scoring rules, such as the continuously ranked probability score (CRPS), the MAE, and the MSE.
To overcome this problem, we derived a new class of scoring rules and, we propose the scaled CRPS (SCRPS). The proposed proper scoring rule works in the case of varying uncertainty and is, like the CRPS, computationally available for output from ensemble forecasts and does not require the evaluation of the density of the forecast. We illustrate the findings through applications from finance, spatial statistics, and regression analysis.
Brief Biography
David Bolin is an associate professor of statistics in the CEMSE Division at KAUST. Prior to joining KAUST, he was an associate professor in mathematical statistics at the University of Gothenburg. He received his PhD degree in mathematical statistics from Lund University in 2012 and also holds an M.S. in Engineering Mathematics from Lund University. Bolin‘s main research interests are in stochastic partial differential equations and their applications in statistics, with a focus on the development of practical, computationally efficient tools for modeling non-stationary and non-Gaussian processes. He also works on uncertainty quantification and visualization for applications in spatial statistics and medical imaging.