On the Use of "Conventional" Unconstrained Minimization Solvers for Training Regression Problems in Scientific Machine Learning
This talk introduces PETScML, a framework leveraging traditional second-order optimization solvers for use within scientific machine learning, demonstrating improved generalization capabilities over gradient-based methods routinely adopted in deep learning.
Overview
In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis, utilizing deep-learning techniques, of data produced by computational science and engineering applications. At the core of these methods is the supervised training algorithm to learn the neural network realization, a highly non-convex optimization problem that is usually solved using stochastic gradient methods. However, distinct from deep-learning practice, scientific machine-learning training problems feature a much larger volume of smooth data and better characterizations of the empirical risk functions, which make them suited for conventional solvers for unconstrained optimization.
In this talk, we introduce PETScML, a lightweight software framework built on top of the Portable and Extensible Toolkit for Scientific Computation (PETSc) to bridge the gap between deep-learning software and conventional solvers for unconstrained minimization. Using PETScML, we empirically demonstrate the superior efficacy of a trust region method based on the Gauss-Newton approximation of the Hessian in improving the generalization errors arising from regression tasks when learning surrogate models for a wide range of scientific machine-learning techniques and test cases. All the conventional solvers tested, including L-BFGS and inexact Newton with line-search, compare favorably, either in terms of cost or accuracy, with the adaptive first-order methods used to validate the surrogate models.
Presenters
Brief Biography
Stefano Zampini earned his PhD in Computational Mathematics from the University of Milan in 2010. His work mainly focused on non-overlapping domain decomposition preconditioners of the dual-primal type (namely, BDDC and FETI-DP type methods) for solving large and sparse linear systems arising from finite elements discretizations and IsoGeometric Analysis. Before joining KAUST in 2014, he worked for the Italian Supercomputing center CINECA, with a specific interest in optimization and parallelization of oil and gas applications, and for the Italian weather forecast agency.
While a theorist by training, he spent his working career in the design and implementation of algorithms for the simulation of physical applications including electromechanical cardiology, computational fluid dynamics, electromagnetics, geophysics, chemistry, isogeometric analysis, fractional diffusion, and PDE constrained optimization. His contributions to the field of Domain Decomposition have been recognized by two plenary talk invitations at the sesquiannual International Conference on Domain Decomposition Methods.
