Randomized Greedy Algorithms for Neural Network Optimization in Solving Partial Differential Equations

Overview

Neural networks have demonstrated remarkable approximation capabilities for solving partial differential equations (PDEs). However, their practical effectiveness remains limited by optimization challenges, even in the case of shallow neural networks. This leads to a significant gap between theoretical approximation rates and practical convergence orders. In this talk, we address the key optimization challenges in using ReLUᵏ shallow neural networks for numerical PDEs by developing the randomized orthogonal greedy algorithm (ROGA). We prove that the orthogonal greedy algorithm achieves the optimal convergence rate of ReLUᵏ shallow neural networks for solving PDEs that admit strongly convex and smooth energy functionals, thereby extending the applicability of OGA to a wide range of variational problems. We further tackle practical challenges in implementing OGA by proposing a randomized variant - the randomized orthogonal greedy algorithm. Comprehensive numerical experiments on a range of linear and nonlinear PDEs confirm its practical effectiveness and optimal convergence behavior. These results demonstrate that our proposed algorithm substantially narrows the gap between theory and practice, validating ROGA as a promising new tool for numerical PDEs.