Adaptive Gradient Methods with Energy for Optimization Problems | Applied and Computational Math Colloquium

Abstract

We propose AEGD, a new algorithm for gradient-based optimization of stochastic objective functions, based on adaptive updates of quadratic energy. The method is shown to be unconditionally energy stable, irrespective of the step size. In addition, AEGD enjoys tight convergence rates, yet allows a large step size. The method is straightforward to implement and requires little tuning of hyper-parameters. Experimental results demonstrate that AEGD works well for various optimization problems: it is robust with respect to initial data, capable of making rapid initial progress, shows comparable and most times better generalization performance than SGD with momentum for deep neural networks.

Speaker

Hailiang Liu is a Professor of Mathematics and Computer Science at the Iowa State University (ISU). He earned his Bachelor degree from Henan Normal University, Master degree from Tsinghua University, and Ph.D. degree from the Chinese Academy of Sciences, all in Mathematics. His research interests include analysis of partial differential equations, the development of high order numerical algorithms for solving these PDE problems, with diverse applications. He is the author of over 160 peer reviewed papers, and the recipient of many awards and honors, including the Alexander von Humboldt-Research Fellow, and the inaugural Holl Chair in Applied Mathematics at Iowa State University.