Skip to main content
eScholarship
Open Access Publications from the University of California

UCLA

UCLA Electronic Theses and Dissertations bannerUCLA

Towards Faster and More Accurate Neural ODEs

Abstract

Neural Ordinary Differential Equations (NODEs) [Che+18] have improved accuracy and memory efficiency over general deep neural networks but suffer from the vanishing gradient problem and the large number of function evaluations (NFEs) problem. In chapter 3, we proposed Heavy Ball NODEs (HBNODEs), leveraging the continuous limit of classical momentum-accelerated gradient descent, to improve NODEs training and inference. We show that HBNODEs has two major advantages: (1) The adjoint of HBNODEs also satisfies Heavy Ball ODEs, accelerating both forward and backward solvers; (2) the spectrum of HBNODEs is well-structured so that HBNODEs is capable of learning long-term dependencies from long time series. In chapter 4, we proposed Graph Neural Diffusion with a Source Term (GRAND++), a class of NODEs on graphs, for deep learning on graphs with a limited number of labels. We study the limiting behavior of GRAND++ and show that it does not converge to a constant even when the depth goes to infinity. We provide experiments to show that GRAND++ can provide accurate classification even when the number of labels is limited. In chapter 5, we study how proximal implicit solvers can improve NODEs training in some scenarios. We show that for stiff NODEs, proximal implicit solvers have smaller NFEs than commonly used explicit solvers, and thus speed up training.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View