Optimization Theory for ReLU Neural Networks Trained with Normalization Layers
Skip to main content
eScholarship
Open Access Publications from the University of California

Optimization Theory for ReLU Neural Networks Trained with Normalization Layers

  • Author(s): Dukler, Yonatan
  • Gu, Quanquan
  • Montúfar, Guido
  • et al.

Published Web Location

https://arxiv.org/abs/2006.06878
No data is associated with this publication.
Abstract

The success of deep neural networks is in part due to the use of normalization layers. Normalization layers like Batch Normalization, Layer Normalization and Weight Normalization are ubiquitous in practice, as they improve generalization performance and speed up training significantly. Nonetheless, the vast majority of current deep learning theory and non-convex optimization literature focuses on the un-normalized setting, where the functions under consideration do not exhibit the properties of commonly normalized neural networks. In this paper, we bridge this gap by giving the first global convergence result for two-layer neural networks with ReLU activations trained with a normalization layer, namely Weight Normalization. Our analysis shows how the introduction of normalization layers changes the optimization landscape and can enable faster convergence as compared with un-normalized neural networks.

Many UC-authored scholarly publications are freely available on this site because of the UC's open access policies. Let us know how this access is important for you.

Item not freely available? Link broken?
Report a problem accessing this item