Nobody knows how deep learning works. We have heuristics and hacks that give us some intuition, but compared to other engineered systems --- skyscrapers, airplanes, computer chips --- we have precious little first-principles understanding of neural networks.
I stand with those who believe that principled understanding is possible. This thesis presents five years of work towards developing mathematically-grounded theory describing the learning behavior of neural networks. In Part I, we study the learning behavior of \textit{kernel methods}, a class of learning rule equivalent to a large-width limit of neural networks, and obtain simple closed-form theories describing their training dynamics and test-time performance in various settings. We also derive an inverse mapping by which certain kernel methods can be transformed back into trainable neural networks. In Part II, we study the learning behavior of neural networks in the more realistic feature learning regime. We first give a simple derivation of the feature-learning large-width limit using the perspective of the spectral norm. We conclude with a scaling analysis of the feature learning strength which suggests the existence of an ``ultra-rich'' limit opposite the kernel limit. My hope and belief is that these efforts constitute small but useful steps in the broader research endeavor of putting scientific backing to the dark art of neural network training.
A unifying theme throughout this thesis is the emergence of an understandable discretized or quantized picture from the messy continuum of deep network training when an appropriate limit is taken or a suitable lens applied. This is realized in different ways in different chapters: when studying kernel methods, we often obtain an understandable theory phrased in terms of kernel eigenmodes, and when studying feature learning, we find discretization arising in the singular spectrum of weight matrices and in stepwise loss trajectories in the ultra-rich regime.
Cookie SettingseScholarship uses cookies to ensure you have the best experience on our website. You can manage which cookies you want us to use.Our Privacy Statement includes more details on the cookies we use and how we protect your privacy.