UC Berkeley

This thesis explores the theoretical foundations of deep learning, focusing on generalization, optimization, and scaling properties of neural networks. We begin by examining the generalization properties of neural networks trained in the Neural Tangent Kernel (NTK) regime as they evolve over time in a high-dimensional asymptotic setting. This regime simplifies the training dynamics, allowing us to characterize the learning process and identify key phenomena. We then address fundamental questions about the trade-offs between overparameterization and overfitting that were prompted by empirical observations in modern deep learning. For the case of linear models, we show that fitting noise while generalizing well requires excess overparameterization, a principle which we believe holds more generally. To advance our understanding of optimization dynamics beyond the NTK regime, where the dynamics are simple and there is no feature learning, we analyze stochastic gradient training of a single-neuron autoencoder. Our study reveals the critical role that the batch size plays in determining the learned solution. To facilitate our analysis, we introduce powerful tools from the theory of non-homogeneous random walks. We then turn to questions concerning the scaling behavior of neural networks. Drawing on insights from the theory of infinite-width neural networks, we provide practically relevant principles and methods to reduce the computational burden of finetuning. For the highly popular LoRA finetuning approach, we identify the suboptimality of the standard choice of learning rates and propose a fix which results in a roughly 2x speed up on several LLM finetuning tasks. We also develop an approach for performing hyperparameter transfer of the optimal learning rate for model finetuning. This approach allows one to construct a smaller version of a pretrained network such that the optimal learning rate of the smaller network approximates the optimal learning rate of the original network, reducing the computational burden by allowing the hyperparameter search to be done on a smaller network.