UC Berkeley

Generative artificial intelligence has emerged as one of the most transformative developments in science and technology. Modern AI models—such as large language models (LLMs), diffusion models, and multimodal systems—have achieved remarkable success across language, vision, and cross-modal tasks. Despite this empirical progress, rigorous theoretical foundations for these models remain limited, posing challenges for understanding their capabilities and guiding the development of more powerful and trustworthy AI. This dissertation contributes to the statistical foundations and algorithms for modern AI from three complementary angles. First, motivated by the concept of sufficient statistics, we develop a statistical theory of contrastive pre-training and multimodal generative AI. We introduce approximate sufficient statistics as a measure of representation quality and show that near-minimizers of the contrastive loss are approximately sufficient, making them adaptable to diverse downstream tasks such as zero-shot classification, conditional diffusion models, and vision-language models. We further propose the Joint Generative Hierarchical Model (JGHM) for the joint image-text distribution and prove that Transformers can efficiently approximate belief propagation on JGHMs, yielding end-to-end sample complexity guarantees for multimodal contrastive learning. Second, we study the theory of neural scaling laws in a high-dimensional linear regression setting. Under a power-law spectral assumption on the data covariance, we derive matching upper and lower bounds on the test error of one-pass SGD applied to sketched linear models. Our analysis shows that the variance error, which classical theory predicts to grow linearly with model size, is of smaller order due to the implicit regularization of SGD, thus partially explaining the inconsistency between standard learning theory and empirically observed scaling laws. Third, we address the problem of LLM unlearning: removing specific knowledge from a pretrained model without retraining from scratch. We propose the Negative Preference Optimization (NPO) loss, inspired by preference optimization. NPO mitigates the catastrophic collapse that results from gradient ascent by adaptively downweighting already-unlearned samples. We provide theoretical guarantees showing that NPO diverges logarithmically slower than gradient ascent, and demonstrate strong empirical performance on the TOFU unlearning benchmark.