UC Berkeley

Modern diffusion models are famous for their sample quality. However, viewing these models only as samplers risks overlooking their rich internal representations. Much of the progress in computer vision was facilitated by transferring features from image classifiers, yet the features of diffusion models, which have been scaled to far larger datasets, remain unexamined. If their representations were better understood, these generative models could be repurposed for new capabilities, such as discriminative tasks or controllable generation. In this thesis, we show that diffusion models indeed contain rich representations that can be reused in new tasks and domains. We start in Chapter 2 by designing a general-purpose feature extractor for diffusion models, enabling an image generator to achieve state-of-the-art performance on the discriminative task of semantic correspondence. Then, in Chapter 3, we use this same extractor for controllable generation, allowing diffusion models to condition on new visual inputs with orders of magnitude less data compared to prior methods. In Chapters 4 to 5 we extend these ideas to new domains beyond visual generation. Finally, this thesis culminates in Chapter 6, where we look at the new domain of meta-modeling and train a diffusion model on representations themselves, affording more faithful steering of model behavior and more effective probing of model knowledge without the strong structural assumptions of prior work.