UCLA

This thesis explores how interpretable semantic structures can be identified within, or incorporated into, the internal representations of deep neural networks. For the class of large-scale autoregressive models, we define a notion of `meaning' representation based on the distribution over future continuations of a given input. The resulting space of meanings is shown to exhibit geometric and compositional properties which can be shown to align with human annotations. Leveraging text-conditioned image diffusion models, meaning representations can be grounded in visual imagery by adopting an alternate definition based on the distributions over conjured imageries. Semantic similarity scores can be obtained as the Jensen-Shannon divergence between the reverse-time diffusion stochastic differential equations induced by different textual expressions. Beyond the discovery of emergent semantic structures, this thesis shows how compositional structures designed to be interpretable can be achieved through deliberate architectural choices. In addition to their interpretability, these structures offer significant practical advantages, enabling state-of-the-art performance in tasks such as retrieval-augmented generation and model composition.