UCLA

Despite the impressive performance of modern deep generative models such as Transformer-based large language models (LLMs) and diffusion models, their intractability imposes significant limitations on how they can be queried: for instance, marginalization is generally infeasible. Motivated by such challenges, I study the theory, algorithms, and applications of Probabilistic Circuits (PCs), a class of tractable generative models that support efficient and exact inference for various probabilistic queries, offering a path toward generative modeling with improved controllability. In Part 1, I demonstrate how PCs can help improve the controllability of LLMs by guiding their outputs to satisfy logical constraints. Part 2 presents a systematic study on the algorithms of circuit multiplication, introducing techniques for efficiently multiplying circuits with different structures. Recognizing the modeling limitations introduced by the structural constraints on PCs, I explore the theoretical limits of PC expressiveness in Part 3, where I establish exponential lower bounds for existing families of PCs and propose Probabilistic Generating Circuits (PGCs) as their generalization. PGCs are not only more expressive but also retain tractable marginalization, providing a theoretical foundation for scaling PCs beyond current limits.