UC Santa Cruz

Artificial intelligence has produced a growing collection of powerful pretrained foundation models, including large language models, multimodal large language models, image and video diffusion models, image-to-3D generators, and depth estimators. The embodied AI community has, in parallel, begun training vision-language-action models on large-scale robot data. Yet generalist embodied competence spans more than any single pretrained model is designed to cover: grounding instructions in physical action, communicating through coherent multimodal content, and constructing 3D worlds. This dissertation studies how to build embodied systems by composing pretrained foundation models, rather than training new large-scale embodied systems from scratch. The central claim is that, for each capability needed by an embodied agent, a strong pretrained model already exists, and the right interfaces between such models can turn independently trained components into capable embodied agents. The dissertation develops four such interfaces: symbolic, learnable token, spatial, and iterative. Part I examines how pretrained perception and language models can be composed into embodied reasoning systems. I introduce VLMbench, a compositional benchmark for vision-and-language manipulation that exposes where current vision-language models fail to ground compositional instructions in physically constrained motion. I then present JARVIS, a neuro-symbolic framework that treats pretrained vision and language models as sensors and binds their outputs through symbolic task-level and action-level reasoning, enabling agents to complete long-horizon household tasks from free-form dialog. Part II studies the interface between pretrained multimodal language models and pretrained image generators. I present MiniGPT-5, which connects them through generative vokens, a small set of learnable tokens that project language hidden states directly into the diffusion conditioning space. Both backbones remain frozen, and only a lightweight interface is trained, enabling description-free interleaved text-and-image generation. Part III asks the same composition question for 3D scene construction without any scene-level training data. SceneFuse-3D composes a pretrained image-to-3D object generator into a scene generator through region-wise decomposition and spatial-aware completion, using pretrained depth and detection priors as spatial anchors. EvoScene then couples pretrained 3D and video generators in a self-evolving 2D--3D loop, in which depth-conditioned video synthesis provides novel views that progressively refine the underlying mesh. Together, these contributions outline a methodology for embodied AI grounded in the composition of pretrained foundation models through deliberately designed interfaces: symbolic glue in JARVIS, learnable token interfaces in MiniGPT-5, spatial priors in SceneFuse-3D, and iterative 2D--3D feedback in EvoScene. The dissertation argues that the next generation of embodied agents can be built by reusing, rather than rebuilding, the foundation models the field is producing.