UCLA

The rapid ascent of Large Language Models (LLMs) and Vision-Language Models (VLMs) has revolutionized semantic understanding, yet these systems remain “spectators”—observing the world through 2D projections without true physical grounding. This dissertation, Building World Models with Spatial Intelligence, addresses this critical gap by establishing a blueprint for AI systems that possess Spatial Intelligence: the ability to perceive, imagine, interact with, and reason about the 3D world. We posit that achieving this requires a symbiotic architecture that combines the vast semantic knowledge of 2D foundation models with the geometric precision of explicit 3D/4D representations, specifically Gaussian Splatting.The research is presented through four core contributions corresponding to the essential faculties of a World Model. First, we introduce Feature 3DGS, a framework for Perception that transforms 2D observations into a semantic, editable, and promotable 3D scene representation. By distilling high-dimensional features from 2D foundation models (e.g., SAM, CLIP) into explicit 3D Gaussian fields, this approach enables spatially grounded multimodal world modeling. Second, to address scenarios where direct 2D observations are absent, we present DreamScene360, an engine for Imagination that generates immersive, globally consistent 3D environments from natural language descriptions by lifting synthesized 360 degree panoramas into explicit 3D Gaussians. Third, recognizing that the true physical world is inherently dynamic rather than static, we advance towards a more generalized world modeling framework for Interaction. We introduce Feature4X, which transforms monocular videos into a dynamic, multimodal, and interactive 4D world via a unified spatiotemporal latent feature field. This explicit representation equips VLM-driven agents with the spatial and temporal grounding needed to perform complex tasks such as 4D scene editing and spatiotemporal Visual Question Answering (VQA). However, whether frontier AI is truly capable of solving complex, high-level cognitive tasks within the real, dynamic physical world remains an open question. To investigate this, we finally propose VLM4D, a rigorous benchmark for Reasoning that exposes the fundamental limitations of current VLMs in understanding dynamic physical interactions, driving future advancements in physical AI. Together, these works demonstrate a path toward building World Models that do not just recognize patterns in pixels, but understand the geometry and dynamics of the physical world, laying the groundwork for the next generation of Spatial Computing and Embodied AI.