- Main
Scaling Robot Learning with World Models and Real2Sim
- Escontrela, Alejandro
- Advisor(s): Abbeel, Pieter
Abstract
This dissertation develops methods that enable embodied agents — legged robots and other physical systems — to acquire complex behaviors by exploiting structured priors over motion, dynamics, and visual appearance. Across five chapters, the central thesis is that learning performant and natural-looking behaviors at scale requires moving beyond brittle hand-engineered reward functions and ad hoc simulation, and instead grounding learning in priors extracted from data — be it short motion-capture clips, trial-and-error rollouts in the real world, unlabeled videos, or photorealistic reconstructions of real environments. The first chapter introduces Adversarial Motion Priors for quadrupedal hardware, showing that complex hand-designed style rewards can be replaced by a style reward learned adversarially from a few seconds of dog motion-capture data. The resulting policies transfer to a real Unitree A1 robot and produce naturalistic gait transitions and lower cost of transport. The second chapter presents GaussGym, a real-to-sim framework that integrates 3D Gaussian Splatting as a drop-in renderer inside vectorized physics simulators. With over 100,000 simulator steps per second on a single consumer GPU and thousands of photorealistic scenes drawn from smartphone scans, large-scale datasets, and generative video models, GaussGym makes it possible to train RGB locomotion and navigation policies that transfer zero-shot to real hardware and reason about semantic affordances invisible to depth-only sensors. The third chapter presents DayDreamer, which applies world models to physical robot learning. By learning a Recurrent State-Space Model directly from a single robot's own experience, the same algorithm with the same hyperparameters teaches a quadruped to roll over, stand up, and walk in one hour, and learns visuomotor pick-and-place and navigation policies on three other robots — all without simulators or demonstrations. The fourth chapter introduces Video Prediction Rewards (VIPER), which leverages pre-trained autoregressive video models to score agent behavior. Used as a dense reward signal for reinforcement learning, video model log-likelihoods enable expert-level control across 28 DeepMind Control, Atari, and RLBench tasks from pixels, and generalize across embodiments without task-specific demonstrations. The fifth chapter pivots to the structure of the actor itself. Q-Score Matching (QSM) treats the score of a diffusion-model policy as the optimization target and aligns it with the action gradient of the learned Q-function. The resulting off-policy algorithm trains expressive, multi-modal policies sample-efficiently across continuous-control benchmarks, matching or exceeding SAC and TD3 while sidestepping the cost of differentiating through full diffusion rollouts. Together, these contributions chart a progression from data-driven priors over individual motions, to photorealistic simulators that close the visual sim-to-real gap, to world models over single-robot experience, to large-scale priors learned from video, and finally to expressive policy classes trained directly from rewards.