Scaling Robot Learning with World Models and Real2Sim
Skip to main content
eScholarship
Open Access Publications from the University of California

UC Berkeley

UC Berkeley Electronic Theses and Dissertations bannerUC Berkeley

Scaling Robot Learning with World Models and Real2Sim

Abstract

This dissertation develops methods that enable embodied agents — legged robots and other physical systems — to acquire complex behaviors by exploiting structured priors over motion, dynamics, and visual appearance. Across five chapters, the central thesis is that learning performant and natural-looking behaviors at scale requires moving beyond brittle hand-engineered reward functions and ad hoc simulation, and instead grounding learning in priors extracted from data — be it short motion-capture clips, trial-and-error rollouts in the real world, unlabeled videos, or photorealistic reconstructions of real environments. The first chapter introduces Adversarial Motion Priors for quadrupedal hardware, showing that complex hand-designed style rewards can be replaced by a style reward learned adversarially from a few seconds of dog motion-capture data. The resulting policies transfer to a real Unitree A1 robot and produce naturalistic gait transitions and lower cost of transport. The second chapter presents GaussGym, a real-to-sim framework that integrates 3D Gaussian Splatting as a drop-in renderer inside vectorized physics simulators. With over 100,000 simulator steps per second on a single consumer GPU and thousands of photorealistic scenes drawn from smartphone scans, large-scale datasets, and generative video models, GaussGym makes it possible to train RGB locomotion and navigation policies that transfer zero-shot to real hardware and reason about semantic affordances invisible to depth-only sensors. The third chapter presents DayDreamer, which applies world models to physical robot learning. By learning a Recurrent State-Space Model directly from a single robot's own experience, the same algorithm with the same hyperparameters teaches a quadruped to roll over, stand up, and walk in one hour, and learns visuomotor pick-and-place and navigation policies on three other robots — all without simulators or demonstrations. The fourth chapter introduces Video Prediction Rewards (VIPER), which leverages pre-trained autoregressive video models to score agent behavior. Used as a dense reward signal for reinforcement learning, video model log-likelihoods enable expert-level control across 28 DeepMind Control, Atari, and RLBench tasks from pixels, and generalize across embodiments without task-specific demonstrations. The fifth chapter pivots to the structure of the actor itself. Q-Score Matching (QSM) treats the score of a diffusion-model policy as the optimization target and aligns it with the action gradient of the learned Q-function. The resulting off-policy algorithm trains expressive, multi-modal policies sample-efficiently across continuous-control benchmarks, matching or exceeding SAC and TD3 while sidestepping the cost of differentiating through full diffusion rollouts. Together, these contributions chart a progression from data-driven priors over individual motions, to photorealistic simulators that close the visual sim-to-real gap, to world models over single-robot experience, to large-scale priors learned from video, and finally to expressive policy classes trained directly from rewards.