- Main
Building Generally Intelligent Robots
- Walke, Homer
- Advisor(s): Levine, Sergey
Abstract
Large-scale imitation learning is a promising approach to building a generally intelligent robot. In principle, by training on a large and diverse dataset of human demonstrations we should obtain a model that generalizes broadly across tasks and environments, similar to foundation models in language and vision. However, training a foundation model for robotics presents unique challenges. Collecting robot data is expensive, so robot pre-training datasets are small and often do not provide enough coverage for meaningful generalization across the vast amount of complexity in the real world. Additionally, the model architecture must balance competing demands. The architecture needs to be flexible, expressive, and high-capacity to fit heterogenous data collected on robots with widely varying sensor and actuators. At the same time, the model needs to be efficient enough to support real-time control, even when conditioned on a history of high-dimensional image observations. To address these challenges, we first present techniques for collecting robot data with sufficient coverage of tasks and environments as well as a dataset for generalization research. Then, we propose a robot foundation model architecture and demonstrate its broad generalization when trained on a dataset of 800K demonstrations on single-arm manipulators. Next, we extend the architecture to enable joint training on data from multiples types of robots, including bimanual arms, wheeled robots, and quadrupeds, and we find promising signs of positive transfer across embodiments. Finally, we tackle the challenge of adding memory to a robot foundation model. We design a memory system that enables tasks requiring anywhere from a few seconds to many minutes of observation history.