Skip to main content
eScholarship
Open Access Publications from the University of California

UC Berkeley

UC Berkeley Electronic Theses and Dissertations bannerUC Berkeley

General Robot Manipulation with Multi-Modal Vision-Language-Action Models

Abstract

Robot learning holds tremendous potential to bridge the gap between artificial intelligence and the physical world. However, achieving human-level dexterous manipulation across a wide variety of robots, environments, and tasks remains a great challenge. Often, this is blamed on a dearth of robot training data, but that situation is quickly changing. Spurred by the tremendous successes of digital AI agents, robotics companies and academic labs alike have increasingly expanded the scale of robot data collection. In this dissertation, we explore how to best use that data to train and deploy general-purpose robot manipulation policies; much like how the large-scale pre-training of language models sparked a revolution in cognitive labor, we aim to do the same for physical labor. We begin by proposing a full training system for a generalist robot policy, including a model architecture that is designed for the particular needs of generalist robotics. We train a series of models of different sizes on a highly heterogeneous amalgamation of datasets, and study the behavior of large-scale robot policy pre-training. Then, we address the challenge of real-time control with big, compute-intensive robot policies; we devise an algorithm to execute such policies asynchronously and smoothly, showing that our algorithm greatly improves performance in the presence of inference latency. Lastly, we propose a hierarchical system that can directly leverage Internet-scale non-robot video data to improve its semantic and visual reasoning, and find that this system exhibits surprising generalization capabilities even with relatively little robot data.