UC Berkeley

Robots have demonstrated extraordinary capabilities over the past few decades, from performing surgeries to exploring space. Despite this progress, robots are not yet commonplace in our everyday lives; instead, they are confined to executing tasks where the humans behind them can account for everything the robot will encounter. The challenge in deploying robots that can be autonomous stems from the diversity and unpredictability of the physical world. Humans constantly encounter new situations, and we face this by quickly adapting to them as they arise. Could we also enable robots to face our unpredictable world by allowing them to learn online, from their real-world experiences? Reinforcement learning provides a framework for learning through interaction with and feedback from an environment. In this thesis, we study challenges in applying reinforcement learning to physical robot systems that are not confined to lab settings, and in doing so, propose algorithmic solutions, provide empirical analysis, and build practical training systems that demonstrate their efficacy. We begin by building a legged locomotion learning system that incorporates simulated pre-training, autonomous failure recovery, multi-task training, onboard sensors, and sample-efficient RL, and demonstrate that a small amount of real-world practice can enable effective fine-tuning in unstructured settings. We then show how to enable efficient learning with more complex reward functions, derived from supervision that is general and available in the real world: human preferences. We further simplify the assumptions and study learning directly in the real world, demonstrating a system capable of enabling a quadruped to learn to walk in various natural environments, purely from real-world experience. Lastly, we look towards learning more complex tasks by leveraging priors. First, we extend the efficient learning framework to effectively ingest offline, mixed-quality data. In discussing how this is practical for robotics applications, we show that this method enables flexible agile quadrupedal locomotion such as running jumps and bipedal walking. Finally, we explore how foundation models can adapt language-conditioned manipulation to new situations in the real world.