UC Berkeley

In robotic manipulation, action chunking, a technique that predicts and executes a sequence of actions rather than one at a time, is a powerful tool for capturing noisy and non-Markovian behavior in human data or prior experience. While action chunking policies have seen lots of successes in imitation learning, they are typically trained with supervised learning on human demonstration data that are often costly to collect. Reinforcement learning (RL) offers a promising alternative by enabling robots to autonomously collect data and continuously self-improve from a well-specified reward function, but existing RL methods rely on simple policy classes that often struggle to capture the multi-modality in prior data. In this dissertation, we discuss algorithmic and theoretical foundations for scalable RL with action chunking policies. We start by presenting practical algorithms for optimizing flow-matching policies in Q-learning. Then, we establish a theoretical framework where we identify the key open-loop consistency condition under which the learned action chunking policies are guaranteed to be near-optimal as well as the bounded optimality variability condition under which we can mitigate the open-loop bias via closed-loop execution. Inspired by these analyses, we introduce a practical algorithm that demonstrates the scalability of action chunking RL to longer-horizon tasks. Altogether, these contributions provide a foundation for scalable RL with action chunking policies and bring RL closer to real-world deployment in robotics.