UC San Diego

Visual reinforcement learning holds great promise for training robot controllers end-to-end from image inputs, offering capabilities that classical controllers fail to attain. However, scaling visual reinforcement learning to diverse backgrounds, setups and tasks remains nontrivial. These limitations are not due to a lack of data alone, but due to current visual reinforcement learning methods being too brittle and inefficient for widespread adoption. To further democratize visual robot learning, we tackle three core limitations of visual reinforcement learning for continuous control: poor visual generalization, poor visual robustness, and poor visual training efficiency.We first address poor visual generalization by introducing SADA (Stabilized Actor-Critic under Data Augmentation), a training recipe that augments the agent’s replay buffer with large image datasets and visual transformations, enabling invariance to visual perturbations without destabilizing training. Second, we tackle visual robustness under multi-camera and multi-sensor setups with MAD (Merge and Disentangle), a lightweight method that maximizes training efficiency with multiple inputs while remaining robust to sensor failures at deployment. Third, we improve training efficiency with Squint, a novel off-policy visual reinforcement learning algorithm that achieves faster wall-time training than prior methods through parallel simulation, resolution squinting, a tuned updated-to-data ratio, a distributional critic, layer normalization, and an optimized implementation. Squint succeeds in training deployable visual reinforcement learning policies to our real SO-101 robot setup in minutes. Together, these three contributions lower the barrier to entry for visual robot learning, making visual reinforcement learning agents more practical and accessible for the broader research community.