UC San Diego

Learning-based control offers a promising alternative to classical control algorithms due to its flexibility and ability to learn directly from data. Despite success in domains like robotics and autonomous systems, these methods often ignore existing control theory, fail to scale in high dimensional state-spaces, and are unable to generalize to out-of-distribution data. In contrast, traditional control algorithms thrive in real-world engineering applications due to their simplicity and rich theoretical guarantees. For these reasons, linear controllers such as PID and LQR are typically favored over learning-based methods in engineering applications, even when applied to inherently nonlinear systems that must be approximated as locally linear. This work aims to bridge the gap between traditional and learning-based control by incorporating imperfect expert controllers into the reinforcement learning process. Specifically, we use expert control laws to regularize the advantage estimate of policy gradient algorithms to guide exploration. The policy initially learns to align with the expert’s behavior and attempts to improve upon it by learning the additional nonlinear control needed to outperform the expert in complex settings. We test our method on applications with high dimensional state spaces including fully and partially observable 6DoF quadrotor environments and control of common 1D hyperbolic and parabolic partial differential equations. Results show increased sample efficiency compared to baseline RL algorithms and, in some cases, improved performance over the imperfect expert.