UC Merced

Cyber-physical systems rely on many control and decision-making algorithms. However, the majority of cyber-physical systems today are still operated by simple rule-based and feedback controls such as on-off control or proportional-integral derivative (PID) control. These prescriptive and reactive control strategies do not take into consideration predictive information on disturbances, such as weather changes, making their performance sub-optimal. Optimal control strategies such as MPC address these drawbacks by iteratively optimizing an objective function over a receding time horizon. However, despite many successful cyber-physical systems applications of MPC, its wide-spread adoption has been limited by the need of accurate models. This is especially challenging because cyber-physical systems are heterogeneous. Thus, custom models are required for cyber-physical systems, limiting the scalability of MPC. To overcome the above challenges, we investigate deep reinforcement learning (DRL) for solving challenging cyber-physical system optimization problems. Deep reinforcement learning is a data-driven control method and learns an optimal control policy directly from data, without using any pre-programmed control rules or explicit assumptions about the cyber-physical systems. DRL also allows us to use domain knowledge to train a control agent (a neural network) without labeled data. The generalization ability of the neural network enables the control agent to handle the dynamically-varying factors (e.g., weather changing). We use this approach to build a series of cyber-physical systems for important applications including building control, irrigation control, and virtual machine rescheduling. In the first building control application, we take a holistic approach to deal with the trade-offs between energy use and comfort in commercial buildings. Further, we investigate the common data-efficiency problem when leveraging DRL to cyber-physical systems and propose two data-efficient model-based RL methods for multi-zone building control. In the second irrigation control application, we design a DRL-based irrigation system that generates optimal irrigation control commands according to current soil water content, current weather data, and forecastedweather information. In the third virtual machine rescheduling application, we design a two-stage DRL agent framework to balance rescheduling performance and solution speed. In future work, we will use the insights from these systems to identify common problems and develop new data-efficient reinforcement learning techniques for cyber-physical systems.