Proceedings of the Annual Meeting of the Cognitive Science Society

Agents trained using deep reinforcement learning (DRL) are capable of meeting or exceeding human-levels of performance in multi-agent tasks. However, the behaviors exhibited by these agents are not guaranteed to be human-like or human-compatible. This poses a problem if the goal is to design agents capable of collaborating with humans in cooperative or team-based tasks. Previous approaches to encourage the development of human-compatible agents have relied on pre-recorded human data during training. However, such data is not available for the majority of everyday tasks. Importantly, research on human perceptual-motor behavior has found that task-directed behavior is often low-dimensional and can be decomposed into a defined set of dynamical perceptual-motor primitives (DPMPs). Accordingly, we propose a hierarchical approach to simplify DRL training by defining the action dynamics of agents using DPMPs at the lower level, while using DRL to train the decision-making dynamics of agents at the higher level. We evaluate our approach using a multi-agent shepherding task used to study human and multi-agent coordination. Our hierarchical, DRL-DPMP approach resulted in agents which trained faster than vanilla, black-box DRL agents. Further, the hierarchical agents reached higher levels of performance not only when interacting with each other during self-play, but also when completing the task alongside agents embodying models of novice and expert human behavior. Finally, the hierarchical DRL-DPMP agents developed decision-making policies that outperformed heuristic-based agents used in previous research in human-agent coordination.