Proceedings of the Annual Meeting of the Cognitive Science Society

Human infants have remarkable abilities to reason about the underlying and invisible causes that drive others' actions. These abilities are at the core of human social cognition throughout life. Artificial Intelligence (AI) systems continue to fall short in achieving this same commonsense social knowledge. Recent benchmarks focusing on social cognition and theory of mind have begun to address the gap between human and machine social intelligence, but they do not fully consider the social reasoning required to understand scenarios with multiple interacting agents. Building on such benchmarks, we present eight new tasks focusing on different early social competencies, as informed by behavioral experiments with infants. We use a self-supervised Transformer model as a baseline test of our new tasks, and in addition, we evaluate this model on a previous social-cognitive benchmark. While our model shows improved performance on the previous benchmark compared with other data-driven models, it performs sub-optimally on our new tasks, revealing the challenge of learning complex social interactions through visual data alone.