Young infants reason about the goals, preferences, and actions of others. State of the art computational models, however, still fail in such reasoning. The Baby Intuitions Benchmark (BIB) was designed to test agency reasoning in AI using an infant behavioral paradigm. While BIB’s presentation of simple animations makes it particularly suitable for testing AI, such vignettes have yet to be validated with infants. In this pilot, 11-month-old infants watched two sets of animations from BIB, one on agents’ consistent preferences and the other on agents’ efficient actions. Infants looked longer towards violations in agents’ behavior in both the preference (N = 24, β = 3.24 p = .040) and efficiency task (N = 24, β = 4.50 p = .016). These preliminary results suggest that infants’ agency reasoning is abstract enough to be elicited by simple animations and validate BIB as a test of agency reasoning for humans and AIs.

Virtual Reality (VR) presents a novel tool to study the mind by generating dynamic, immersive environments. However, the adoption of VR in cognitive science has been slow. In order to use VR as a method, we need to better understand how humans think and act in this novel, virtual environment. Here, we draw on research in intuitive physics to investigate whether people’s reasoning about objects and physical scenes in VR is similar to traditional experimental settings. In Exp.1, participants made judgments on a physical scene prediction task similar to those collected using conventional methods (Battaglia et al., 2013). In Exp.2, participants preferentially explored objects that violated core principles of physics and engaged in hypothesis-testing behaviors, similar to Stahl & Feigenson (2015). These findings suggest that people bring real-world intuitions about physics into the virtual world, and that VR can be an exciting method to study human behavior.

As AI systems like language models are increasingly integrated into decision-making processes affecting people's lives, it's critical to ensure that these systems have sound moral reasoning. To test whether they do, we need to develop systematic evaluations. We provide a framework that uses a language model to translate causal graphs that capture key aspects of moral dilemmas into prompt templates. With this framework, we procedurally generated a large and diverse set of moral dilemmas---the OffTheRails benchmark---consisting of 50 scenarios and 400 unique test items. We collected moral permissibility and intention judgments from human participants for a subset of our items and compared these judgments to those from two language models (GPT-4 and Claude-2) across eight conditions. We find that moral dilemmas in which the harm is a necessary means (as compared to a side effect) resulted in lower permissibility and higher intention ratings for both participants and language models. The same pattern was observed for evitable versus inevitable harmful outcomes. However, there was no clear effect of whether the harm resulted from an agent's action versus from having omitted to act. We discuss limitations of our prompt generation pipeline and opportunities for improving scenarios to increase the strength of experimental effects.