Proceedings of the Annual Meeting of the Cognitive Science Society

Accurate behavior prediction is essential for safe human-AI collaboration. However, existing models are often data-hungry or brittle, assuming unrealistic rationality or requiring immense computation to adapt. Our insight is that many everyday social interactions follow predictable "scripts''---efficient routines like "wait for the green light, then go'' that minimize cognitive load for actors and observers. We propose modeling these routines as behavioral programs in computer code, rather than policies conditioned on beliefs and desires. We introduce ROTE, an algorithm leveraging LLMs to synthesize a hypothesis space of programs and probabilistic inference to reason over uncertainty. In gridworld tasks and a large-scale embodied household simulator, ROTE predicts human and AI behaviors from sparse observations, outperforming baselines—including behavior cloning—by up to 50% in accuracy and generalization. By treating action understanding as program synthesis, ROTE enables AI to efficiently and effectively predict human behavior in the real world.