Proceedings of the Annual Meeting of the Cognitive Science Society

People use physical knowledge in remarkably flexible ways, yet most prior work models one aspect of physical reasoning at a time. Here we study the flexibility of people's physical reasoning, building on theories that language guides the construction of ad-hoc mental models grounded in capabilities for forming and manipulating representations of the world. We instantiate this theory in the Physical Reasoning via Interpretable Synthesis of Models (PRISM) framework, which uses structured reasoning with language models to generate task-specific programs in response to a question, relying on primitive functions for perceiving, editing, and simulating physical scenes. We compare PRISM against people's judgments on four distinct physics scenarios inspired by prior research, finding that it explains people's behavior as well as custom-written models from prior research while outperforming vision-language model baselines. PRISM thus provides a cognitively plausible framework for understanding how we assemble physical concepts to flexibly reason about the world.