Humans possess rich knowledge of the structure of the world, including co-occurrences among entities, and co-variation among their discrete and continuous features. But how people learn, infer and predict this structure is not wellunderstood. Here we explore everyday scene understanding as a case study of people’s structural knowledge and reasoning.We introduce a probabilistic model over scene graphs that can learn the relational structure of objects and their arrangementsand support inference and generation. Our model was able to learn the underlying structure of real-world scenes, and use it forinference and compression. In two human psychophysical experiments we found that a corresponding computational cognitivemodel was able to explain how people learn novel scene distributions and use it for classification and construction. Our workrepresents the first computational theory of human scene understanding that can account for people’s rich capacity for learningand reasoning about structure.

Physical simulation enables people to make intuitive predictions about physical scenes and interact flexibly with the objectsaround them, from a stack of books balanced on a ledge to the turrets and moats of a sandcastle. We hypothesize that whenthe number of possible objects makes simulation intractable, people use chunked abstractions that reduce the number ofobjects they need to simulate while also minimizing simulation error. We tracked participants gaze while they viewedcomplex towers of blocks and predicted whether the towers would remain stable under gravity. We developed a resource-rational model of how people might optimally partition towers into chunks of blocks. Subsequently, we compared thismodel to participants fixations over the scene. We explore how efficient, resource-rational chunkings of physical scenesmight underlie peoples ability to make rapid and robust inferences in this domain.

Liquids can splash, squirt, gush, slosh, soak, drip, drain, trickle, pool, and be poured–complex behaviors that we can easily distinguish, imagine, describe, and, crucially, predict, despite tremendous diversity among different liquids’ material and dynamical characteristics. This proficiency suggests the brain has a sophisticated cognitive mechanism for reasoning about liquids, yet to date there has been little effort to study this mechanism quantitatively or describe it computationally. Here we find evidence that people’s reasoning about how liquids move is consistent with a computational cognitive model based on approximate probabilistic simulation. In a psychophysical experiment, participants predicted how different liquids would flow around solid obstacles, and their judgments agreed with those of a family of models in which volumes of liquid are represented as collections of interacting particles, within a dynamical fluid simulation. Our model explains people’s accuracy, and their predictions’ sensitivity to liquids of different viscosity. We also explored several models that did not involve simulation, and found they could not account for the experimental data as well. Our results are consistent with previous reports that people’s physical understanding of solid objects is based on simulation, but extends this thesis to the more complex and unexplored domain of reasoning about liquids

While current deep learning systems excel at tasks such asobject classification, language processing, and gameplay, fewcan construct or modify a complex system such as a tower ofblocks. We hypothesize that what these systems lack is a “re-lational inductive bias”: a capacity for reasoning about inter-object relations and making choices over a structured descrip-tion of a scene. To test this hypothesis, we focus on a task thatinvolves gluing pairs of blocks together to stabilize a tower,and quantify how well humans perform. We then introducea deep reinforcement learning agent which uses object- andrelation-centric scene and policy representations and apply itto the task. Our results show that these structured represen-tations allow the agent to outperform both humans and morena ̈ıve approaches, suggesting that relational inductive bias isan important component in solving structured reasoning prob-lems and for building more intelligent, flexible machines.