The physical world is rich with social information that people readily detect and extract, such as inferring that someone was present when we encounter a stack of rocks in the woods. How do people recognize that a physical scene contains social information? Research in developmental psychology has argued that this capacity is supported by a sensitivity to violations of randomness. Here we present a computational model of this idea and test its explanatory power in a quantitative manner. Our model infers agency by estimating the likelihood that a scene would arise naturally, as determined by human intuitive physics instantiated as a physics engine. Our results suggest that people's ability to detect agency in a physical scene is sensitive not only to the superficial visual properties, but also to the underlying physical generative process. Our results highlight how people use intuitive physics to decide when to engage in nuanced social reasoning.