Conventional theories of visual object recognition treat objectseffectively as abstract, arbitrary patterns of image features.They do not explicitly represent objects as physical entities inthe world, with physical properties such as three-dimensionalshape, mass, stiffness, elasticity, surface friction, and so on.However, for many purposes, an object’s physical existence iscentral to our ability to recognize it and think about it. Thisis certainly true for recognition via haptic perception, i.e., per-ceiving objects by touch, but even in the visual domain an ob-ject’s physical properties may directly determine how it looksand thereby how we recognize it. Here we show how a physi-cal object representation can allow the solution of visual prob-lems, like perceiving an object under a cloth, that are other-wise difficult to accomplish without extensive experience, andwe provide behavioral and computational evidence that peoplecan use such a representation.