Humans make plans with remarkable flexibility by leveraging symbolic representations. How are these representationslearned? We present a model that starts out with a language of low-level physical constraints and, by observing expertdemonstrations, builds up a library of high-level concepts that afford planning and action understanding. We demonstrateits versatility through experiments inspired by developmental psychology literature.