The physical interactions of growing bacterial cells with each other and with their surroundings significantly affect the structure and dynamics of biofilms. Here a 3D agent-based model is formulated to describe the establishment of simple bacterial colonies expanding by the physical force of their growth. With a single set of parameters, the model captures key dynamical features of colony growth by non-motile, non EPS-producing E. coli cells on hard agar. The model, supported by experiment on colony growth in different types and concentrations of nutrients, suggests that radial colony expansion is not limited by nutrients as commonly believed, but by mechanical forces. Nutrient penetration instead governs vertical colony growth, through thin layers of vertically oriented cells lifting up their ancestors from the bottom. Overall, the model provides a versatile platform to investigate the influences of metabolic and environmental factors on the growth and morphology of bacterial colonies.