Evidence for a reduction in stature between Mesolithic foragers and Neolithic farmers has been interpreted as reflective of declines in health, however, our current understanding of this trend fails to account for the complexity of cultural and dietary transitions or the possible causes of phenotypic change. The agricultural transition was extended in primary centers of domestication and abrupt in regions characterized by demic diffusion. In regions such as Northern Europe where foreign domesticates were difficult to establish, there is strong evidence for natural selection for lactase persistence in relation to dairying. We employ broad-scale analyses of diachronic variation in stature and body mass in the Levant, Europe, the Nile Valley, South Asia, and China, to test three hypotheses about the timing of subsistence shifts and human body size, that: 1) the adoption of agriculture led to a decrease in stature, 2) there were different trajectories in regions of in situ domestication or cultural diffusion of agriculture; and 3) increases in stature and body mass are observed in regions with evidence for selection for lactase persistence. Our results demonstrate that 1) decreases in stature preceded the origins of agriculture in some regions; 2) the Levant and China, regions of in situ domestication of species and an extended period of mixed foraging and agricultural subsistence, had stable stature and body mass over time; and 3) stature and body mass increases in Central and Northern Europe coincide with the timing of selective sweeps for lactase persistence, providing support for the "Lactase Growth Hypothesis."