It has been postulated that pubertal hormones may drive some neuroanatomical changes during adolescence, and may do so differently in girls and boys. Here, we use growth curve modeling to directly assess how sex hormones [testosterone (T) and estradiol (E₂)] relate to changes in subcortical brain volumes utilizing a longitudinal design. 126 adolescents (63 girls), ages 10 to 14, were imaged and restudied ∼2 years later. We show, for the first time, that best-fit growth models are distinctly different when using hormones as compared to a physical proxy of pubertal maturation (Tanner Stage) or age, to predict brain development. Like Tanner Stage, T and E₂ predicted white matter and right amygdala growth across adolescence in both sexes, independent of age. Tanner Stage also explained decreases in both gray matter and caudate volumes, whereas E₂ explained only gray matter decreases and T explained only caudate volume decreases. No pubertal measures were related to hippocampus development. Although specificity was seen, sex hormones had strikingly similar relationships with white matter, gray matter, right amygdala, and bilateral caudate volumes, with larger changes in brain volume seen at early pubertal maturation (as indexed by lower hormone levels), followed by less robust, or even reversals in growth, by late puberty. These novel longitudinal findings on the relationship between hormones and brain volume change represent crucial first steps toward understanding which aspects of puberty influence neurodevelopment.