Individual differences in white matter tract microstructure, measured with diffusion tensor imaging (DTI), demonstrate substantial heritability. However, it is unclear to what extent this heritability reflects global genetic influences or tract-specific genetic influences. The goal of the current study was to quantify the proportion of genetic and environmental variance in white matter tracts attributable to global versus tract-specific influences. We assessed fractional anisotropy (FA), mean diffusivity (MD), axial diffusivity (AD), and radial diffusivity (RD) across 11 tracts and 22 subdivisions of these tracts in 392 middle-aged male twins from the Vietnam Era Twin Study of Aging (VETSA). In principal component analyses of the 11 white matter tracts, the first component, which represents the global signal, explained 50.1% and 62.5% of the variance in FA and MD, respectively. Similarly, the first principal component of the 22 tract subdivisions explained 38.4% and 47.0% of the variance in FA and MD, respectively. Twin modeling revealed that DTI measures of all tracts and subdivisions were heritable, and that genetic influences on global FA and MD accounted for approximately half of the heritability in the tracts or tract subdivisions. Similar results were observed for the AD and RD diffusion metrics. These findings underscore the importance of controlling for DTI global signals when measuring associations between specific tracts and outcomes such as cognitive ability, neurological and psychiatric disorders, and brain aging.