Aging entails a multifaceted complex of changes in macro- and micro-structural properties of human brain gray matter (GM) and white matter (WM) tissues, as well as in intellectual abilities. To better capture tissue-specific brain aging, we combined volume and distribution properties of diffusivity indices to derive subject-specific age scores for each tissue. We compared age-related variance between younger and older adults for GM and WM age scores, and tested whether tissue-specific age scores could explain different effects of aging on fluid (Gf) and crystalized (Gc) intelligence in younger and older adults. Chronological age was strongly associated with GM (R2 = 0.73) and WM (R2 = 0.57) age scores. The GM age score accounted for significantly more variance in chronological age in younger relative to older adults (p < 0.001), whereas the WM age score accounted for significantly more variance in chronological age in older compared to younger adults (p < 0.025). Consistent with existing literature, younger adults outperformed older adults in Gf while older adults outperformed younger adults in Gc. The GM age score was negatively associated with Gf in younger adults (p < 0.02), whereas the WM age score was negatively associated with Gc in older adults (p < 0.02). Our results provide evidence for differences in the effects of age on GM and WM in younger versus older adults that may contribute to age-related differences in Gf and Gc.