To identify the top brain regions affected by MS-specific atrophy (i.e., atrophy in excess of normal aging) and to test whether normal aging and MS-specific atrophy increase or decrease in these regions with age. Six hundred fifty subjects (2,790 MRI time points) were analyzed: 520 subjects with relapse-onset MS from a 5-year prospective cohort with annual standardized 1-mm 3D T1-weighted images (3DT1s; 2,483 MRIs) and 130 healthy controls with longitudinal 3DT1s (307 MRIs). Rates of change in all FreeSurfer regions (v5.3) and Structural Image Evaluation Using Normalization of Atrophy (SIENA) were estimated with mixed-effects models. All FreeSurfer regions were ranked by the MS-specific atrophy slope/standard error ratio (βMS × time/SEβMS × time). In the top regions, age was added as an effect modifier to test whether MS-specific atrophy varied by age. The top-ranked regions were all gray matter structures. For SIENA, normal aging increased from 0.01%/y at age 30 years to -0.31%/y at age 60 years (-0.11% ± 0.032%/decade, p < 0.01), whereas MS-specific atrophy decreased from -0.38%/y at age 30 years to -0.12%/y at age 60 years (0.09% ± 0.035%/decade, p = 0.01). Similarly, in the thalamus, normal aging increased from -0.15%/y at age 30 years to -0.62%/y at age 60 years (-0.16% ± 0.079%/decade, p < 0.05), and MS-specific atrophy decreased from -0.59%/y at age 30 years to -0.05%/y at age 60 years (0.18% ± 0.08%/decade, p < 0.05). In the putamen and caudate, normal aging and MS-specific atrophy did not vary by age. For SIENA and thalamic atrophy, the contribution of normal aging increases with age, but does not change in the putamen and caudate. This may have substantial implications to understand the biology of brain atrophy in MS.