Reduction in hippocampal and amygdala volume measured via structural magnetic resonance imaging is an early marker of Alzheimer's disease (AD). Whether genetic risk factors for AD exert an effect on these subcortical structures independent of clinical status has not been fully investigated. We examine whether increased genetic risk for AD influences hippocampal and amygdala volumes in case-control and population cohorts at different ages, in 1674 older (aged >53 years; 17% AD, 39% mild cognitive impairment [MCI]) and 467 young (16-30 years) adults. An AD polygenic risk score combining common risk variants excluding apolipoprotein E (APOE), and a single nucleotide polymorphism in TREM2, were both associated with reduced hippocampal volume in healthy older adults and those with MCI. APOE ε4 was associated with hippocampal and amygdala volume in those with AD and MCI but was not associated in healthy older adults. No associations were found in young adults. Genetic risk for AD affects the hippocampus before the clinical symptoms of AD, reflecting a neurodegenerative effect before clinical manifestations in older adults.