Atrophy of the hippocampus and surrounding temporal regions occurs in Alzheimer's disease (AD). APOE ε4, the major genetic risk factor for late-onset AD, has been associated with smaller volume in these regions before amyloidosis can be detected by AD biomarkers. To examine APOE ε4 effects in relation to aging, we performed a longitudinal magnetic resonance imaging study involving cognitively normal adults (25 APOE ε4 carriers and 31 ε3 homozygotes), initially aged 51-75 years. We used growth curve analyses, which can provide information about APOE ε4-related differences initially and later in life. Hippocampal volume was the primary outcome; nearby medial temporal regions were secondary outcomes. Brain-derived neurotrophic factor, val66met was a secondary covariate. APOE ε4 carriers had significantly smaller initial hippocampal volumes than ε3 homozygotes. Rate of hippocampal atrophy was not greater in the APOE ε4 group, although age-related atrophy was detected in the overall sample. The findings add to the growing evidence that effects of APOE ε4 on hippocampal size begin early in life, underscoring the importance of early interventions to increase reserve.