Objective: To investigate whether APOE ε4 affects the association of verbal memory with neurodegeneration presented by the hippocampal volume/intracranial volume ratio (HpVR). Methods: The study sample included 371 individuals with normal cognition (NC), 725 subjects with amnestic mild cognitive impairment (aMCI), and 251 patients with mild Alzheimer's disease (AD) from the Alzheimer's Disease Neuroimaging Initiative (ADNI) who underwent the rey auditory verbal learning test (RAVLT). Multiple linear regression models were conducted to assess the effect of the APOE ε4^∗HpVR interaction on RAVLT in all subjects and in each diagnostic group adjusting for age, gender and educational attainment, and global cognition. Results: In all subjects, there was no significant APOE ε4 × HpVR interaction for immediate recall or delayed recall (p > 0.05). However, in aMCI subjects, there was a significant APOE ε4 × HpVR interaction for delayed recall (p = 0.008), but not immediate recall (p = 0.15). More specifically, the detrimental effect of APOE ε4 on delayed recall altered by HpVR such that this effect was most evident among subjects with small to moderate HpVR, but this disadvantage was absent or even reversed among subjects with larger HpVR. No significant interaction was observed in the NC or AD group. Conclusion: These findings highlight a potential role of APOE ε4 status in affecting the association of hippocampus size with delayed recall memory in the early stage of AD.