Previous neuroimaging studies suggest a negative relationship between the apolipoprotein (ApoE) ε4 allele and brain integrity in human immunodeficiency virus (HIV)-infected (HIV+) individuals, although the presence of this relationship across adulthood remains unclear. The purpose of this study is to clarify the discrepancies using a large, diverse group of HIV+ individuals and multiple imaging modalities sensitive to HIV. The association of ApoE ε4 with structural neuroimaging and magnetic resonance spectroscopy (MRS) was examined in 237 HIV+ individuals in the CNS HIV Anti-Retroviral Therapy Effects Research (CHARTER) study. Cortical and subcortical gray matter, abnormal and total white matter, ventricles, sulcal cerebrospinal fluid (CSF), and cerebellar gray matter, white matter, and CSF volumes, and MRS concentrations of myo-inositol, creatine, N-acetyl-aspartate, and choline in the frontal white matter (FWM), frontal gray matter (FGM), and basal ganglia were examined. Secondary analyses explored this relationship separately in individuals ≥50 years old (n = 173) and <50 years old (n = 63). No significant differences were observed between ApoE ε4+ (ApoE ε3/ε4 and ApoE ε4/ε4) individuals (n = 69) and ApoE ε4- (ApoE ε2/ε3 and ApoE ε3/ε3) individuals (n = 167). When individuals were further divided by age, no significant genotype group differences were identified in individuals <50 or ≥50 years of age on any neuroimaging outcome. The ApoE ε4 allele did not affect brain integrity in this large, diverse sample of HIV+ individuals. The effects of ApoE ε4 may not be apparent until more advanced ages and may be more prominent when present along with other risk factors for neuronal damage.