The risk of APOE for Alzheimer's disease (AD) is modified by age. Beyond APOE, the polygenic architecture may also be heterogeneous across age. We aim to investigate age-related genetic heterogeneity of AD and identify genomic loci with differential effects across age. Stratified gene-based genome-wide association studies and polygenic variation analyses were performed in the younger (60-79 years, N = 14,895) and older (≥80 years, N = 6559) age-at-onset groups using Alzheimer's Disease Genetics Consortium data. We showed a moderate genetic correlation (r_g = 0.64) between the two age groups, supporting genetic heterogeneity. Heritability explained by variants on chromosome 19 (harboring APOE) was significantly larger in younger than in older onset group (p < 0.05). APOE region, BIN1, OR2S2, MS4A4E, and PICALM were identified at the gene-based genome-wide significance (p < 2.73 × 10^-6) with larger effects at younger age (except MS4A4E). For the novel gene OR2S2, we further performed leave-one-out analyses, which showed consistent effects across subsamples. Our results suggest using genetically more homogeneous individuals may help detect additional susceptible loci.