There is an urgent need for identifying nondemented individuals at the highest risk of progressing to Alzheimer's disease (AD) dementia. Here, we evaluated whether a recently validated polygenic hazard score (PHS) can be integrated with known in vivo cerebrospinal fluid (CSF) or positron emission tomography (PET) biomarkers of amyloid, and CSF tau pathology to prospectively predict cognitive and clinical decline in 347 cognitive normal (CN; baseline age range = 59.7-90.1, 98.85% white) and 599 mild cognitively impaired (MCI; baseline age range = 54.4-91.4, 98.83% white) individuals from the Alzheimer's Disease Neuroimaging Initiative 1, GO, and 2. We further investigated the association of PHS with post-mortem amyloid load and neurofibrillary tangles in the Religious Orders Study and Memory and Aging Project (ROSMAP) cohort (N = 485, age at death range = 71.3-108.3). In CN and MCI individuals, we found that amyloid and total tau positivity systematically varies as a function of PHS. For individuals in greater than the 50th percentile PHS, the positive predictive value for amyloid approached 100%; for individuals in less than the 25th percentile PHS, the negative predictive value for total tau approached 85%. High PHS individuals with amyloid and tau pathology showed the steepest longitudinal cognitive and clinical decline, even among APOE ε4 noncarriers. Among the CN subgroup, we similarly found that PHS was strongly associated with amyloid positivity and the combination of PHS and biomarker status significantly predicted longitudinal clinical progression. In the ROSMAP cohort, higher PHS was associated with higher post-mortem amyloid load and neurofibrillary tangles, even in APOE ε4 noncarriers. Together, our results show that even after accounting for APOE ε4 effects, PHS may be useful in MCI and preclinical AD therapeutic trials to enrich for biomarker-positive individuals at highest risk for short-term clinical progression.