Neighborhood socioeconomic status (NSES) is associated with cognitive function, independently of individual demographic, health, and socioeconomic characteristics. However, research has been largely cross-sectional, and mechanisms of the association are unknown. In 1992-1993, Cardiovascular Health Study participants (n = 3,595; mean age = 74.8 years; 15.7% black) underwent cognitive testing and magnetic resonance imaging of white matter hyperintensities (WMH), and their addresses were geocoded. NSES was calculated using 1990 US Census data (block groups; 6 measures of wealth, education, and occupation). The Modified Mini-Mental State Examination (3MS) was used to assess general cognition, and the Digit Symbol Substitution Test (DSST) was used to assess speed of processing annually for 6 years. Associations of race-specific NSES tertiles with 3MS, DSST, and WMH were estimated using linear mixed-effects models accounting for geographic clustering, stratified by race, and adjusted for demographic, health, and individual socioeconomic status (education, income, lifetime occupational status) variables. In fully adjusted models, higher NSES was associated with higher 3MS scores in blacks (mean difference between highest and lowest NSES = 2.4 points; P = 0.004) and whites (mean difference = 0.7 points; P = 0.02) at baseline but not with changes in 3MS over time. NSES was marginally associated with DSST and was not associated with WMH. Adjustment for WMH did not attenuate NSES-3MS associations. Associations of NSES with cognition in late adulthood differ by race, are not explained by WMH, and are evident only at baseline.