Background To determine if accelerometer measured sedentary behavior (SED), light-intensity physical activity (LPA), and moderate-to-vigorous-intensity physical activity (MVPA) in midlife is prospectively associated with cognitive function. Methods and Results Participants were 1970 adults enrolled in the CARDIA (Coronary Artery Risk Development in Young Adults) study who wore an accelerometer in 2005 to 2006 (ages 38-50 years) and had cognitive function assessments completed 5 and/or 10 years later. SED, LPA, and MVPA were measured by an ActiGraph 7164 accelerometer. Cognitive function tests included the Digit Symbol Substitution Test, Rey Auditory Verbal Learning Test, and Stroop Test. Compositional isotemporal substitution analysis examined associations of SED, LPA, and MVPA with repeated measures of the cognitive function standardized scores. In men, statistical reallocation of 30 minutes of LPA with 30 minutes of MVPA resulted in an estimated difference of SD 0.07 (95% CI, 0.01-0.14), SD 0.09 (95% CI, 0.02-0.17), and SD -0.11 (95% CI, -0.19 to -0.04) in the Digit Symbol Substitution Test, Rey Auditory Verbal Learning Test, and Stroop scores, respectively, indicating better performance. Associations were similar when reallocating time in SED with MVPA, but results were less robust. Reallocation of time in SED with LPA resulted in an estimated difference of SD -0.05 (95% CI, -0.06 to -0.03), SD -0.03 (95% CI, -0.05 to -0.01), and SD 0.05 (95% CI, 0.03- 0.07) in the Digit Symbol Substitution Test, Rey Auditory Verbal Learning Test, and Stroop scores, respectively, indicating worse performance. Associations were largely nonsignificant among women. Conclusions Our findings support the idea that for men, higher-intensity activities (MVPA) may be necessary in midlife to observe beneficial associations with cognition.