Neurodevelopmental origins of functional variation in older age are increasingly being acknowledged, but identification of how early factors impact human brain and cognition throughout life has remained challenging. Much focus has been on age-specific mechanisms affecting neural foundations of cognition and their change. In contrast to this approach, we tested whether cerebral correlates of general cognitive ability (GCA) in development could be extended to the rest of the lifespan, and whether early factors traceable to prenatal stages, such as birth weight and parental education, may exert continuous influences. We measured the area of the cerebral cortex in a longitudinal sample of 974 individuals aged 4-88 y (1,633 observations). An extensive cortical region was identified wherein area related positively to GCA in development. By tracking area of the cortical region identified in the child sample throughout the lifespan, we showed that the cortical change trajectories of higher and lower GCA groups were parallel through life, suggesting continued influences of early life factors. Birth weight and parental education obtained from the Norwegian Mother-Child Cohort study were identified as such early factors of possible life-long influence. Support for a genetic component was obtained in a separate twin sample (Vietnam Era Twin Study of Aging), but birth weight in the child sample had an effect on cortical area also when controlling for possible genetic differences in terms of parental height. Our results provide novel evidence for stability in brain-cognition relationships throughout life, and indicate that early life factors impact brain and cognition for the entire life course.