Although glucocorticoid receptors are highly expressed in the prefrontal cortex, the hippocampus remains the predominant focus in the literature examining relationships between cortisol and brain. We examined phenotypic and genetic associations of cortisol levels with the thickness of prefrontal and anterior cingulate cortex regions, and with hippocampal volume in a sample of 388 middle-aged male twins who were 51-59 years old. Small but significant negative phenotypic associations were found between cortisol levels and the thickness of left dorsolateral (superior frontal gyrus, left rostral middle frontal gyrus) and ventrolateral (pars opercularis, pars triangularis, pars orbitalis) prefrontal regions, and right dorsolateral (superior frontal gyrus) and medial orbital frontal cortex. Most of the associations remained significant after adjusting for general cognitive ability, cardiovascular risk factors, and depression. Bivariate genetic analyses suggested that some of the associations were primarily accounted for by shared genetic influences; that is, some of the genes that tend to result in increased cortisol levels also tend to result in reduced prefrontal cortical thickness. Aging has been associated with reduced efficiency of hypothalamic-pituitary-adrenal function, frontal lobe shrinkage, and increases in health problems, but our present data do not allow us to determine the direction of effects. Moreover, the degree or the direction of the observed associations and the extent of their shared genetic underpinnings may well change as these individuals age. Longitudinal assessments are underway to elucidate the direction of the associations and the genetic underpinnings of longitudinal phenotypes for changes in cortisol and brain morphology.