Previous studies have reported associations of polymorphisms in the IGF1 gene with phenotypes of body composition (BC). The purpose of this study was to identify phenotypes of BC and physical function that were associated with the IGF1 promoter polymorphism (rs35767, −C1245T). Subjects from the Health, Aging, and Body Composition Study, white males and females (n = 925/836) and black males and females (533/705) aged 70–79 years were genotyped for the polymorphism. Phenotypes of muscle size and function, bone mineral density, and BC were analyzed for associations with this polymorphism. To validate and compare these findings, a cohort of young (mean age = 24.6, SD = 5.9) white men and women (n = 173/296) with similar phenotypic measurements were genotyped. An association with BC was identified in elderly females when significant covariates (physical activity, age, smoking status, body mass index) were included. White women with C/C genotype had 3% more trunk fat and 2% more total fat than those with C/T (P < 0.05). Black women with C/C genotype had 3% less total lean mass and 3% less muscle mass than their T/T counterparts (P < 0.05). Associations were identified with muscle strength in white women (P < 0.01) that were in agreement with the C/C genotype having lower muscle function. Thus, in an elderly population but not a young population, a polymorphism in the IGF1 gene may be predictive of differences in body composition, primarily in black females.