We compared the cognitive and language development at 4, 14, 24, 36, 54 months, and 7 years of siblings of children with autism (SIBS-A) to that of siblings of children with typical development (SIBS-TD) using growth curve analyses. At 7 years, 40% of the SIBS-A, compared to 16% of SIBS-TD, were identified with cognitive, language and/or academic difficulties, identified using direct tests and/or parental reports. This sub-group was identified as SIBS-A-broad phenotype (BP). Results indicated that early language scores (14–54 months), but not cognitive scores of SIBS-A-BP and SIBS-A-nonBP were significantly lower compared to the language scores of SIBS-TD, and that the rate of development was also significantly different, thus pinpointing language as a major area of difficulty for SIBS-A during the preschool years.