Fragile X syndrome (FXS), the most common monogenic cause of autism, is often associated with hypersensitivity to sound. Several studies have shown abnormalities in the auditory brainstem in FXS; however, the emergence of these auditory phenotypes during development has not been described. Here, we investigated the development of phenotypes in FXS model [Fmr1 knockout (KO)] mice in the ventral cochlear nucleus (VCN), medial nucleus of the trapezoid body (MNTB), and lateral superior olive (LSO). We studied features of the brainstem known to be altered in FXS or Fmr1 KO mice, including cell size and expression of markers for excitatory (VGLUT) and inhibitory (VGAT) synapses. We found that cell size was reduced in the nuclei with different time courses. VCN cell size is normal until after hearing onset, while MNTB and LSO show decreases earlier. VGAT expression was elevated relative to VGLUT in the Fmr1 KO mouse MNTB by P6, before hearing onset. Because glial cells influence development and are altered in FXS, we investigated their emergence in the developing Fmr1 KO brainstem. The number of microglia developed normally in all three nuclei in Fmr1 KO mice, but we found elevated numbers of astrocytes in Fmr1 KO in VCN and LSO at P14. The results indicate that some phenotypes are evident before spontaneous or auditory activity, while others emerge later, and suggest that Fmr1 acts at multiple sites and time points in auditory system development.