The precise and specialized circuitry in the auditory brainstem develops through adaptations of cellular and molecular signaling. We previously showed that elimination of microglia during development impairs synaptic pruning that leads to maturation of the calyx of Held, a large encapsulating synapse that terminates on neurons of the medial nucleus of the trapezoid body (MNTB). Microglia depletion also led to a decrease in glial fibrillary acidic protein (GFAP), a marker for mature astrocytes. Here, we investigated the role of signaling through the fractalkine receptor (CX3CR1), which is expressed by microglia and mediates communication with neurons. CX3CR1^-/- and wild-type mice were studied before and after hearing onset and at 9 weeks of age. Levels of GFAP were significantly increased in the MNTB in mutants at 9 weeks. Pruning was unaffected at the calyx of Held, but we found an increase in expression of glycinergic synaptic marker in mutant mice at P14, suggesting an effect on maturation of inhibitory inputs. We observed disrupted tonotopic gradients of neuron and calyx size in MNTB in mutant mice. Auditory brainstem recording (ABR) revealed that CX3CR1^-/- mice had normal thresholds and amplitudes but decreased latencies and interpeak latencies, particularly for the highest frequencies. These results demonstrate that disruption of fractalkine signaling has a significant effect on auditory brainstem development. Our findings highlight the importance of neuron-microglia-astrocyte communication in pruning of inhibitory synapses and establishment of tonotopic gradients early in postnatal development.