Chromatin immunoprecipitation followed by massively parallel, high throughput sequencing (ChIP-seq) is the method of choice for genome-wide identification of DNA segments bound by specific transcription factors or in chromatin with particular histone modifications. However, the quality of ChIP-seq datasets varies widely, with a substantial fraction being of intermediate to poor quality. Thus, it is important to discern and control the factors that contribute to variation in ChIP-seq. In this study, we focused on sonication, a user-controlled variable, to produce sheared chromatin. We systematically varied the amount of shearing of fixed chromatin from a mouse erythroid cell line, carefully measuring the distribution of resultant fragment lengths prior to ChIP-seq. This systematic study was complemented with a retrospective analysis of additional experiments. We found that the level of sonication had a pronounced impact on the quality of ChIP-seq signals. Over-sonication consistently reduced quality, while the impact of under-sonication differed among transcription factors, with no impact on sites bound by CTCF but frequently leading to the loss of sites occupied by TAL1 or bound by POL2. The bound sites not observed in low-quality datasets were inferred to be a mix of both direct and indirect binding. We leveraged these findings to produce a set of CTCF ChIP-seq datasets in rare, primary hematopoietic progenitor cells. Our observation that the amount of chromatin sonication is a key variable in success of ChIP-seq experiments indicates that monitoring the level of sonication can improve ChIP-seq quality and reproducibility and facilitate ChIP-seq in rare cell types.