BTB/POZ domain-containing 3 (BTBD3) was identified as a potential risk gene in the first genome-wide association study of obsessive-compulsive disorder (OCD). BTBD3 is a putative transcription factor implicated in dendritic pruning in developing primary sensory cortices. We assessed whether BTBD3 also regulates neural circuit formation within limbic cortico-striato-thalamo-cortical circuits and behaviors related to OCD in mice. Behavioral phenotypes associated with OCD that are measurable in animals include compulsive-like behaviors and reduced exploration. We tested Btbd3 wild-type, heterozygous, and knockout mice for compulsive-like behaviors including cage-mate barbering, excessive wheel-running, repetitive locomotor patterns, and reduced goal-directed behavior in the probabilistic learning task (PLT), and for exploratory behavior in the open field, digging, and marble-burying tests. Btbd3 heterozygous and knockout mice showed excessive barbering, wheel-running, impaired goal-directed behavior in the PLT, and reduced exploration. Further, chronic treatment with fluoxetine, but not desipramine, reduced barbering in Btbd3 wild-type and heterozygous, but not knockout mice. In contrast, Btbd3 expression did not alter anxiety-like, depression-like, or sensorimotor behaviors. We also quantified dendritic morphology within anterior cingulate cortex, mediodorsal thalamus, and hippocampus, regions of high Btbd3 expression. Surprisingly, Btbd3 knockout mice only showed modest increases in spine density in the anterior cingulate, while dendritic morphology was unaltered elsewhere. Finally, we virally knocked down Btbd3 expression in whole, or just dorsal, hippocampus during neonatal development and assessed behavior during adulthood. Whole, but not dorsal, hippocampal Btbd3 knockdown recapitulated Btbd3 knockout phenotypes. Our findings reveal that hippocampal Btbd3 expression selectively modulates compulsive-like and exploratory behavior.