Recent exome sequencing studies have implicated polymorphic Brg1-associated factor (BAF) complexes (mammalian SWI/SNF chromatin remodeling complexes) in several human intellectual disabilities and cognitive disorders. However, it is currently unknown how mutations in BAF complexes result in impaired cognitive function. Postmitotic neurons express a neuron-specific assembly, nBAF, characterized by the neuron-specific subunit BAF53b. Mice harboring selective genetic manipulations of BAF53b have severe defects in long-term memory and long-lasting forms of hippocampal synaptic plasticity. We rescued memory impairments in BAF53b mutant mice by reintroducing BAF53b in the adult hippocampus, which suggests a role for BAF53b beyond neuronal development. The defects in BAF53b mutant mice appeared to derive from alterations in gene expression that produce abnormal postsynaptic components, such as spine structure and function, and ultimately lead to deficits in synaptic plasticity. Our results provide new insight into the role of dominant mutations in subunits of BAF complexes in human intellectual and cognitive disorders.