Neural processing of sensory information is strongly influenced by context. For instance, cortical responses are reduced to predictable stimuli, while responses are increased to novel stimuli that deviate from contextual regularities. Such bidirectional modulation based on preceding sensory context is likely a critical component or manifestation of attention, learning, and behavior, yet how it arises in cortical circuits remains unclear. Using volumetric two-photon calcium imaging and local field potentials in primary visual cortex (V1) from awake mice presented with visual "oddball" paradigms, we identify both reductions and augmentations of stimulus-evoked responses depending, on whether the stimulus was redundant or deviant, respectively. Interestingly, deviance-augmented responses were limited to a specific subset of neurons mostly in supragranular layers. These deviance-detecting cells were spatially intermixed with other visually responsive neurons and were functionally correlated, forming a neuronal ensemble. Optogenetic suppression of prefrontal inputs to V1 reduced the contextual selectivity of deviance-detecting ensembles, demonstrating a causal role for top-down inputs. The presence of specialized context-selective ensembles in primary sensory cortex, modulated by higher cortical areas, provides a circuit substrate for the brain's construction and selection of prediction errors, computations which are key for survival and deficient in many psychiatric disorders.