UC San Diego

Perception is multimodal in that it results from the integration of sensory inputs simultaneously provided by natural stimuli in multiple modalities. Investigations on how the brain combines information from different senses are key to understanding the mechanism of perception. Multisensory illusions are an interesting aspect of integration of the senses, wherein the percept in one modality is altered by the co-presentation of stimuli in another modality. The neural bases of such illusions can provide fundamental insights into the implementation of sensory integration in the brain, yet have not been investigated in much detail. One such illusion is the sound-induced extra flash illusion wherein a flash presented in conjunction with two pulsed sounds generates the percept of two flashes, of which the second is illusory (Shams et al., 2000, 2002). A thorough analysis of the cortical mechanisms that underlie this striking audio-visual illusion is the focus of the present thesis. The neural correlates of the sound-induced illusory extra flash were investigated using event related potential (ERP) recordings that provide high temporal resolution, along with anatomical source localization techniques. The sensory properties of the illusory flash were further explored in a behavioral study. The key findings revealed that the illusion results from a rapid dynamic interplay between processing in auditory and visual cortical areas in conjunction with activity within polymodal superior temporal cortex. This activation sequence included the finding of a novel ERP component, the PD120, that was elicited rapidly within 30-60 ms of the second sound in the illusion inducing audio-visual configuration. The amplitude of the PD120 was found to be strongly correlated with the frequency of illusory percepts in individual subjects. Source localization analyses in two separate studies, and the absence of polarity inversion of the PD120 component as a function of stimulus field location, demonstrated that the principal generator of this component lies outside of striate cortex, in the ventral occipito-temporal region of extrastriate visual cortex. It was also found that subjects who frequently perceived the illusion showed this early modulation (PD120) in response to other combinations of auditory and visual stimuli, thus pointing to consistent individual differences in the neural connectivity that underlies cross-modal integration. Attention was found to significantly enhance the cross-modal integration processes underlying the illusion and in particular was shown to be crucial to the generation of the PD120 component. Finally, a behavioral study found that the extra flash illusion is a robust phenomenon that can be generated for visual stimulus features such as color and basic shape. Overall the findings underscore the emerging view that multisensory integration involves a rapid dynamic interplay, rather than a serial progression, between unisensory cortices and the traditional cortical association areas such as the superior temporal region. These studies emphasize further that even though unisensory brain areas show preferential responsiveness to a specific modality, they are crucial to the process of multimodal perception