UC San Diego

The world presents information via a variety of sensory channels. To make sense of this information, we must determine what is relevant and ignore unhelpful noise. We then integrate congruent information within and across modalities to build coherent perceptions. Importantly, immediate goals and prevailing environmental factors may interact to affect our perceptual decisions. This dynamic process of multisensory integration is essential to successful perception in the real world, but can also lead to errors. The current project exploits some of these perceptual errors to explore how endogenous (task-directed) and exogenous (stimulus intensity) factors may influence multisensory integration.

In a series of four experiments, we use the sound-induced flash illusion (SFI; Shams et al., 2000; 2002) and related audiovisual effects as indices of multisensory integration. Endogenous attention was manipulated using a focused attention visual task and a novel bimodal conditional attention task. In our first two experiments, we found that participants reported more illusions when attending to both sensory modalities. This effect was larger when the auditory stimuli were presented at near-threshold levels. Perceptual sensitivity (d′) was also found to decrease in the bimodal condition. We then manipulated auditory intensity in each of these tasks independently. Reports of the SFI were found to increase with the higher intensity auditory stimuli. However, differences in reporting these illusions within the same task were attributable to both changes in bias (c) and d′.

Event-related potentials recorded in our first experiment revealed that the SFI was associated with smaller P3 potentials than found in valid targets. We also noted differences in the response-locked error positivity (Pe), with illusory stimuli having more positive amplitudes than real targets. However, the earlier occurring error-related negativity (ERN) was indistinguishable in real and illusory targets. This suggests that participants were less confident of the illusion during stimulus evaluation and one stage of response monitoring. We evaluate these results in terms of the directed attention and information reliability hypotheses (Andersen et al., 2004, 2005) and discuss how these and similar experiments may deepen our understanding of how multisensory perception is impacted at multiple stages of stimulus and response evaluation.