Proceedings of the Annual Meeting of the Cognitive Science Society

Spoken language processing requires integrating acoustic signals with visual information. Deictic gestures such as pointing may facilitate this integration by constraining the referential domain before linguistic disambiguation. How multimodal cues modulate processing under cognitive load remains unclear. Simultaneous interpreting provides a task environment for examining speech processing under maximal cognitive load. In this study, we combined the Visual World Paradigm (VWP) with pupillometry to investigate how congruent, incongruent, and neutral pointing gestures influence attention and cognitive load in 24 professional interpreters. Eye-tracking revealed that congruent gestures elicited early anticipatory target fixations, incongruent gestures directed gaze toward competitors before rapid correction and looks towards the target, while the absence of gestures delayed target identification. Pupillometric measures yielded a counterintuitive pattern: the neutral condition evoked greater pupil dilation than the incongruent condition, whereas congruent and incongruent conditions did not differ significantly in the amount of generated load. This suggests that visual cues, even misleading ones, reduce processing demands when they accompany auditory input. These findings suggest that multimodal information alleviates cognitive load in a task environment competing for resources.