Proceedings of the Annual Meeting of the Cognitive Science Society

Listeners are influenced by speakers hand gestures. However, it is not clear what processes support gesture processing.We investigated listeners behavior after observing speech with videotaped gestures or with point light gesture trajectoriesin the Tower of Hanoi task. Listeners were influenced by the synchrony of the visual and auditory information but not thenature of the information both videotaped and point light gestures reliably influenced behavior. Thus, visual informationthat is not perceived as produced by the speaker nonetheless reliably influences listeners behavior, so long as informationis synchronized across modalities. Thus, observers do not appear to rely on functional or biological links between speechand hand gesture but rather on more general processes of multimodal integration. The principles underlying integrationof auditory language with visual information from hand gestures appear to different from those underlying integration ofauditory language and visual speech.