Proceedings of the Annual Meeting of the Cognitive Science Society

Listeners can use gestures to predict upcoming referents when the semantic information in speech is partially constraining referents in a visual context. However, less is known about the predictive use of gestures when semantically constraining information is in gesture alone or in both speech and gesture. In a visual-world eye-tracking paradigm, 60 Mandarin Chinese speakers viewed four objects and one video. The speaker produced sentences ("I today myself alone VERB a (whole) afternoon NOUN"), paired with an iconic gesture that began 700ms before the verb, the same sentence without gesture, or the same gesture withe the verb replaced by a speech disfluency. In all conditions, participants had more looks to the target object than non-target objects before they heard the noun, but this difference was greater in the gesture plus speech condition. These results demonstrate independent and combined contributions of gesture and speech for predictions grounded in the visual-referential context.