To investigate how co-speech gestures modulate linguistic understanding, we experimentally test two Bayesian Pragmatic models. We identify the semantic effect of a spoken or gestural utterance with the change it makes in listener’s probabilistic predictions of the speaker’s communicative intentions. We focus on action-expressing gestures and the respective verbs that correspond to action-affording instruments or, respectively, their denoting nouns. Combining Pustejovsky’s’ Generative Lexicon approach with Gibson’s affordance theory, we ask: (1) Does a co-speech gesture make any difference for semantic comprehension and the corresponding probabilistic prediction? (2) Is the semantic effect of a gesture similar or identical to the one of the corresponding verb? (3) To which extent does the gesture’s semantic effect depend on the listener’s recognition of the gesture as an expression of the corresponding verb? (4) Does the comprehended affordance predict the instrument better than the co-occurrence statistics (GloVe) regarding the verb and the noun?

To investigate how co-speech gestures modulate linguistic understanding, we conducted an EEG experiment exploring the amplitude changes in the N400 component. We used videos of a person uttering underspecified action sentences which either featured no gesture or an iconic co-speech gesture that represented a more specific action. The following target sentence contained an instrument noun followed by its required action verb; these could either match the action represented in the previously seen gesture or mismatch it. We measured ERPs for both the nouns and the verbs and found an N400 effect for mismatching target words as well as a sustained positivity effect for both gesture conditions.