How do iconic co-speech gestures contribute to the truth-conditions of assertions: A surprisal-based ERP investigation targeting N400 and late positivity effects
Abstract
To investigate how co-speech gestures modulate linguistic understanding in comparison to specific verbs with similar contents, we conducted an EEG experiment to measure semantic surprisal by exploring the amplitude changes in the N400 component for both information types. We used videos of a person uttering (i) specified sentences, whose verb semantically denoted a specific action, and (ii) underspecified sentences with an unspecific verb accompanied by an iconic co-speech gesture that indicated the specific action of the previous condition. The subsequent sentence contained an instrument noun as target, which either matched or mismatched the specific action. We measured ERPs on the target noun for both information types and found an N400 effect for mismatching target nouns as well as a late positivity effect for gesture conditions. Crucially, no interaction between information type and congruency was observed in the N400 time-window, indicating similar semantic surprisal induced by mismatching specific verbs and content-similar gestures. An interaction was revealed in the late time-window, suggesting that unlike for gestures, the mismatch negativity effect was prolonged for specific verbs.