Proceedings of the Annual Meeting of the Cognitive Science Society

Human communication has a remarkable capacity todescribe events that occurred elsewhere and at othertimes. In particular, when describing complex narratives,speakers must communicate temporal structure using amixture of words (e.g., “after”), gestures (e.g., pointingrightward for a later event), and discourse structure (e.g.,mentioning earlier events first). How do listenersintegrate these sources of temporal information to makesense of complex narratives? In two experiments, wesystematically manipulated gesture, speech, and order-of-mention to investigate their respective impacts oncomprehension of temporal structure. Gesture had asignificant effect on interpretations of temporal order.This influence of gesture, however, was weaker than theinfluence of both speech and order-of-mention. Indeed,in some cases, order-of-mention trumped explicitdescriptions in speech; for instance, if ‘earlier’ eventswere mentioned second, they were sometimes thought tohave occurred second. Listeners integrate multiplesources of information to interpret what happened when.