Proceedings of the Annual Meeting of the Cognitive Science Society

This paper provides empirical evidence that multimodal signals are produced and understood as integrated units of communication called composite signals, rather than being independently interpretable "channels" of communication. I propose that using composite signals relies on two communicative norms, co-expressivity and consistency: - co-expressivity: each element of a composite signal refers to the same underlying referent • consistency: elements of the same composite do not contradict each other. This paper will show that these norms are consistent with data comprising a set of explanations of how locks work in which participants spoke while gesturing, drawing diagrams, and manipulating a sample lock. Co-expressivity is supported by the fact that co-expressive speech segments can be found in nearby speech for communicative nonverbal behaviors but not for non-communicative nonverbal behaviors. Consistency is evidenced in inferences that maintain number and modality consistency in cases of apparent contradiction.