Proceedings of the Annual Meeting of the Cognitive Science Society

This study investigated the neural substrate of processing gesture-speech semantic variation which arises when the same entity word is uttered along with an iconic gesture that depicts object knowledge or event knowledge of the entity concept, or with a beat gesture that enacts a rhythmic movement, or with a grooming gesture. The fMRI results attested that the observation of gestures with speech, as compared to speech alone, centers on the high-level visual processing and recognition of complex stimuli in the bilateral fusiform gyrus, and the association of various sensory information in spatio-motoric and semantic processing of the inputs in the left inferior/superior parietal gyrus, upon which multiple functional networks are engaged in response to cross-modal semantic variation. The visuo-sensorimotor network of gesture processing with speech does not resemble the processing of hand actions on real objects in the frontal and parietal lobes for the recognition of object-directed motor acts.