Iconicity, i.e. resemblance between form and meaning, is awidespread feature of natural language vocabulary (Perniss,Thompson, & Vigliocco, 2010), and has been shown tofacilitate vocabulary acquisition (Imai, Kita, Nagumo, &Okada). But what kind of advantage does iconicity actuallygive? Here we use cross-situational learning (Yu & Smith,2007), to address the question for sound-shape iconicity (theso-called kiki-bouba effect, Ramachandran & Hubbard,2001). In contrast to Monaghan, Mattock, and Walker (2012),Experiment 1 suggests that the iconicity advantage comesfrom referential disambiguation rather than more efficientmemory encoding. Experiments 2 and 3 replicate this result,and moreover show that the kiki-bouba effect is roughlyequally strong for sharp and rounded shapes, a property thatclassic experiments were unable to confirm, and which hasimplication for the effect’s mechanism

Iconicity refers to instances in which the form of language resembles its meaning (Perniss et al., 2010). The most prominent example in spoken language is onomatopoeia (e.g., woosh, which sounds like a gust of wind). Here we tested whether iconicity makes language more vivid by depicting the sensorimotor experiences being referred to. In Experiment 1, 44 participants read ten short passages (five iconic, five non-iconic), and then rated their vividness on several scales. These passages differed on two key words, which were either iconic (e.g., screech) or non-iconic (e.g., yell). We found no evidence that iconic language was more vivid. In Experiment 2, 199 participants each rated one longer passage that was either iconic or non-iconic (differing on eight key words). We found only marginal evidence of iconic language being more vivid, on subscales related to felt vividness. These results suggest that iconicity does not make written language more vivid.

Iconicity refers to a resemblance between word form and meaning. Previous work has shown that iconic words are learned earlier and processed faster. Here we examined whether iconicity would also affect a recognition memory task. We also manipulated the level at which items were encoded—with a focus on either their meaning or their form—in order to gain insight into the mechanism by which iconicity would affect memory. In comparison with non-iconic words, iconic words were associated with a higher false alarm rate, a lower d’ score, and a lower criterion C. We did not observe any interaction between iconicity and encoding condition. We also conducted an analysis of recognition memory megastudy data and found that iconicity was predictive of higher false alarms and a lower criterion C across 1,646 items. We interpret these results as suggesting that iconicity leads to a feeling of familiarity in recognition memory.

A number of recent models of semantics combine linguistic information, derived from text corpora, and visual information, derived from image collections, demonstrating that the resulting multimodal models are better than either of their unimodal counterparts, in accounting for behavioural data. However, first, while linguistic models have been extensively tested for their fit to behavioural semantic ratings, this is not the case for visual models which are also far more limited in their coverage. More broadly, empirical work on semantic processing has shown that emotion also plays an important role especially for abstract concepts, however, models integrating emotion along with linguistic and visual information are lacking. Here, we first improve on visual representations by choosing a visual model that best fit semantic data and extending its coverage. Crucially then, we assess whether adding affective representations (obtained from a neural network model designed to predict emojis from co-occurring text) improves model’s ability to fit semantic similarity/relatedness judgements from a purely linguistic and linguistic-visual model. We find that adding both visual and affective representations improve performance, with visual representations providing an improvement especially for more concrete words and affective representations improving especially fit for more abstract words.

We report a controlled study investigating the effect of visual information (i.e., seeing the speaker) on spoken language comprehension. We compare the ERP signature (N400) associated with each word in audio-only and audio-visual presentations of the same verbal stimuli. We assess the extent to which surprisal measures (which quantify the predictability of words in their lexical context) are generated on the basis of different types of language models (specifically n-gram and Transformer models) predict N400 responses for each word. Our results indicate that cognitive effort differs significantly between multimodal and unimodal settings. In addition, our findings suggest that while Transformer-based models, which have access to a larger lexical context, provide a better fit in the audio-only setting, 2-gram language models are more effective in the multimodal setting. This highlights the significant impact of local lexical context on cognitive processing in a multimodal environment.

Representational gestures are co-speech gestures that carry semantic content related to the content of speech. Previous studies focusing on adult-adult conversation have investigated the temporal alignment of gestures and speech finding that the overwhelming majority of representational gestures are produced right before the lexical content they refer to (their lexical affiliate, LA). However, nothing is yet known about whether caregivers would also time their gestures in the same way in naturalistic interactions. We annotated representational gestures from a large corpus (ECOLANG) of semi-naturalistic conversations between caregivers and their 3-4 year old children (n = 899 gestures from n=36 caregivers). We found that, just as in adult-directed language (ADL), representational gestures in child-directed language (CDL) were more tightly linked to the onset of LAs than the onset of the utterance in which LAs were produced (hence planned when full events are encoded); with the overall majority of the representational gestures starting before their LAs. We further found that age of acquisition (AoA) rating of the LA had a significant effect on the speech-gesture latency. We found that for words acquired earlier, the gesture’s stroke (the meaningful part of a gesture) tended to be produced before the LA’s onset; for the later acquired word, the stroke tended to be produced at the same time or after the onset of the LA in speech. Our findings suggest that: (1) Regardless of their addressee, speakers always time the production of representational gestures to specific conceptual/linguistic units, rather than the full event/utterance. (2) In contrast to ADL, caregivers’ gestures may support addresses’ linguistic processing not only by supporting word prediction (of likely better-known words), but also by supporting the learning of conceptual features (of likely less well-known words).