Limited evidence shows that visual input can facilitate learning novel sound-to-meaning mappings that are crucial to learning a second language. However, the mechanisms by which visual information influences auditory learning are still unclear. Here, we investigate to what extent visual input can lead to effective learning in another domain. We trained atonal speakers with Mandarin tones in 4 conditions: Auditory Only (AO) where only auditory tones were given as input; Animated Contour (AC) where moving visual pitch contours indicating the dynamic changes of tones were given in addition to auditory tones; Static Contour (SC) where static visual pitch contours were given in addition to auditory tones; Incongruent Contour (IC) where mismatched pitch contours were given in addition to auditory tones. The results show the advantage of AC and SC over AO in learning tonal categories and that IC inhibits learning, suggesting that extracting ‘compatible’ properties cross modalities benefits learning most.

Children’s language development is strongly associated with the qualitative variations in their language environment. Here we evaluate how lexical variability differs across input modalities and its impact on word recognition in Chinese-speaking children. Four indices were computed to measure variability in words’ occurrences – type-token ratio, word frequency (WF) – and in words’ contextualized meaning – contextual diversity (CD), semantic diversity (SD) – in child-directed speech, animated cartoons, and picture books. Three models (Cartoon, Book, Mixed) were then built to assess how well the indices predict Grade 2 children’s word reading. Picture books had the highest variability in type-token ratio and WF. Whereas cartoons provide the highest meaningful variations for words (i.e., CD and SD). Children’s word recognition was best explained by the Mixed model, containing multimodal lexical variability indices. The findings reveal cross-modality differences in lexical variability in beginner readers’ language environment and suggested that multimodal language environment impacts children’s word knowledge.