Humans can readily generalize their learning to new visual concepts, and infer their associated meanings. How do people align the different conceptual systems learned from different modalities? In the present paper, we examine emojis— pictographs uniquely situated between visual and linguistic modalities—to explore the role of alignment and multimodality in visual and linguistic semantics. Simulation experiments show that relational structures of emojis captured in visual and linguistic conceptual systems can be aligned, and that the ease of alignment increases as the number of emojis increases. We also found that emojis with subjective impressions of high popularity are easier to align between their visual and linguistic representations. A behavioral experiment was conducted to measure similarity patterns between 48 emojis, and to compare human similarity judgments with three models based on visual, semantic and multimodal-joint representations of emojis. We found that the model trained with multimodal data by aligning visual and semantic spaces best accounts for human judgments.

Computational models of verbal analogy and relational similarity judgments can employ different types of vector representations of word meanings (embeddings) generated by machine-learning algorithms. An important question is whether human-like relational processing depends on explicit representations of relations (i.e., representations separable from those of the concepts being related), or whether implicit relation representations suffice. Earlier machine-learning models produced static embeddings for individual words, identical across all contexts. However, more recent Large Language Models (LLMs), which use transformer architectures applied to much larger training corpora, are able to produce contextualized embeddings that have the potential to capture implicit knowledge of semantic relations. Here we compare multiple models based on different types of embeddings to human data concerning judgments of relational similarity and solutions of verbal analogy problems. For two datasets, a model that learns explicit representations of relations, Bayesian Analogy with Relational Transformations (BART), captured human performance more successfully than either a model using static embeddings (Word2vec) or models using contextualized embeddings created by LLMs (BERT, RoBERTa, and GPT-2). These findings support the proposal that human thinking depends on representations that separate relations from the concepts they relate.

Large Language Models (LLMs), which match or exceed human performance on many linguistic tasks, are nonetheless commonly criticized for not “understanding” language. These critiques are hard to evaluate because they conflate “understanding” with reasoning and common sense—abilities that, in human minds, are dissociated from language processing per se. Here, we instead focus on a form of understanding that is tightly linked to language: mapping sentence structure onto an event description of “who did what to whom” (thematic roles). Whereas LLMs can be directly trained to solve to this task, we asked whether they naturally learn to extract such information during their regular, unsupervised training on word prediction. In two experiments, we evaluated sentence representations in two commonly used LLMs—BERT and GPT-2. Experiment 1 tested hidden representations distributed across all hidden units, and found an unexpected pattern: sentence pairs that had opposite (reversed) agent and patient, but shared syntax, were represented as more similar than pairs that shared the same agent and same patient, but differed in syntax. In contrast, human similarity judgments were driven by thematic role assignment. Experiment 2 asked whether thematic role information was localized to a subset of units and/or to attention heads. We found little evidence that this information was available in hidden units (with one exception). However, we found attention heads that reflected thematic roles independent of syntax. Therefore, some components within LLMs capture thematic roles, but such information exerts a much weaker influence on their sentence representations compared to its influence on human judgments.