Humans can readily generalize their learning to new visual concepts, and infer their associated meanings. How do people align the different conceptual systems learned from different modalities? In the present paper, we examine emojis— pictographs uniquely situated between visual and linguistic modalities—to explore the role of alignment and multimodality in visual and linguistic semantics. Simulation experiments show that relational structures of emojis captured in visual and linguistic conceptual systems can be aligned, and that the ease of alignment increases as the number of emojis increases. We also found that emojis with subjective impressions of high popularity are easier to align between their visual and linguistic representations. A behavioral experiment was conducted to measure similarity patterns between 48 emojis, and to compare human similarity judgments with three models based on visual, semantic and multimodal-joint representations of emojis. We found that the model trained with multimodal data by aligning visual and semantic spaces best accounts for human judgments.

Resolving overloading in communication requires attention to context. Previous research has found that the mutual assumption of cooperation during communication can act as a powerful constraint, allowing successful resolution under ambiguity. In this study, we investigate two specific types of cooperative context used in a communicative task which arise from different sources: beliefs and actions. In belief-driven communication, signals are interpreted in context of what else a speaker could have said about the world. Here communicators assume that the speaker aims to change the listener's beliefs by providing the most straightforward signal. In action-driven communication, signals are considered in terms of what a speaker can reasonably ask others to do. Signaling can be sensitive to utility considerations of acting and interacting in the physical world. Through a communication game, we tested how listeners would interpret an ambiguous signal using belief-driven or action-driven strategies. We find that while no one strategy is dominant overall, individuals are highly consistent in which strategy they employ when forced to decide.

Communication is highly overloaded. Despite this, even young children are good at leveraging context to understand ambiguous signals. We propose a computational account of overloaded signaling from a shared agency perspective which we call the Imagined We for Communication. Under this framework, communication is a way for cooperators to coordinate their perspectives, allowing them to act together to achieve shared goals. We assume agents are rational, utility maximizing cooperators, which puts constraints on how signals can be sent and interpreted. We implement this model in a set of simulations which demonstrate this model’s success under increasing ambiguity as well as increasing layers of reasoning. Our model is capable of improving performance with deeper recursive reasoning; however, it outperforms comparison baselines at even the shallowest level of reasoning, highlighting how shared knowledge and cooperative logic can do much of the heavy-lifting in language.