Human real-time communication creates a limitation on the flow of information, which requires the transfer of carefully chosen and concise data in various situations. Although pointing is sparse, overloaded, and indirect, it allows humans to effectively decode shared information, (ex)change their minds, and plan accordingly. I introduce a model that explains how humans choose information for communication and understand communication by utilizing the linguistics concept of ``relevance'' derived from decision-making theory and theory of mind.

The modeling approach taken in this dissertation is inspired by many seemingly separated domains. First, I apply theory of mind from cognitive science and partially observable Markov decision process to formally model the components of human mind and how they make decisions, building a scaffold for modeling human communication. Second, I derive how humans coordinate and share their mind by applying the concepts of paternalistic helping in developmental psychology and philosophical discussion about empathy. Third, I derived the definition of utility-based relevance as how much a signaler's belief can make a positive difference to its receiver's well-being, utilizing the cooperative assumption of human communication in linguistics and comparative psychology. I conducted simulation and human behavioral experiments to show that relevance-based communication model can model the overloaded and indirect human communication and can predict humans' choices of signals in communication. Artificial intelligence agents that communicate with relevance-based models are more well-received by humans. Finally, I use Markov decision process and partially observable Markov decision process to propose a way of finding the best timing for sparse human communication.

Cooperation is a fundamental characteristic of humans that enables complex social interactions and joint achievements beyond the capacity of individuals. Multi-agent reinforcement learning (MARL) is one prevailing approach employed to model such cooperative behavior. While MARL as a generic algorithm serves well for modeling agents’ different social intentions through adjusting the relations between their reward functions, theories from cognitive science suggest that MARL per se does not suffice for modeling human-unique cooperation. During cooperation, humans spontaneously establish a sophisticated framework of shared agency, in which a joint representation of an imagined central agent emerges automatically and owns normative power. We delve into the discrepancies between these two cooperation paradigms through two case studies. Inspired by the theory of shared intentionality in cognitive science, we further introduce the Imagined We framework, a novel approach that emulates human behavior across various tasks requiring joint efforts and communication.

How can a pointing gesture or simple utterance come to mean so much? Unlike a code or fixed mapping that can be used to uniquely identify a referent, humans flexibly use the same signal to mean many different things. As a result, to formally capture the impromptu, sparse nature of communication, it should be viewed as an inferential process. Both sending and understanding a signal in context requires reasoning about the underlying mind at the other end. This dissertation takes a cognitive approach to developing a computational framework for this type of uniquely human communication. Even young children who cannot yet speak in full sentences use simple gestures and utterances in uniquely flexible and intelligent ways. This highlights the promise of a reverse engineering approach: the underlying cognitive mechanisms and commonsense reasoning accumulated during pre-linguistic development become the foundation for modeling intelligent communication.

The modeling approach taken here formalizes this theoretical account by connecting and extending three lines of work which have traditionally been viewed as separate domains. First, I adopt an existing model that draws from game theory and probabilistic inference to formalize flexible signal understanding. Second, I integrate this with socially rational models of individual agency, which involve understanding why individuals act the way they do in terms of their underlying mental states. This follows the tradition of viewing communication as an inference problem, where understanding a signal is about understanding the underlying mind that generated it. Here I argue communication can be viewed in terms of its use: signals are a special type of rational action that can be used to coordinate individuals. This perspective connects modeling flexible signaling to models of intentional agents. Finally, my last step is to shift from individual agency to treating communication as a cooperative, shared agency problem. While shared agency has been a promising approach for coordinating cooperators, it has not yet been modeled in conjunction with communication. This dissertation bridges this gap, leading to the development of a novel framework for communication, called the Imagined We (IW). I justify this cooperative shared agency approach by drawing from a wealth of behavioral evidence in developmental and comparative psychology demonstrating how communication can be viewed as a way to facilitate increasingly sophisticated cooperation. Through a set of simulations in cooperative tasks, I demonstrate theoretical advantages of this perspective. Moreover, I combine this with behavioral evidence that can begin to support some of the theoretical claims this model makes.