Our primate Order is known for the expansion of the neocortex relative to other mammals. This distinction is coupled with a characteristic complex society that is facilitated by dynamic social cognitive mechanisms and systems of communication. Because of this intricate relationship, investigating the neural basis of communication within primates affords the opportunity to better understand how different dimensions of sociality are supported by the structures of the brain itself. Much of the research on the neuroscience of communication in primates has hinged on studies of vocalization processing in head-restrained monkeys either passively listening to stimuli or engaged in a conditional behavioral task. But the information communicated by social signals are heavily influenced by the natural contexts they occur in, and auditory processing of vocalizations within the brain may likewise be heavily affected by the context in which conspecific vocalizations are heard; thus, the experiments may not fully capture the neural basis of communication. I hypothesize that the traditional experimental contexts typical of nonhuman primate neuroscience research has divorced the signal from its natural context, and, consequently, limited our understanding of how various neocortical structures support these processes. Here I sought to address this critical gap in our knowledge by implementing novel experimental paradigms designed to explicate the neurobiology and behavior of natural communication in freely-moving marmoset monkeys (Callithrix jacchus). In this dissertation, I detail the results of new insights gained from the innovative experiments that support my hypothesis. Chapter 1 shows how broad ‘states’ of neural populations in frontal cortex during natural, untrained behavior of antiphonal conversations in the marmoset predicts whether subjects respond to a conspecific call. Chapter 2 shows robust within-neuron differences in how prefrontal cortex neurons respond to vocalizations between traditional head-restrained contexts and natural behavior suggesting that data recorded in the former context is not predictive of the latter. Finally, Chapter 3 shows my novel multi-speaker paradigm that simulates the natural communication networks in marmosets (i.e. “Cocktail Party”) to study the vocal processing of marmosets in complex acoustic environments previously inaccessible to researchers for any other animal model. Results demonstrate that marmosets employ similar perceptual mechanisms as humans to communicate in these dynamic acoustic and social landscapes. These findings establish a novel paradigm in which to explore the neurobiology of primate communication in dynamic, multi-speaker communication networks that more closely resemble their natural communication systems.

Complex social behaviors are at the heart of primate societies. Abiding by social rules that govern these behaviors is a critical part of the life of a social primate. In communication, social rules exist to regulate if, when, and what an individual vocalizes, as well as how a receiver responds. These rules are crucial to the efficacy of social communication systems. But what happens when individuals stop following these rules? Here, I investigated the consequences of rule-breaking during communication in common marmosets (Callithrix jacchus). I utilized one of their natural rule-governed behaviors known as antiphonal calling, which involves a back and forth exchange of calls between conspecifics. Rules guiding this vocal behavior include a high level of reciprocity, short response latencies, and turn-taking with minimal interrupting. This dissertation examines how captive marmosets respond to individuals breaking vs. following these rules. I created a novel interactive playback environment where subjects communicated with two marmosets called Virtual Monkeys (VMs). Each VM was a speaker that represented an individual by broadcasting prerecorded calls from a single marmoset. In Chapter 1, I manipulated timing and reciprocity rules by varying how quickly and how likely each VM was to respond when subjects vocalized. I found that subjects were biased against communicating with rule-breakers (i.e., one that responds slowly and infrequently) when a rule-follower (i.e., one that responds quickly and often) was also present. In Chapter 2, I manipulated turn-taking rules by having one VM occasionally interrupt subjects (a rule-breaker) and the second VM behave normally (a rule-follower). I found that subjects were less likely to respond to interruptions, though there was no lasting bias in communication preferences to either VM. In Chapter 3, I sought to establish a foundation for investigating auditory processes in the awake marmoset brain with functional magnetic resonance imaging. Using a battery of auditory stimuli, I found representative properties of the auditory system, including strong activation of auditory cortices, a tonotopic auditory core, and secondary regions driven by complex stimuli. This preparation can one day be used to investigate brain areas involved in processing rule violations during antiphonal calling. Taken together, this dissertation demonstrates that marmosets use flexible strategies during communication and adapt their behavior to different contexts. This likely reflects a broader social cognitive strategy for successfully navigating primate societies and the close relationship between communication and cognition in our Order.