How language and music are processed by the biological connection in our heads is one of the most significant challenges to neuroscience. A critical aspect of this challenge is that the signals are necessarily sequential and the brain must process events at multiple hierarchical levels simultaneously. Aspects of this are not unique to humans, however, but shared with many animals who rely on vocal communication. In this work, we focus on characterizing the capacity of European starlings to discriminate between sequences and understanding how individual neurons and populations of neurons support sequence discrimination. We train starlings on novel behavioral protocols wherein subjects must discriminate between probabilistically generated sequences composed of vocal elements using only the sequential relationships between elements. When faced with uncertainty, subjects discriminate sequences in a way which is consistent with weighing the evidence afforded by the sequence. By recording the spiking activity of neurons in anesthetized subjects, we found that neurons in the caudomedial nidopallium (NCM) are sensitive to sequences and that their capacity to encode sequences is higher than their capacity to encode elements. Neuronal responses to sequences are shaped by a combination of reward-association and behavioral demands. Recordings of population activity in awake behaving subjects revealed that neurons in caudolateral mesopallium (CLM) exhibit mixed selectivity for elements and position, supporting robust population representations of element identity. Element encoding fidelity in CLM is state-dependent—accurate decoding of element identity from either fast-spiking or regular-spiking populations improves the reliability and encoding fidelity of the complementary population. These results emphasize the importance of behavioral goals in understanding how sequences are processed by neuronal populations.

Relating the collective activities of neural populations to external sensory stimuli or to motor output is essential to understanding how nervous systems support behavior. Equally important is examining these processes in the context of ethological signals, which are typically high-dimensional with a wide range of co-varying features at multiple timescales. The synaptic and network mechanisms for encoding these complex signals are largely unknown, due in part to limitations inherent in experimental design practices leveraging dimensionality reduction rather than embracing the full suite of stimulus complexity. Additional limitations arise because most studies to date examine spiking statistics in randomly-selected populations without consideration for the functional relevance of sub-network groupings. We do not know how spiking responses of individual neurons are pooled, functionally, by downstream neurons – a mechanism that could significantly alter population coding. Stimulus and behavior likely modulate the functional selection of sub-populations, which then likely exhibit different spiking statistics than the population at large.

Using a combination of intracellular and extracellular electrophysiology techniques in the caudal mesopallium (CM) and the caudal nidopallium (NCM) of the common European Starling, I examine synaptic and spiking activity driven by previously-recorded conspecific vocalizations, maintaining the full ethologically-relevant complexity contained in each signal. Uniquely, I feature the synaptic response as an independent variable in the examination of population spiking activity. The main implication of the collective results presented in this study is that, rather than a model of hierarchical processing in which stimulus-specific information is restricted to parallel circuits within each region, sensory integration and processing are supported by a system in which information about even the most complex stimuli is likely massively redundant and shared among the population at large. This scaffolds a sensory processing model in which flexible network re-organization - on short timescales and in a stimulus-specific way - support the complexities of spiking output observed in sensory cortex in response to learning, adaptation, attention, and context to meet the demands of an ever-changing environment.

From the noisy information bombarding our senses, our brains must construct percepts that are veridical – reflecting the true state of the world – and informative – conveying what we did not already know. Influential theories suggest that both challenges are met through mechanisms that use expectations about the likely state of the world to shape perception. Predictive coding, a theoretical framework in which the brain compares a generative model to incoming sensory signals, seeks to explain this inferential process of perception. Implementation of predictive coding at a mechanistic level in individual neurons is however not well understood. This dissertation builds on the field of computational modeling of sensory systems to further our understanding of perceptual inference in individual neurons in the auditory system of songbirds (European starling). Chapter 1 summarizes some possible ways in which predictive coding could be realized at the algorithmic level. The main contribution of this chapter is not to present models that can uniquely account for sensory data, but to reconcile different cognitive behaviors under the umbrella of predictive coding which were previously considered incompatible. Chapter 2 combines computational modeling with electrophysiological activity to study internal representations of sensory information under predictive coding framework in single auditory neurons in songbirds. This study examines unique components of the response variance guided by predictive coding. It is shown that during song listening, individual neuron responses in primary and secondary auditory regions are modeled by expectations of future song and uncertainty of song. Generative machine learning approaches enable us to make hypotheses about internal generative models for complex natural stimuli. Chapter 3 explores the flexibility of the internal model and whether prediction error is correlated with behavior. This study draws primarily upon the complex structure of birdsong, developing methods to analyze the acoustic and temporal structure in vocal signals and then behaviorally and physiologically probing the underpinnings of the predictive structure of birdsong. The findings in this chapter indicate that prediction error encoding is modulated by task-relevant generative models and behavior. Taken together, my dissertation work offers a unified explanation of inferential frameworks of perception in contexts of systems processing behaviorally relevant signals, showcasing its adaptation to these systems.

Our interactions with the world unfold over time. Whether it's speaking, where one word follows the next, or walking, where each step follows another, the organization of our behaviors in time tends to follow a predictable pattern. Those patterns are dictated by a multitude of underlying factors, influenced both by endogenous physiological factors like the rhythmic nature of our gait as well as by exogenous factors, like the social dynamics underlying turn-taking while speaking. Despite decades of research studying the temporal organization of behavior, dating back to the work of influential biologists like Tinbergen, Lashley, and Dawkins, little is known about the physiological substrates that underlie either the production of the sequential organization of most aspects of behavior. Despite widespread acknowledgment that physiological motor programs and many non-linguistic behaviors are hierarchical, for example, few physiological investigations into the dynamics of behavior extend beyond low-order (Markovian) transition statistics.

In this thesis, I build onto the emerging field of computational neuroethology to further our understanding of what structure underlies the sequential organization of behavior, what physiological mechanisms might be involved in producing, perceiving, and representing sequential behavioral organization, and how sequential behavioral organization might have emerged developmentally and evolutionarily. Throughout the thesis, I draw primarily upon birdsong and human speech, developing methods to analyze the acoustic and temporal structure in vocal signals and then behaviorally and physiologically probing the underpinnings of sequential organization in the songbird. This work advances the field of computational neuroethology in several ways.I uncover novel acoustic structure in vocal signals separating avian and mammalian vocalizations along a spectrum of vocal stereotypy. I observe that both human speech and birdsong are characterized by a combination of long and short-range temporal patterning. I find that the long-range temporal patterning characterizing human speech, believed to be underlied by hierarchical linguistic organization, is present at the earliest developmental stages of human speech, well before complex syntax is produced. I find that the perceptual integration of birdsong syllable sequences can be well explained by Bayesian models of probabilistic perceptual decision-making. Finally, I find that sensory neural representations of syllable sequences are modulated by sequential context and that this modulation reflects the animals underlying perceptual behavior. In the following paragraphs, I give a brief overview of the methods and major results of the chapters comprising this thesis.

In Chapter \ref{chapter:review} I give an introduction to the emerging field of vocal computational neuroethology. This introduction contextualizes the following chapters in a review of current work. I emphasize current tools, challenges, and future directions in vocal neuroethology. I start with a discussion of low-level bioacoustics challenges and build up to a discussion of behavioral organization and physiology. I first discuss challenges in signal processing such as dealing with noise and signals and representing vocal signals as time-frequency representations. I then discuss machine learning approaches used to identify, segment, and label vocalizations. Next, I discuss how to extract relational structure between vocalizations, and cluster latent projections of vocalizations. I then give an overview of methods for capturing temporal relationships in vocal sequences, outlining traditional Markovian descriptions of vocal structure, and new tools for capturing long-range structure, enabled by large datasets. I then move on to machine learning tools that can be used to systematically control and synthesize vocal signals from learned vocal spaces. Finally, I discuss how these techniques are being utilized in several active areas of neuroethology research.

In Chapter \ref{chapter:avgn} I develop a set of methods to visualize and quantify relational structure in vocalizations, which enable the analyses and experiments performed in the following chapters. I use graph-based dimensionality reduction to uncover local structure in vocal communication signals and apply that technique to 19 datasets consisting of vocalizations from 29 species, including songbirds, primates, cetaceans, rodents, and bats. I observe that these methods uncover novel structure in animal vocal signals, including vocal dialects, acoustic units, behaviorally relevant signal information, and sub-syllabic structure.

In Chapter \ref{chapter:parametric_umap}, I extend the methods from Chapter \ref{chapter:avgn} by introducing Parametric UMAP, a graph-based dimensionality reduction algorithm that parametrically learns the relationship between data (here vocal signals) and latent embeddings. Parametric UMAP enables the methods from Chapter \ref{chapter:avgn} to be applied in real-time closed-looped settings over larger datasets due to the learned parametric embeddings. I show that this algorithm has applications in semi-supervised settings, and provides additional control over the trade-off between capturing global and local structure in embeddings.

In Chapter \ref{chapter:parallels} I explore the long and short-range temporal patterning of vocal sequences in birdsong and human speech. I use an information-theoretic framework to analyze statistical dependencies as a function of the distance between elements in vocal sequences. I find that both birdsong and human speech exhibit two forms of structure: short-range relationships captured by Markovian dynamics over short-timescales, and long-range relationships that follow a power-law occurring over longer timescales. In language, the observed short-range organization conforms to phonological processes, which are well-described by finite-state dynamics, while long-range organization suggests more complex dynamics such as underlying hierarchical organization. Previous analyses of birdsong have only identified short-range Markovian dynamics, making our observation of long-range dynamics in birdsong novel.

In Chapter \ref{chapter:lri} I extend our experiment from chapter \ref{chapter:parallels} over human speech to language acquisition. By analyzing corpora of speech throughout language development, we can observe the time course of the emergence of long and short-range relationships over development. Surprisingly, I find that long-range statistical dependencies are present in children's speech as early as 6-12 months, well before complex syntactic structure is present. I discuss these results alongside emerging evidence from computational ethology that long-range relationships are also common to non-linguistic behavioral signals from animals as diverse as zebrafish, drosophila, and whales. Although previous analyses of long-range relationships have suggested that long-range relationships are the product of hierarchical linguistic structure such as syntax and discourse structure, our observations in developmental speech and non-linguistic behaviors suggest that other mechanisms may also be at play.

Finally, in Chapter \ref{chapter:cdcp} I probe how sequential dependencies in vocal sequences are integrated behaviorally and physiologically. I developed a behavioral task in which European starlings are trained to classify morphs of syllables of starling song synthesized from an interpolation between two points in the latent space of a neural network (a Variational Autoencoder). These morph syllables are preceded with a separate syllable (a cue syllable), which holds predictive information about the category of the following morph syllable. I find that classification of the morph syllable is contextually modulated by the predictive probability of the cue syllable, which can be well explained by a model of Bayesian integration. With the same behavioral paradigm, I then record chronic electrophysiology data from auditory nuclei while birds performed this context-dependent categorical perceptual decision-making task. I find that neural activity patterns reflect several aspects of our model of perceptual behavior, including the uncertainty in decision making, and prediction-related perceptual modulation.

Characterization of response properties of neurons in higher-level sensory areas is not well defined. Here we show that firing rates of neurons in a secondary sensory forebrain area of songbirds can be modeled by different representations of birdsong. In this work, we modeled neurons in the caudo-medial nidopallium (NCM) of adult European starlings with three different representations of the natural birdsong called signal, prediction, and error. Prediction spectrogram was computed by training the data as a Gaussian distribution on a loss function given by the negative log likelihood, and then estimating the means and variances of the signal. Using our Maximum Noise Entropy (MNE) model, responses were predicted by the logistic function, the parameters of which are obtained from the MNE model. Predictions of neural responses were computed by using both a full MNE model, and then by only considering the linear parameters of the model. The neural responses to natural stimuli obtained using prediction and error MNEs were close to the actual response in the NCM. The concept of stimulus representations obtained from predictive coding models may be useful for modeling neural responses in higher-order sensory areas whose functions have been poorly understood.

Dolphins use an advanced biosonar system to accurately navigate and forage in their environment. They have the unique capability to rapidly detect and classify underwater targets in a complex acoustic environment and do so more precisely than man-made underwater sonars. Dolphins use the fine-scale temporal and spectral features of the echo-waveforms to determine different attributes of the target and rely on echo-delay, the time between their emitted sound pulse “click” and the return of the reflected echo to determine the range to a target. In this dissertation, phantom echoes are used to determine the effect that changes in mean echo-delay, i.e. range, have on echo-delay discrimination abilities and which fine-scale acoustic features dolphins use to create a coherent mental representation of a target. In Chapter 1, the dolphin’s ability to detect changes in echo-delay as a function of range was determined. Chapter 2 then tested the dolphin’s ability to discriminate between the ranges of two targets at different mean ranges. In this experiment targets were presented successively, meaning, the dolphin had to hold the range of one target in memory while they compared it to the second target. Chapters 1 and 2 conclude that as mean range increases greater than 10 m the dolphin’s ability to discriminate between echo-delays degrades rapidly. Chapter 3 explores the possibility of the dolphin using spectral cues to determine the difference in range between two targets that are presented simultaneously. First, the dolphin’s peripheral auditory system was modeled to determine available spectral cues. The dolphin’s ability to discriminate the difference in range between two targets was then tested and compared under successive and simultaneous conditions. Although spectral cues were available to the dolphin, the results from the simultaneous condition suggest limited improvement when compared to the successive condition. Lastly, Chapter 4 focuses on the auditory perceived cues the dolphin may use to convert fine-scale spectral and temporal information into a coherent mental representation. By manipulating echo-phase information the results suggest that for fine-scale echo-delays the dolphins may use a pitch cue to discriminate between complex targets.

Speech is a highly-structured auditory signal that carries necessary and sufficient information for language comprehension among neurotypical, hearing populations. Speech perception, accordingly, involves the integration of both bottom-up auditory processing and top-down language knowledge. As explored in Chapter 1, however, while much is known about the brain's sensitivity to various acoustic and phonological manipulations, mechanistic transformative links between auditory sensation and linguistic perception remain poorly understood.

This dissertation contributes three studies to the ongoing pursuit of links between auditory and linguistic processing of speech in the brain. Intracranial stereo electroencephalography (sEEG) was recorded while participants listened to conversational English speech, and for each study, different sets of analyses were performed to target different aspects of these links.

Chapter 2 introduces a novel comparative technique for the isolation of phonological and morphological identity from the auditory speech signal. Through structured comparisons of the neural response to sounds in allophonic relationships with one another, sites that dissociate phonemic and morphological identity from acoustic similarity were identified. These results provide crucial evidence that the neural response to speech is sensitive to abstract sub-lexical structure.

Chapter 3 provides general models to account for the unique contributions of categorical phonemic information and spectrographic information to the neural response. In lower frequency bands, phonemic category information explained a greater proportion of the neural response variance than spectrographic information. These results indicate that the explanatory value of phonemic category features in neural encoding models is not reducible to the surface spectrotemporal features that correlate with those categories.

Chapter 4 proposes a model for the relationship between the speech stimulus and neural response that integrates both phonemic and spectrographic information. Inclusion of the stimulus covariance structure increased model accuracy only when categorical phonemic information was available, indicating that phoneme-related neural activity is likely mediated through the acoustic covariance structure of speech. Moreover, phonemic label information conferred no benefit to model fit when participants listened to speech in an unfamiliar language (Catalan).

Together these studies show that neural responses associated with categorical phonemic information are language specific and irreducible to speech acoustics, and they scout a path towards mechanistic, transformative links between auditory sensation and language cognition.

Perception depends upon the coordinated activity of populations of neurons. Howneural populations represent structure in the outside world through their activity is an important open question. One approach to this problem is the concept of the receptive field, which quantifies how external stimulation modulates the activity of individual neurons. While receptive fields are a powerful concept for the experimenter, the brain itself does not have access to its own receptive fields. This dissertation applies methods from the field of algebraic topology to characterize the structure of neural population activity in the secondary auditory region NCM of the European starling. Chapter 2 demonstrates that the simplicial complex associated to population activity in NCM carries behaviorally-relevant information xiiabout learned categories. Along the way, a new similarity measure for population activity is developed, called the Simplicial Laplacian Spectral Entropy. It is shown that this measure quantifies the similarity of simplicial complexes associated with neural activity in a way that depends upon their global topological structure. Chapter 3 explores the connection between neural topology and classical receptive fields, by showing that the intrinsic geometry of the population activity matches the geometry of the receptive fields. This shows that the temporal coativity structure of the population contains a direct representation of the stimulus structure without requiring explicit computation of receptive fields. This validates a previously described theoretical mechanism in a sensory system for the first time in-vivo, and provides a new understanding for population activity in sensory regions. This chapter also introduces a technique for reconstructing acoustic spectrograms of complex, natural vocal signals from neural activity, which will be a powerful technique for exploring population level representations in future studies. The final chapter of this dissertation describes a new mathematical description of the phenomenon of polychronization in spiking neural networks. This serves as a bridge between the experimental results of this dissertation and a theoretical understanding of the emergence of spatiotemporal structure in the activity of neural populations.