Context is an important part of understanding the meaning of natural language, but most neuroimaging studies of meaning use isolated words and isolated sentences with little context. Because the brain may process natural language differently from how it processes simplified stimuli, it is unclear whether prior results on word meaning generalize to natural language. In Chapter 1, I present a neuroimaging experiment that examines whether the results of neuroimaging studies that use stimuli with little context generalize to natural language. Results show that context both affects the quality of neuroimaging data and changes where and how semantic information is represented in the brain. This suggests that findings from studies using stimuli with little context do not generalize to natural language.

In Chapters 2 and 3, I present two neuroimaging experiments that map representations of social information in the brain. Relationships are an integral part of life, and people store extensive knowledge about themselves, other individuals, and their dynamics to maintain these relationships. Many prior neuroimaging studies have investigated where different types of social information are represented in the brain. However, because of methodological limitations in these studies, the representation of social information in the brain remains unclear. In Chapter 2, I present a neuroimaging experiment that simultaneously maps the representation of five types of social information that have been investigated in prior studies. Results show that only three of these five types of social information are represented in the brain, and that individual brain regions each represent one type of social information. In Chapter 3, I present a neuroimaging experiment that maps the representation of the self and six different types of other people. Preliminary results from this experiment suggest that the brain represents the self and different types of others in distinct brain regions. These data also reveal three possible axes along which the brain may organize information about the self and others.

Navigation in the natural world is a complex task that engages many cognitive systems, including vision, attention, motor control, cognitive maps, and planning. These systems recruit many brain regions that form multiple functional networks spanning the brain. Navigation has been the subject of many non-human neurophysiology studies, primarily in rodents, and less frequently in non-human primates.In the recent decades, neuroimaging using functional magnetic resonance imaging has enabled non-invasive brain activity recordings. Neuroimaging studies are able to examine cognitive processes in healthy human subjects. Together, neurophysiology and neuroimaging studies have revealed multiple regions in the human brain that are active during navigation. However, we still know very little about how these regions accomplish the complex task of navigation, and what information is represented by each region. This dissertation describes a next-generation neuroimaging experiment that maps navigational representations across the cortex to gain insight into the complex processes underlying human navigation. Chapter 1 reviews the current neurophysiology and neuroimaging literature on the navigation system of the brain. These current literatures suggest that the brain breaks navigation down into a set of subtasks, each with its associated brain regions. This chapter also describes several limitations in the current understanding of navigation in the human brain. Chapter 2 describes a next-generation, naturalistic neuroimaging paradigm to study human navigation. This paradigm leverages Unreal Engine 4, a modern game engine, to create a realistic and dynamic virtual world. A custom MR-compatible steering wheel and pedal set enables subjects to drive naturally in this virtual world. Chapter 3 describes an active navigation neuroimaging experiment that uses the paradigm developed in Chapter 2. Subjects actively perform a taxi driver task in a virtual world while BOLD activity is recorded from the brain. Voxelwise modelling with banded ridge regression is used to map the cortical representation of over 20,000 features across 33 feature spaces. Chapter 4 examines task-related visual semantic tuning shifts. Visual semantic tunings in the active navigation task are compared with those from a passive movie watching task. Results show that there are significant visual semantic tuning shifts between active navigation and passive movie watching. Chapter 5 describes a route progression model derived from the rodent literature. Results suggest that a topologic representation of route progression in RSC is conserved across species. Furthermore, RSC, unlike other visual navigation areas, strongly prefers the start of routes. Finally, Chapter 6 describes a novel, state-space based method for analyzing high-dimensional brain data. This method treats the activity of the brain as a dynamical system, and finds a low-dimensional subspace in the brain’s activity space that is related to the representation of task variables. As a proof-of-concept, this method is applied to a visual semantic attention task and a video game task, and recovers low-dimensional spaces related to the representation of attentional targets and game states.

Humans are constantly taking in information from a multitude of dissimilar sources to integrate and update their existing mental representations, but the series of computations that underlie this process are still largely unknown. We do know that these complex processes recruit multiple brain networks that interface with each other, but the precise roles that these networks play can change depending on task demands, and the full extent of this is only partially understood. This research presented in this dissertation focuses on furthering this knowledge specifically in the domain of semantic representation in humans.

In this dissertation, I first summarize the existing literature on semantic representations in the human brain. In addition to this, I describe how careful computational modeling methods alongside naturalistic experimental conditions allow us to answer very precise questions about representations in the human brain. I then present two functional magnetic resonance imaging (fMRI) experiments that explore new avenues of research in semantic representation and further show the strength of these methods. The first of these experiments illustrates that the visual and linguistic semantic networks of the brain are more precisely aligned than had previously been hypothesized. This pattern would not have been discovered using most typical fMRI analysis methods. In the second experiment, I analyze how linguistic semantic representations update after learning and memorization. Specifically, after participants have spent many hours memorizing songs, we can see that their semantic representations of the content of the song lyrics can shift dramatically.

Together, these two experiments show the importance of studying semantic representations not only in a single modality or setting. Without looking at the relationships between different types of representations or how representations are updated with learning and experience, we will never have a full picture of how the brain is functioning in complex, naturalistic environments.

The goal of the human sensory system is to build a useful internal representation of the world. In vision, this means that the brain categorizes and identifies all the objects and actions that are being observed. In language, it means that the brain understands the meaning of each word that is being perceived, and integrates across words to understand the meaning of a narrative. Both of these processes can be seen as extracting the meaning, or semantic content, from a dense stream of sensory information. We know very little about how the brain accomplishes these feats. Indeed, entire fields of research (computer vision and natural language processing) are devoted to reproducing on a computer feats of understanding that the human brain accomplishes with ease. We may be able to gain insight into these processes by studying how the semantic information extracted from visual and linguistic stimuli is represented in the brain.

This dissertation describes three functional magnetic resonance imaging (fMRI) experiments that have helped to reveal how visual and linguistic semantic information are represented across the human cerebral cortex. These experiments relied on a relatively new fMRI analysis methodology known as voxel-wise modeling (VM). Although this methodology was developed for modeling how the brain represents the structure of visual information, it was adapted here for modeling representation of the extremely complex semantic information present in natural movies and natural narrative stories.

The first experiment (Chapter 2) showed that information about object and action categories present in natural movies is represented in a low-dimensional semantic space that is shared across subjects. Projecting this semantic space across the cortex revealed that semantic information is represented in broad cortical gradients that cover a surprising amount of the cortical surface. The second experiment (Chapter 3) showed that information about the semantic content of narrative spoken stories is also represented in a low-dimensional space that is shared across subjects. To model the complex cortical maps for this semantic space a new technique was developed called PrAGMATiC (Probabilistic And Generative Model of Areas Tiling the Cortex). This new technique revealed that the cortical semantic maps can be explained by about 230 functional areas that cover much of the prefrontal cortex, temporoparietal junction, precuneus, and temporal cortex. Finally, the third experiment (Chapter 4) showed that a novel hierarchical logistic regression (HLR) model could accurately decode the categories of objects and actions present in natural movies from fMRI responses.

The work presented in this thesis is toward the goal of extracting structure and meaning from neuroscientific data. Data in visual neuroscience is becoming increasingly high dimensional and the stimulus-response relationships can be highly nonlinear. Data in visual neuroscience is also somewhat noisy due to the imprecise separation of signals from multiple neurons on an electrode, nonstationary effects in the brain, and inherent noise in the brain; neurons rarely respond identically to identical stimuli. Finding nonlinear relationships between a high dimensional stimulus and neural responses in the presence of substantial noise is a challenging nonlinear regression problem. This thesis presents effective techniques for solving this problem and creating highly predictive models of neural function. I first introduce linearized regression, a technique for modeling nonlinear responses using linear regression on a nonlinear transformation of the stimulus. Next I demonstrate a method for efficiently finding Volterra series representations of nonlinear neural responses. Finally, I demonstrate that deep neural networks can provide accurate and interpretable models of the neural computations in visual cortex.

Gradient-echo blood oxygen level dependent functional magnetic resonance imaging (BOLD fMRI) is one of the most powerful and widely used tools for studying neural processing in the human brain. However, BOLD fMRI is limited to measuring neural activity indirectly through blood oxygen concentration and is heavily biased toward large draining veins. Although this large vein bias is often ignored when interpreting results of BOLD fMRI experiments, it can lead to inaccurate estimates of the strength, location, and cortical extent of underlying neural activity. In this dissertation, novel methods utilizing the entire complex-valued BOLD signal are proposed. These methods provide robust localization of large veins and quantification of large vein contributions to studies using gradient-echo BOLD fMRI.

The BOLD fMRI signal is complex-valued consisting of both magnitude and phase components. Although most experiments using BOLD fMRI only analyze the magnitude component, the phase component exhibits BOLD activity exclusively from large veins. Thus the phase component can either be used to enhance or suppress BOLD activity from large veins. The work here demonstrates that by using the phase component to enhance magnitude component BOLD activity, functional signal-to-noise ratio (fSNR) is improved equivalent to collecting 115% more data. This improvement is expected to increase at higher field strengths and spatial resolutions. By using the phase component to suppress large vein contributions, it is found that current understanding of semantic category representation may actually reflect vascular differences rather than neural differences across cortex. By suppressing large vein contributions using the methods proposed here, gradient-echo BOLD fMRI can more closely reflect underlying neural activity.

A primary focus of neuroscience is understanding how information about the world is encoded in the activity of neurons. The classic approach to studying neural representation is to observe how activity in a neural system is modulated when presented with stimuli that vary along a specific, hypothesis-driven dimension (e.g. luminance, contrast, orientation, spatial frequency). This approach offers straight-forward interpretation and has allowed neuroscientists to develop computational models of the stimulus-response function for neurons in early processing stages of the visual system. However, the interpretations of these studies often do not generalize to other stimuli such as natural images, and the corresponding computational models often fail to explain later processing stages where the stimulus-response function is strikingly nonlinear. What is needed is an alternative approach that can a) inform new hypotheses without relying on the intuitions of the experimenter, b) generate tractable and interpretable nonlinear models of neural function based on these hypotheses, and c) directly assess model accuracy at predicting neural function under natural conditions. In this dissertation I propose a simple but powerful framework that uses statistical feature learning to automatically derive potential forms of neural representation and generates predictive and testable models of neural function based on these potential representations. The framework is general and can potentially be applied to data acquired using any physiological measurement technique (e.g. neurophysiology, neuroimaging, behavioral) in order to model neural representation throughout the brain. I demonstrate the potential of the proposed framework by first modeling the neural representation of image structure in early cortical stages of the macaque visual system (areas V1 and V2). I then apply the framework to investigate high-level scene processing in the human visual system. In both case studies, I am able to objectively derive accurate and interpretable nonlinear models of neural representation. In each study, I am also able to replicate and extend findings from a multitude of classic neurophysiology and neuroimaging studies using a single, simple experiment based on natural visual stimuli.

fMRI is a proven technique for recording brain activity. Although it measures a signal that is coupled with neural firing through a complex chain of events, it is the only method by which we can non-invasively observe an entire healthy brain at millimeter scale resolution. The technique of fMRI is now fairly well known, yet there are still many gaps in the tools we use to analyze it and our understanding of its underpinnings. Indeed, the complexity and depth of the data is the source of many of its shortcomings.

In this dissertation, I detail a set of tools I developed to make better use of fMRI data. In chapter 2, I describe a tool that I created to visualize fMRI data. It provides an intuitive and powerful interface to explore and share your data with others. In chapter 3, I created a device that significantly improves the performance of fMRI by virtually eliminating head motion. This plastic insert relies on commodity 3D printing technology to create personalized helmets that address many of the shortfalls of fast scan sequences. Finally, in chapter 4, I utilize these fast scan sequences to investigate the temporal response of hemodynamics. I show that BOLD timecourses have very little high frequency power, but the high dimensional nature of fMRI can be used to perform classification extremely quickly.