Natural language processing models based on machine learning (ML-NLP models) have been developed to solve practical problems, such as interpreting an Internet search query. These models are not intended to reflect human language comprehension mechanisms, and the word representations used by ML-NLP models and human brains might therefore be quite different. However, because ML-NLP models are trained with the same kinds of inputs that humans must process, and they must solve many of the same computational problems as the human brain, ML-NLP models and human brains may end up with similar word representations. To distinguish between these hypotheses, we used representational similarity analysis to compare the representational geometry of word representations in two ML-NLP models with the representational geometry of the human brain, as indexed with event-related potentials (ERPs). Participants listened to stories while the electroencephalogram was recorded. We extracted averaged ERPs for each of the 100 words that occurred most frequently in the stories, and we calculated the similarity of the neural response for each pair of words. We compared this 100 × 100 similarity matrix to the 100 × 100 similarity matrix for the word pairs according to two ML-NLP models. We found significant representational similarity between the neural data and each ML-NLP model, beginning within 250 ms of word onset. These results indicate that ML-NLP systems that are designed to solve practical technology problems have a representational geometry that is correlated with that of the human brain, presumably because both are influenced by the structural properties and statistics of language.

Although people with schizophrenia (PSZ) exhibit robust and reliable deficits in working memory (WM) capacity, the neural processes that give rise to this impairment remain poorly understood. One reason for this lack of clarity is that most studies employ a single neural recording modality-each with strengths and weaknesses-with few examples of integrating results across modalities. To address this gap, we conducted a secondary analysis that combined data from an overlapping set of subjects in previously published electroencephalographic and functional magnetic resonance imaging studies that used nearly identical working memory tasks (visual change detection). The prior studies found similar patterns of results for both posterior parietal BOLD activation and suppression of the alpha frequency band within the EEG. Specifically, both signals exhibited abnormally shallow modulation as a function of the amount of information being stored in WM in PSZ. In the present study, both alpha suppression and posterior parietal BOLD activity increased as the number of items stored in WM increased. The magnitude of alpha suppression modulation was correlated with the magnitude of BOLD signal modulation in PSZ, but not in HCS. This finding suggests that the same illness-related biological processes constrain both alpha suppression and BOLD signal modulation as a function of WM storage in PSZ. The complementary strengths of these two techniques may thus combine to advance the identification of the processes underlying WM deficits in PSZ.

Physically salient objects are thought to attract attention in natural scenes. However, research has shown that meaning maps, which capture the spatial distribution of semantically informative scene features, trump physical saliency in predicting the pattern of eye moments in natural scene viewing. Meaning maps even predict the fastest eye movements, suggesting that the brain extracts the spatial distribution of potentially meaningful scene regions very rapidly. To test this hypothesis, we applied representational similarity analysis to ERP data. The ERPs were obtained from human participants (N = 32, male and female) who viewed a series of 50 different natural scenes while performing a modified 1-back task. For each scene, we obtained a physical saliency map from a computational model and a meaning map from crowd-sourced ratings. We then used representational similarity analysis to assess the extent to which the representational geometry of physical saliency maps and meaning maps can predict the representational geometry of the neural response (the ERP scalp distribution) at each moment in time following scene onset. We found that a link between physical saliency and the ERPs emerged first (∼78 ms after stimulus onset), with a link to semantic informativeness emerging soon afterward (∼87 ms after stimulus onset). These findings are in line with previous evidence indicating that saliency is computed rapidly, while also indicating that information related to the spatial distribution of semantically informative scene elements is computed shortly thereafter, early enough to potentially exert an influence on eye movements.SIGNIFICANCE STATEMENT Attention may be attracted by physically salient objects, such as flashing lights, but humans must also be able to direct their attention to meaningful parts of scenes. Understanding how we direct attention to meaningful scene regions will be important for developing treatments for disorders of attention and for designing roadways, cockpits, and computer user interfaces. Information about saliency appears to be extracted rapidly by the brain, but little is known about the mechanisms that determine the locations of meaningful information. To address this gap, we showed people photographs of real-world scenes and measured brain activity. We found that information related to the locations of meaningful scene elements was extracted rapidly, shortly after the emergence of saliency-related information.

Although schizophrenia is classically thought to involve impaired attentional filtering, people with schizophrenia (PSZ) exhibit a more intense and more exclusive attentional focus than healthy control subjects (HCS) in many tasks. To resolve this contradiction, this functional magnetic resonance imaging study tested the impact of attentional control demands on the modulation of stimulus-induced activation in the fusiform face area and parahippocampal place area when participants (43 PSZ and 43 HCS) were looking for a target face versus house. Stimuli were presented individually, or as face-house overlays that challenged attentional control. Responses were slower for house than face stimuli and when prioritizing houses over faces in overlays, suggesting a difference in salience. Blood-oxygen-level-dependent activity reflected poorer attentional selectivity in PSZ than HCS when attentional control was challenged most, that is, when stimuli were overlaid and the task required detecting the lower-salience house target. By contrast, attentional selectivity was exaggerated in PSZ when control was challenged least, that is, when stimuli were presented sequentially and the task required detecting the higher-salience face target. These findings are consistent with 2 distinct attentional abnormalities in schizophrenia leading to impaired and exaggerated selection under different conditions: attentional control deficits, and hyperfocusing once attention has been directed toward a stimulus.

Little is known about the development of higher-level areas of visual cortex during infancy, and even less is known about how the development of visually guided behavior is related to the different levels of the cortical processing hierarchy. As a first step toward filling these gaps, we used representational similarity analysis (RSA) to assess links between gaze patterns and a neural network model that captures key properties of the ventral visual processing stream. We recorded the eye movements of 4- to 12-month-old infants (N = 54) as they viewed photographs of scenes. For each infant, we calculated the similarity of the gaze patterns for each pair of photographs. We also analyzed the images using a convolutional neural network model in which the successive layers correspond approximately to the sequence of areas along the ventral stream. For each layer of the network, we calculated the similarity of the activation patterns for each pair of photographs, which was then compared with the infant gaze data. We found that the network layers corresponding to lower-level areas of visual cortex accounted for gaze patterns better in younger infants than in older infants, whereas the network layers corresponding to higher-level areas of visual cortex accounted for gaze patterns better in older infants than in younger infants. Thus, between 4 and 12 months, gaze becomes increasingly controlled by more abstract, higher-level representations. These results also demonstrate the feasibility of using RSA to link infant gaze behavior to neural network models. A video abstract of this article can be viewed at https://youtu.be/K5mF2Rw98Is.