Research in reading comprehension traditionally relied on the experimental setting of word-by-word presentation which eventually revealed many neural biomarkers as well as establishing the basis of modern reading research. Since the development of brain-computer techniques and computational methods in the past decades, it has become possible to study reading comprehension in natural settings. This study used a variety of advanced technologies to analyze a dataset collected by the ZuCo group regarding reading comprehension. With natural language processing tools, we extracted the words essential for understanding sentences, and identified eye-tracking patterns that relate to these words. Using the EEG time series and frequency series, we also looked at neural patterns associated with those words and tried to build up a statistical model and neural network model that predicted the linguistic patterns. Consequently, the study is likely to provide new insights into future cognitive linguistics and brain-computer interaction research, which may help advance reading-aid technologies.

Recognition and continuous monitoring of human emotions is a key problem that is spread across multiple research disciplines like electrical engineering, computer science, neuroscience, and cognitive science. Instead of the widely used method of approaching this problem from the perspective of computer vision i.e. by tracking user's facial expressions, we take a multi-modal approach. This approach utilizes various bio-sensing modalities in addition to computer vision for assessing and classifying human affects. In the process, we address various limitations—hardware and software—of such bio-sensing modalities. We evaluate the developed framework on real-world applications ranging from watching emotional multimedia content (such as videos) to driving an automobile. Our holistic framework contributes toward (1) the development of a novel multi-modal bio-sensing headset that is capable of recording, monitoring, and tracking various bio-signals in real-time, (2) the development of algorithms that utilize signal processing as well as deep learning tools for various bio-sensor and vision-based modalities for emotion recognition, (3) the utilization of such algorithms in a wide range of applications namely emotion classification, studying emotion elicitation, and monitoring driver's awareness, and (4) the performance comparison of various sensor modalities for the above applications when they are used individually or in fusion with each other. Although the above multi-modal sensor platform has been evaluated in this thesis dissertation on affective computing applications only, it is generalizable. Thus, this platform can work in various applications in the field of human-computer interaction that requires the use of bio-sensing modalities.

Neural interfaces enable interaction with the brain through direct recordings of neural activity. For individuals suffering from conditions limiting their ability to interact with and experience the world, solutions leveraging neural interfaces restore function and improve quality of life. Recent advances in brain-machine interfaces have enabled applications of such technology to achieve functional restoration in subjects with conditions spanning anarthria and tetraplegia. While these solutions begin to restore lost function, neural decoders require further improvement to match fully-able human performance.

Building robust neural decoders generally involves leveraging volitional responses, commonly motor intent, to train and deploy a model. Importantly, once a model is deployed, it is only updated in limited contexts (e.g. calibration). This can lead to loss of utility over time due to neural, measurement, and environmental sources. Using augmentative non-volitional responses, meaningful context can be provided to controllers online for adaptation and retraining to improve decoder robustness and usability.

In this work, we explore the full neural decoder development process. We first address challenges spanning clinical and laboratory data collection using a novel flexible experiment framework. We then critically review deep feature transforms and their utility in neural decoders. We next extend LFADS to intracranial human recordings and evaluate its application learning neural dynamics of reward. Finally, we seek to improve the algorithmic exploration and development process by demonstrating a pathway for scaling novel algorithms and improving disseminability. Through this work, we work towards further enabling high-performance neural prostheses.

A brain-computer interface (BCI) allows human to communicate with a computer by thoughts. Recent advances in brain decoding have shown the capability of BCIs in monitoring physiological and cognitive state of the brain, including drowsiness. Since drowsy driving has been an urgent issue in vehicle safety that causes numerous deaths and injuries, BCIs based on non-invasive electroencephalogram (EEG) are developed to monitor drivers’ drowsiness continuously and instantaneously. Nonetheless, on the pathway of transitioning laboratory-oriented BCI into real-world applications, there are major challenges that limit the usability and convenience for drowsiness detection (DD). To completely understand the association between human EEG and drowsiness, this study employed a large-scale dataset collected from simulated driving experiments with a lane-keeping task and EEG recordings. A DD-BCI that acquires EEG from only non-hair-bearing (NHB) areas was proposed to maximize the comfort and convenience. The performance of the NHB DD-BCI was validated and compared with that using whole-scalp EEG, showing no significant difference in the accuracy of alert/drowsy classification. In addition, a subject-transfer framework that leverages large-scale existing data from other subjects was proposed to reduce the calibration time of a DD-BCI. Alert baseline data were involved to enhance the efficiency of subject-to-subject model transfer. The subject-transfer approach significantly reduced the calibration time of the DD-BCI, exhibiting the potential in facilitating plug-and-play brain decoding for real-world BCI applications. Overall, this thesis presents the contributions to developing a DD-BCI for real-world use with maximal usability and convenience. The methodologies and findings could further catalyze the exploration of real-world BCIs in more applications.

Stress has been a prevalent part of modern life, particularly during the time of the pandemic. While short-term stress may cause little harm to productivity, if left untreated for a long period of time, it could eventually lead to anxiety and depression, which significantly decrease the quality of life. Such a problem is even more severe among students. A recent survey conducted in 2020 with 15,346 graduate and professional students had shown that 32% of them screened positive for major depressive disorder. This study, which took place in a real-world classroom, aims to uncover some of the neuronal mechanisms behind stress among young adults with their recorded EEG data. Such understanding could provide the theoretical foundation for stress reduction and prevention techniques such as real-time stress detection and non-invasive neurostimulation. This thesis is structured into four chapters. In the first chapter, the author introduced the importance of the problem, the experiment design, and the dataset. In the next chapter, the author began the stress analysis by studying the power spectral density of the 30 EEG electrodes. Results showed that most theta (4-8Hz) and alpha (8-13Hz) frequency bands in the frontal, central, and right parietal regions showed statistical significance among the elevated and normal stress groups. While the power spectra information is helpful for understanding stress, it is important to remember that EEG signals are mixtures of source activities, which makes the underlying source activities and locations unknown. Thus, in chapter three, the author decomposed the EEG data into independent sources using Independent Component Analysis (ICA) and analyzed the effect of stress in terms of cortical source activities. The results showed that some sources responded to stress while others did not. One limitation of this study was that each source was analyzed individually. Thus, in the final chapter, the author focused on exploring the interactions between regions (i.e. effective connectivity) under stress. The results showed that the information inflow and outflow near the central region were statistically different between the elevated stress and normal stress groups.

Cognitive deficits associated with neurological disorders such as schizophrenia (SCZ) and mild cognitive impairment (MCI) present a significant challenge to both individuals and healthcare systems. Despite advancements in medical technology, treatment options for cognitive symptoms or conditions like MCI and Alzheimer’s disease (AD) remain limited, leaving over 23 million people in the United States affected by early-stage cognitive decline without effective interventions. For SCZ and MCI patients, working memory (WM) deficit is a common symptom, linked to impaired frontal gamma activity for both patient groups. Gamma oscillations, particularly in the frontal cortex, are essential for cognitive functions such as attention and working memory, making them a critical target for therapeutic intervention. Yet, restoring or modulating these oscillations through conventional methods remains challenging, emphasizing the need for novel approaches to enhance cognitive function in these populations.This dissertation explores clinical gamma neuromodulation as a promising therapeutic approach, focusing on the mechanisms of gamma activity modulation, identifying neurophysiological predictors of effective training, and applying brain-computer interface (BCI) technology to facilitate this modulation in SCZ and MCI patients. Through EEG-based neurofeedback (NFB), patients are trained to increase frontal gamma coherence, a neural signature associated with cognitive performance. In SCZ patients, a 12-week BCI-based gamma-NFB training protocol resulted in significant improvements in frontal gamma coherence and enhanced EEG markers of working memory, such as increased frontal P3 amplitude. This novel BCI system, developed using MATLAB/EEGLAB, converts real-time gamma coherence into reinforcement signals, enabling patients to modulate their brain activity directly. The double-blind, placebo-controlled randomized clinical trial (RCT) of SCZ patients demonstrated that those undergoing active NFB training achieved significant increases in F3-F4 gamma coherence and WM, demonstrating the potential of BCI technology to offer an adaptive, non-invasive intervention for cognitive deficits. Similarly, in MCI patients, a small sub-sample from the ongoing placebo-controlled RCT showed that active gamma-NFB training significantly increased frontal gamma coherence compared to a placebo group, with baseline gamma power at electrode F4 predicting the degree of training-related improvement. These findings suggest that gamma-NFB can enhance cognitive function in MCI patients, providing a tailored and scalable therapeutic option. By identifying neurophysiological predictors of training success and demonstrating the effectiveness of BCI-driven neuromodulation in clinical populations, this dissertation contributes to developing personalized, brain feature targeted interventions for cognitive impairments. These findings support the potential of gamma neuromodulation as a viable therapeutic approach for improving memory and cognitive function in both SCZ and MCI patients, paving the way for future research into long-term benefits and broader clinical applications.

The availability of affordable and portable electroencephalogram (EEG) devices has sparked interest in using passive EEG-based brain-computer interfaces (BCIs) in real-world applications such as neuroergonomics and neuromarketing. These fields require objective measurement of human cognitive and affective states. Although studies have explored EEG features for different mental states and affective responses in these areas, there is still a gap between laboratory research and real-world implementation.

Two critical questions need to be addressed to bridge the gaps between laboratory research and real-world implementation. Firstly, can the EEG features identified in controlled laboratory conditions be reliably detected in real-world settings? Secondly, how can transfer learning streamline the calibration process for new users or sessions of passive BCI features? Can laboratory-oriented tasks be employed to calibrate the model for real-world applications?

This dissertation aims to address the questions raised earlier by developing EEG signal-processing and feature-extraction methods, and exploring transfer learning techniques for assessing human cognitive and affective states in naturalistic environments. Chapter 2 describes a study demonstrating how EEG can be used in neuroergonomic research to monitor changes in an individual's memory workload during a regular office task. Chapter 3 presents a study on affective states, examining how EEG and eye-tracking can detect human interest levels in images of electronic products. These two chapters prove that robust EEG features found in laboratory settings can also be observed in real-world settings.

Chapter 4 investigates the transferability of EEG features in monitoring human cognitive loads. The study's outcomes can inform the development of transfer learning techniques for more effective BCI applications in real-world settings. Chapter 5 demonstrates the feasibility of cross-task transfer learning for passive BCIs and illustrates how EEG signatures from lab-controlled tasks can be applied to real-world scenarios. Finally, Chapter 6 concludes all the studies.

Overall, this dissertation offers valuable contributions to the EEG-based assessment of human cognitive and affective states in real-world settings and has significant implications for developing more practical and effective passive BCI applications.