Code summarization and generation are valuable tasks to master for their wide range ofapplications in code readability and code translation to name a few. This research work is an extension of previously conducted research on the use of PLBART, a sequence-tosequence transformer model used for a variety of program and language understanding and generation (PLUG) tasks. The ultimate goal is to improve the performance of PLBART by modifying the noise function of it’s denoising autoencoder. The current noise function corrupts code tokens randomly, but we hope to improve performance by masking nodes on the corresponding Abstract Syntax Tree (AST) instead.To integrate the AST structure into the self-attention mechanism, we adopt the dependency-guided self-attention mechanism explored in NLP literature in particular [ZKC21]. However, from the AST, we cannot compute distances between all tokens that appear in a code since they need not necessarily appear in the parse tree structure. So, we investigate how we can derive distances between tokens from the AST structure.

To act and survive in an environment, humans and other organisms need to form useful representations of it. Such representations are formed through co-ordinated transformations across areas (or layers), and often change through developmental experience. We find internal representations in trained deep neural networks capture the key features of multi-area neural recordings during a perceptual decision-making task, where minimal sufficient representations of sensory information emerge along a cortical hierarchy. Using our models, we show that these minimal sufficient representations emerge through preferential propagation of task-relevant information between areas.\n To better understand how such representations emerge through learning, we introduce a notion of usable information, and use it to show that a noisy learning process (e.g. Stochastic Gradient Descent) plays an important role in forming these minimal sufficient representations. We find that the learning process is highly nonlinear: semantically meaningful information is initially encoded in the representation, even if it is not needed for the task. Additionally, we show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early stages of learning. In particular we find, using analytical models and through simulations, that depth and competition between sources has a significant effect on critical learning periods observed in biological and artificial networks. \n We further study how multisensory information can be decomposed, and develop novel approximations to compute the redundant information shared between a set of sources about a target, and show that the common information shared between a set of sources can be used to guide the learning of meaningful representations.

Brain-computer interfaces (BCIs) offer a promising avenue for restoring movement and communication to individuals with paralysis. While invasive BCIs achieve high-performance control, they require neurosurgery, posing significant risks. In contrast, non-invasive BCIs eliminate surgical risks but suffer from low neural signal fidelity, often resulting in frustratingly poor usability. Addressing this challenge, this thesis explores the development of shared autonomy for non-invasive BCIs augmented with artificial intelligence (AI) to enhance control performance while maintaining user autonomy.

To improve signal decoding in non-invasive BCIs, I introduce a novel adaptive decoding framework that combines deep learning with shared autonomy. This approach enables more accurate neural signal interpretation, facilitating improved control in both 2D and 3D control tasks. AI copilots that infer user intentions and provide selective assistance, a concept rooted in shared autonomy (SA), are integrated into the decoding pipeline for EEG based BCIs. Unlike traditional SA frameworks where AI intervention requires hyper-parameter tuning and can be fixed, potentially overriding user intent, this thesis introduces interventional assistance (IA) as an improved framework for shared control. This approach dynamically determines when AI intervention is beneficial by comparing the expected action value of the human user against that of the AI copilot in a goal-agnostic fashion.

Building on these foundations, I present a diffusion-based AI copilot, termed intervention diffusion assistance (IDA), which simultaneously improves task performance and user autonomy. Through both simulations and human-in-the-loop experiments, IDA demonstrates superior performance compared to conventional SA methods. Notably, human participants report increased autonomy and a preference for IDA, highlighting its ability to balance assistance and control.

By merging non-invasive BCI technology with intelligent and adaptive shared autonomy mechanisms, this thesis advances the feasibility of AI-assisted BCIs for clinical applications. These developments pave the way for non-invasive BCIs that offer both high-performance control and intuitive user experience, bringing the field closer to practical, real-world deployment.

Surface electromyography (sEMG) has been commonly used to register the activation of the peripheral nervous system and as part of non-invasive neural interfaces. More specifically, s-EMG has gained attention due to the increasing use of AI-powered human-computer interfaces (HCI) requiring hand gesture recognition (HGR).These interfaces have a range of applications, including the control of extended reality, agile prosthetics, and exoskeletons. However, the natural variability of sEMG among individuals and across multiple days has led researchers to focus on subject-specific and single-day solutions. In this dissertation, we address the robustness challenges of sEMG-based HGR, meaning that we propose and develop HGR systems that can be easily calibrated to be applicable to multiple subjects and across multiple days. We start by considering the subject generalization of sEMG-based HGR and show that by leveraging a common concept in natural language processing (NLP), namely "Embeddings", we can design a HGR system that can be efficiently and accurately calibrated to be used for a new subject with minimal required data. Next, we move on to solve the multiday generalization challenge of sEMG-based inference and use the concept of "Transformers" to facilitate system calibration across multiple days. Inspired by the superior performance of embeddings and transformers for HGR, we extend these models to address the finger kinematics tracking problem. In other words, we re-purpose the Vision Transformer architecture to forecast finger joint angles from 100 ms up to 1.5 seconds ahead of time. We show that our model significantly outperforms state of the art in terms of evaluation metrics and also visual quality of tracking. In this work we also study the theoretical properties of estimation of embeddings in high-dimensions. I.e. , we consider estimating factors of a low-rank matrix when the matrix is observed through a noisy channel with possibly unknown biases. We use a class of approximation algorithms called approximate message passing (AMP) to quantify the accuracy of the estimation in certain high-dimensional limits and show the relations of key parameters of the model. Finally, inspired by the applications of embeddings in sEMG-base inference and noting the data availability challenges of such problems, we analyze the problem of Few Shot Learning, where a few samples of data are used to calibrate a model to a downstream task. We propose a two step estimation framework to apply to such problems and we provide rigorous theoretical analysis for a bilinear model. We show that our proposed framework significantly outperforms a regular baseline training scheme when sufficient data is not available. For future directions of research we propose to investigate the following: (1) addressing the generalization of the joint angles tracking problem to new subjects and across multiple days. (2) Extending our theoretical results to address not only linear and bilinear cases, but also models that include deep neural networks as their components.

The P300 speller is a common brain–computer interface (BCI) application designed to allow patients with neuromuscular disorders such as amyotrophic lateral sclerosis (ALS) produce text output through the detection of P300 signals in their electroencephalogram (EEG) signals. The standard P300 speller relies on the detection of signals evoked by visual stimuli, usually consisting of rows and columns highlighted in a grid of characters. Since the visual field is substantially larger than these stimuli, adjacent flashes to the attend characters may cause false positive signals and lead to erroneous output. While previous work has tried to address this issue by limiting the number of adjacent stimuli, no attempts have been made to account for these adjacency false positives in the classifier or utilize information from adjacent flashes to optimize the system. In this study we added a bias to the target character detection based on adjacent flashes and created a new probability model to improve the accuracy and speed of classification. We tested our adjacency classifier in both the standard P300 paradigm (we call it SWLDA method in the following content) and in conjunction with natural language processing. The new algorithm was evaluated offline on a dataset of 69 healthy subjects, which showed increases in speed and accuracy when compared to standard classification methods. On population level, LDA classifier shows a significant improvement, but the improvement for NLP is not significant. However, for some subject in both algorithms the enhancement of information transfer rate is substantial. As for LDA classifier, 57.4% subjects’ peak ITR increases of more than 5 bits per minute, 30.9% subjects’ peak ITR increases more than 10 bits per minutes, and 8.8% subjects’ peak ITR increases more than 15 bits per minutes. As for NLP, of the subjects, 21.7% had peak ITR increases of more than 5 bits per minute, 10.1% subjects’ peak ITR increased more than 10 bits per minutes, and 5.8% subjects’ peak ITR increased more than 15 bits per minutes. Therefore, incorporating adjacent fleshes can potentially provide a better communication system for some users.

Intracortical brain-computer interfaces (BCIs) are expensive and time-consuming to design because accurate evaluation traditionally requires real-time experiments. In a BCI system, a user interacts with an imperfect decoder and continuously changes motor commands in response to unexpected decoded movements. Decoder optimization using previously collected ``offline'' data will therefore not capture this closed-loop response.However, this “closed-loop” nature of BCI leads to emergent interactions between the user and decoder that are challenging to model. The gold standard for BCI evaluation is therefore real-time experiments, which significantly limits the speed and community of BCI research. My dissertation is focused on creating a pure software BCI simulator which is accurate and versatile in evaluating and characterizing performance of never-before-tested decoders. In our approach, there are three components: the decoder, neural activity, and user control policy. Our design approach uses (1) published BCI experiments as ground truth decoder comparisons, collected with empirical neural recordings and a user-in-the-loop, (2) physical BCI emulator experiments allows us to optimize the neural encoder in isolation to reproduce published ground truth decoder performance, and finally (3) software BCI simulator experiments to implement approximate control policies for each decoder, replacing the user-in-the-loop. We demonstrate that our simulator is accurate and versatile, reproducing the published results of three distinct types of BCI decoders: (1) a state-of-the-art linear decoder (FIT-KF), (2) a “two-stage” BCI decoder requiring closed-loop decoder adaptation (ReFIT-KF), and (3) a nonlinear recurrent neural network decoder (FORCE). We anticipate this simulator will help democratize and significantly accelerate BCI research.

Graph representation learning serves as the core of many important tasks on graphs, ranging from friendship recommendation, name disambiguation, drug discovery, and fraud detection. Recently, deep learning has revolutionized various domains such as computer vision, natural language processing, speech recognition, etc. Inspired by the success of deep neural networks, there has been an increasing interest to learn graph representations with deep learning models such as autoencoders, convolutional neural networks, etc. However, graphs in real-life applications usually have complex structures such as sparse connections, task-irrelevant information, and rapidly evolving structures. The complexity poses great challenges to the existing frameworks, such as network embedding models with random walks and graph neural networks based on the neighborhood aggregation.

In this dissertation, we propose several deep learning frameworks to tackle the aforementioned problems of graph representation learning on complex graphs. We propose a novel model to learn network representations with adversarially regularized autoencoders to overcome the sparse sampling issue of random walks on graphs. To resolve the task-irrelevant noise, we propose a general framework that is trained to simultaneously select task-relevant edges and learn graph representations by the feedback signals from downstream tasks. To learn from the dynamic evolving graphs, we propose to extract local features by performing convolutions in nodes' neighborhoods defined in joint temporal-structural space. The methodologies presented in these frameworks span different research areas, including deep network embedding, graph representation learning, temporal graph modeling, and node classification on graphs. As a result, these methodologies not only tackle specific challenges in the graph learning tasks mentioned above but also shed light on other applications like social network analysis.

Traditional input methods to interface with computer systems prove to be challenging for individuals with amputations or paralysis. Although several brain-machine interfaces were developed to address this problem, their invasive nature prevents widespread adoption. Alternatively, developing interfaces using non-invasive signals has been shown to be effective but they require large, non-intuitive gestures to function. In this work, we propose a framework to decode the subtle finger movements that occur naturally during typing via analyzing non-invasive EMG signals. Here, we establish synchronized communication with an amplifier to get signal recordings, perform signal preprocessing and utilize deep learning architectures for feature extraction and classification. Our approach achieves a within-session accuracy of up to 89.23% in detecting individual finger movements during a randomized typing task, with an average accuracy of 77.64% across all sessions. The time needed for classification is 4.16 ms per sample, making our framework suitable for real-time operation. Our framework demonstrates the possibility of identifying finger movements during typing in real-time using non-invasive EMG signals and provides a starting point for future work to allow individuals with amputations or disabilities to communicate effectively with computers.

Naturalistic escape requires versatile context-specific flight with rapid evaluation of local geometry to identify and use efficient escape routes. It is unknown how spatial navigation and escape circuits are recruited to produce context-specific flight. Using mice, we show activity in cholecystokinin-expressing hypothalamic dorsal premammillary cells (PMd-cck) is sufficient and necessary for context-specific escape that adapts to each environment’s layout. Contrastingly, numerous other nuclei implicated in flight only induced stereotyped panic-related escape. We reasoned the PMd can induce context-specific escape because it projects to both escape and spatial navigation nuclei. Indeed, activity in PMd-cck projections to thalamic spatial navigation circuits are only necessary for context-specific escape induced by moderate threats, but not panic-related stereotyped escape caused by perceived asphyxiation. Conversely, the PMd projection to the escape-inducing dorsal periaqueductal gray projection is necessary for all escapes tested. Thus, PMd-cck controls versatile flight, engaging spatial navigation and escape circuits.

It is additionally unknown if a single circuit controls escape vigor from innate and conditioned threats. We further demonstrate that PMd-cck cells are activated during escape, but not other defensive behaviors. PMd-cck ensemble activity can also predict future escape. Furthermore, PMd inhibition decreases escape speed from both innate and conditioned threats. Inhibition of the PMd-cck projection to the dlPAG also decreased escape speed. Intriguingly, PMd- cck and dlPAG activity in mice showed higher mutual information during exposure to innate and conditioned threats. In parallel, human fMRI data show that a posterior hypothalamic-to-dlPAG pathway increased activity during exposure to aversive images, indicating that a similar pathway may possibly have a related role in humans. Our data identify the PMd-dlPAG circuit as a central node, controlling escape vigor elicited by both innate and conditioned threats.

Distressing reactions to perceived threats in safe environments is a hallmark feature of anxiety-related disorders such as post-traumatic stress disorder (PTSD) and generalized anxiety disorder (GAD). The development of novel targeted neural circuit therapies requires an understanding of how neurophysiological activity relates to behavior and subjective emotional experience. To determine electrophysiological signatures related to emotionally negative and clinically relevant states, we characterize intracranial oscillatory changes within the amygdala and other fear circuitry nodes (including the hippocampus and ventromedial Prefrontal Cortex), during a range of aversive experiences in humans with and without treatment-resistant post-traumatic stress disorder (TR-PTSD) in support of clinical trial (NCT04152993). Additionally, we present pilot neural and behavioral data during a novel virtual reality fear conditioning task compatible with simultaneous intracranial electroencephalography and fear behavior recordings in naturalistically behaving humans. We found increases in amygdala theta activity during unpleasant portions of six separate paradigms (emotional image task, fear conditioning task, script driven imagery task, virtual reality fear conditioning task, and at home recordings during symptomatic TR-PTSD states). Subsequent use of elevations in low-frequency amygdala bandpower as a trigger for closed-loop neuromodulation led to significant reductions in TR-PTSD symptoms and changes in amygdala theta activity following one year of treatment. Altogether, our findings connect decades of rodent literature to clinical symptoms and intracranial neurophysiology in humans and demonstrate the viability of responsive neurostimulation in TR-PTSD.