Machine Learning is a powerful tool for both processing and generating data. It has been demonstrated to be more efficient than humans at distinguishing hundreds of different types of images. Such classification and game-based metrics are easily quantifiable, making it simple to demonstrate improvements in efficiency and quality over previous iterations. In contrast, Generative Models (GMs) that create synthetic samples by simulating a distribution are much harder to evaluate cleanly. When looking at images, an approach known as Maximum Mean Discrepancy (MMD) has recently become quite popular, as it can non-parametrically compare samples and assign a similarity score between their relative distributions. A major flaw that MMD has is that there is no accepted approach for defining a score that would deem the generated images as similar enough to real images for them to pass an independent analysis. Introducing humans has been a stopgap solution, but this adds a subjective element to the process workflow. An ideal solution would wholly remove the human element and determine an appropriate MMD-based quality target value using solely the data provided. In this thesis, we present a solution for this situation, as we introduce a novel statistical test that can more accurately compare distributions of data and determine a target score using solely the sample sets. By inspecting the quality of this solution on various models, we can train and analyze models that perform at sufficiently good levels via a fully automated procedure.

Many computer vision problems require image upsampling, where the number of pixels per unit area is increased by inferring values in high-dimensional image space from low-dimensional representations. Recent research has shown that deep learning-based solutions achieve state-of-the-art performance on such tasks by training deep neural networks (DNNs) on large annotated datasets. Yet, their adoption in real-time applications is predicated on the deployment costs of the resulting models since end-user devices impose significant compute and memory constraints on inference pipelines. To address this, many researchers and practitioners have proposed methods to reduce inference costs without sacrificing model quality. However, many of these works focus on DNNs designed for image downsampling. In this thesis, we study inference optimization techniques designed for deep learning-based image upsampling networks. While some inference optimizations are applicable to both upsampling and downsampling networks, we show that specifically tailoring optimizations for image upsampling workloads can lead to more efficient and effective deployment.

We maintain a holistic view of inference optimization, from training through deployment to execution, by integrating hardware-aware deep learning techniques, compute graph transformations, and computer architecture optimizations into an end-to-end pipeline. We begin by characterizing this pipeline and the different requirements for image upsampling and downsampling workloads. We then introduce novel statistical approaches to hardware-aware deep learning techniques based on quantization and pruning. Once trained, we then introduce novel compute kernels and graph transformations that reduce the compute costs of common upsampling workloads by up to a factor of 3.3. Finally, we adapt our novel inference algorithms to a specialized hardware architecture that reduces resource utilization and improves dataflow on FPGA-based accelerators.

We evaluate a wide range of computer vision benchmarks covering both stochastic and deterministic models to show that our approaches improve power efficiency, throughput, and resource utilization without damaging model quality. Our research highlights the importance of end-to-end inference optimization for deep learning-based image upsampling networks and provides an effective solution for reducing the deployment costs of DNNs designed for real-time computer vision applications on resource-constrained platforms.

The electroencephalogram (EEG) provides a non-invasive, minimally restrictive, and relatively low-cost measure of mesoscale brain dynamics with high temporal resolution. Although signals recorded in parallel by multiple, near-adjacent EEG scalp electrode channels are highly correlated and combine signals from many different sources, biological and non-biological, independent component analysis (ICA) has been shown to isolate the various source generator processes underlying those recordings. While ICA-based methods have been seeing more and more use, EEG researchers are hampered by the additional manual intervention necessary for source-resolved analyses. These issues can be largely mitigated through the automation of several stages of EEG source analysis. To this end, we developed and evaluated the ICLabel classifier, an automated independent component classifier trained on a large dataset with crowdsourced labels. The crowdsourced labels were estimated using the novel crowd labeling (CL) algorithm, crowd labeling latent Dirichlet allocation (CL-LDA), developed here. The ICLabel dataset that was used to train the ICLabel classifier was also made public to aid in future development of IC classifiers. We also evaluated artifact subspace reconstruction (ASR), an algorithm for artifact removal which is applicable both offline and in real-time, and aids both channel-level and source-level analyses. These tools are combined in the Real-time EEG Source-mapping toolbox (REST) to showcase the utility and ease of real-time, source-level analyses once the individual components of an EEG analysis pipeline are automated. Finally we evaluate adaptive mixture ICA (AMICA) and explore its utility for automatic EEG segmentation and nonstationary analysis. All of these tools and methods are open-source and freely available online.

Despite several decades of research, most mental health treatments are based on pharmacological manipulations that globally affect the nervous system. Such treatments often lead to undesired side effects and short term symptomatic relief. The difficulty of diagnosing and treating mental health illnesses stems from the overwhelming complexity of the brain and is exacerbated by the fact that our ability to probe, simultaneously, the activity of dynamic and distributed brain networks is limited. In this dissertation, I propose an alternative way to tackle the mental health problem by using high-resolution imaging-based brain-computer interface (BCI) neurotechnology. I focus on new neuroimaging technology that allows us to monitor the electrical activity of cortical networks at low-cost and high spatiotemporal resolution using noninvasive electroencephalographic (EEG) measurements. This technology will serve as the ``neural decoder'' component of yet to come imaging-based closed-loop systems that can effectively restore impaired cognition. The decoder allows a BCIs to dynamically probe specific cognitive abilities of the subject in search for signatures of circuit dysfunctions. Then, various types of feedback can be designed to induce the engagement of neural populations that can compensate for the detected aberrant neuronal activity.

In this dissertation, first, I develop the mathematical framework to efficiently map scalp EEG responses back into the cortical space, and by doing so, I show that the biological mechanisms responsible for the neurocognitive processes of interest are easy to study. Of theoretical and practical relevance, I demonstrate that this framework successfully unifies three of the most common problems in EEG analysis: data cleaning, source separation, and imaging. Then, I develop the algorithmic and software machinery necessary to implement high-resolution imaging-based BCIs. Finally, I analyze data from healthy adults performing a self-paced unconstrained schoolwork-like computerized task and show that within the proposed framework, I can identify brain network correlates of attention switches at a millisecond time scale. Since attention-related dysfunctions are linked to several psychiatric disorders, these results represent a step forward towards developing BCI interventions to treat several mental health illnesses.

In recent years deep learning algorithms have shown extremely high performance on machine learning tasks such as image classification and speech recognition. In support of such applications, various FPGA accelerator architectures have been proposed for convolutional neural networks (CNNs) that enable high performance for classification tasks at lower power than CPU and GPU processors. However, to date, there has been little research on the use of FPGA implementations of deconvolutional neural networks (DCNNs). DCNNs, also known as generative CNNs, encode high-dimensional probability distributions and have been widely used for computer vision applications such as scene completion, scene segmentation, image creation, image denoising, and super-resolution imaging. We propose an FPGA architecture for deconvolutional networks built around an accelerator which effectively handles the complex memory access patterns needed to perform strided deconvolutions, and that supports convolution as well. We also develop a three-step design optimization method that systematically exploits statistical analysis, design space exploration and VLSI optimization. To verify our FPGA deconvolutional accelerator design methodology we train DCNNs offline on two representative datasets using the generative adversarial network method (GAN) run on Tensorflow, and then map these DCNNs to an FPGA DCNN-plus-accelerator implementation to perform generative inference on a Xilinx Zynq-7000 FPGA. Our DCNN implementation achieves a peak performance density of 0.012 GOPs/DSP.

We propose an extension of the mixture of factor (or independent component) analyzers model to include strongly super-gaussian mixture source densities. This allows greater economy in representation of densities with (multiple) peaked modes or heavy tails than using several Gaussians to represent these features. We derive an EM algorithm to find the maximum likelihood estimate of the model, and show that it converges globally to a local optimum of the actual non-gaussian mixture model without needing any approximations. This extends considerably the class of source densities that can be used in exact estimation, and shows that in a sense super-gaussian densities are as natural as Gaussian densities. We also derive an adaptive Generalized Gaussian algorithm that learns the shape parameters of Generalized Gaussian mixture components. Experiments verify the validity of the algorithm.

With the recent growth in data owing to ubiquitous internet connectivity, practical problems involving prediction which abound in multiple fields such as healthcare, finance, climate analysis, image processing and other disciplines are now becoming solvable. Traditionally such problems have been addressed by Model-Based approaches; i.e. by employing assumptions about the data generating process (DGP) to construct a model which can then used for predicting future values. However it is well known that such methods may suffer from disadvantages such as model non-robustness, overfitting among others. In this thesis along with Model-Based approaches the alternative paradigm of Model-Free Prediction is explored for problems such as regression, time series and random fields. The Model-Free framework does not assume knowledge of the DGP and is applied directly to the available data to construct estimators for both point prediction and prediction intervals. Both Model-Based and Model-Free prediction methods are applied to synthetic and real-life data

and their prediction performances are compared using standard metrics to demonstrate the applicability and usefulness of both approaches.

A new era of processing has dawned: the demands for low latency and low power processing at the edge have ushered in unprecedented opportunity computer architects and embedded designers. In pursuit of new performance standards, chip designers in industry and academia have begun the march towards domain specific processors, a paradigm whose core philosophy and methods are in many ways contrary to the mantras that dominate the processors seen in today's datacenters and technology hubs. The increasing complexity of neural networks and deep learning algorithms being deployed at these edge locations has made this pursuit anything but trivial. Some of the most powerful models that we are seeing deployed, known as deep generative models, use techniques that are effectively capable of generating new data by capturing the full joint data distribution over some input space. These models frequently use upsampling layers to take lower dimensional latent spaces to higher dimensional ones before making inferences about our world.In this work, we perform a deep analysis of one of these upsampling techniques, known as deconvolution (or equivalently transpose convolution), and propose a novel computer architecture for low latency acceleration in edge applications. Our work is the first to fuse together systolic processing and an algorithmic transformation known in this area as the TDC method \cite{chang2018optimizing}. We illustrate how and why this pairing is so powerful for inference acceleration and provide some preliminary performance numbers benchmarked against a pre-existing Wasserstein Generative Adversarial Network (GAN).

Electroencephalography (EEG) is a fundamental technique for studying and understanding the brain through its electrical oscillations. Within this field, the phenomena where the phase of slow brain rhythm modulates a faster rhythm is known as phase-amplitude coupling (PAC). The study of this phenomenon has been gaining attention due to its occurrence in normal and pathological brain processes in humans and across different mammalian species. In the quest to understand PAC's physiological role in the brain, the development of signal processing methods and software supporting its study is key for further advancement. Several methods have been proposed to estimate a measure of the PAC process, but only a few have addressed its temporal dynamics. Furthermore, it appears that no studies have addressed PAC dynamics while assuming its possible directional delay characteristics. \par

This dissertation's goal is twofold: to provide new methods to estimate and characterize PAC dynamics, and to provide dedicated software with high-performance computing (HPC) support for computing PAC in electrophysiological signals from the brain. \par

Here I first propose and validate a mutual information-based method for computing time-resolved PAC, MIPAC. Then, I characterize and lay out the foundations for using a novel directed information theory measure, transfer entropy, to study delayed interactions from the phase to amplitude components in the PAC process. Following, I propose a new EEGLAB plug-in toolbox, PACTools, that allows computing PAC in either continuous or event-related EEG data using multiple currently available methods, including the newly developed MIPAC. PACTools operate seamlessly with EEGLAB, and features built-in access to free HPC resources at the Neurosciences Gateway (\textit{nsgportal.org)}. This feature is possible by developing another EEGLAB plug-in I'm introducing here, nsgportal, that interfaces EEGLAB with NSG directly from within EEGLAB on any personal computer. The tools and methods presented in this dissertation constitute a comprehensive framework aiming at facilitating the study of the PAC's role in the healthy and pathological brain.