We consider data clustering problems where partial grouping is known a priori. We formulate such biased grouping problems as a constrained optimization problem, where structural properties of the data define the goodness of a grouping and partial grouping cues define the feasibility of a grouping. We enforce grouping smoothness and fairness on labeled data points so that sparse partial grouping information can be effectively propagated to the unlabeled data. Considering the normalized cuts criterion in particular, our formulation leads to a constrained eigenvalue problem. By generalizing the Rayleigh-Ritz theorem to projected matrices, we find the global optimum in the relaxed continuous domain by eigendecomposition, from which a near-global optimum to the discrete labeling problem can be obtained effectively. We apply our method to real image segmentation problems, where partial grouping priors can often be derived based on a crude spatial attentional map that binds places with common salient features or focuses on expected object locations. We demonstrate not only that it is possible to integrate both image structures and priors in a single grouping process, but also that objects can be segregated from the background without specific object knowledge.

Magnetic resonance imaging (MRI) has the ability to produce a series of images that each have different visual contrast between tissues, allowing clinicians to qualitatively assess pathologies that may be visible in one contrast-weighted image but not others. Unfortunately, these standard contrast-weighted images do not contain quantitative values, producing challenges for post-processing, assessment, and longitudinal studies. MR fingerprinting is a recent technique that produces quantitative tissue maps from a single pseudorandom acquisition, but it relies on computationally heavy nearest neighbor algorithms to solve the associated nonlinear inverse problem. In this dissertation, we present our deep learning methods to speed up quantitative MR fingerprinting and synthesize the standard contrast-weighted images directly from the same MR fingerprinting scan.

Adapting deep learning methodologies to MR image synthesis presents two specific challenges: 1) complex-valued data and 2) the presence of noise while undersampling.

MRI signals are inherently complex-valued, as they are measurements of rotating magnetization within the body. However, modern neural networks are not designed to support complex values. As an example, the pervasive ReLU activation function is undefined for complex numbers. This limitation curtails the impact of deep learning for complex data applications, such as MRI, radio frequency modulation identification, and target recognition in synthetic-aperture radar images. In this dissertation, we discuss the motivation for complex-valued networks, the changes that we have made to implement complex backpropagation, and our new complex cardioid activation function that made it possible to outperform real-valued networks for MR fingerprinting image synthesis.

In Fourier-based medical imaging, undersampling results in an underdetermined system, in which a linear reconstruction will exhibit artifacts. Another consequence is lower signal-to-noise ratio (SNR) because of fewer acquired measurements. The coupled effects of low SNR and underdetermined system during reconstruction makes it difficult to model the signal and analyze image reconstruction algorithms. We demonstrate that neural networks trained only with a Gaussian noise model fail to process in vivo MR fingerprinting data, while our proposed empirical noise model allows neural networks to successfully synthesize quantitative images. Additionally, to better understand the impact of noise on undersampled imaging systems, we present an image quality prediction process that reconstructs fully sampled, fully determined data with noise added to simulate the SNR loss induced by a given undersampling pattern. The resulting prediction image empirically shows the effects of noise in undersampled image reconstruction without any effect from an underdetermined system, allowing MR pulse sequence and reconstruction developers to determine if low SNR, rather than the underdetermined system, is the limiting factor for a successful reconstruction.

This dissertation is an exploration of data-driven discovery, inspired by neuroscientific studies of the brain. Each of the three projects listed will describe a different domain of input data (self-driving video, medical image, and biological audio), and how investigating neural network behavior trained on that data can reveal insights for each underlying task. Chapter 2 assess motion selectivity in a self-driving network trained to predict two output tasks: steering and motor. We show how different control conditions can define temporal behavior, as well as how frame order is only implicitly learned if relevant for the task, even if the frame order is present in the input and output training data. Chapter 3 assesses self-supervisedly learned representations from retinal fundus images. We show how these learned representations can drive a voting scheme classifier to match supervised and human expert baselines for disease severity prediction in this field, minimizing the bias enforced from clinically relevant ground truth labels. These representations can be further probed to discover mislabeled or easily confused data, as well as phenotype groupings in retinal images that pertain to other pathology and physiology of the subject. These imply NPID and cluster analysis tools could aid clinicians organize and label data from multiple tasks, an expensive process that requires uncommon expertise. Similarly, Chapter 4 extends this idea about data-driven learning to the audio domain. Here, self-supervisedly learned representations from zebra finch data yielded feature encodings that were functionally relevant for classifying vocalization calls, driving a voting scheme classifier to match supervised baseline performance on a generally difficult task of intra-species audio discrimination. We convert audio waveforms to spectrogram image representations of sound signals, and train a CNN on these inputs, so we can probe these visually-defined audio features. To do so, we assessed how neuronal behavioral preferences can be described by a mid-level representation space of audio (the modulation power spectrum), as well as how these features compare to mid-level audio features correlated with zebra finch brain activity. Data-driven algorithms can learn representations with minimal bias, so commonalities between artificial and biological neural systems imply similar encodings are optimally learned. All in all, this dissertation has evaluated deep learning applied on a host of real world tasks aside from standard datasets curated for computer vision. Though each project requires a different lens for explaining functionally salient behavior, we offer data-driven insights into each underlying task that seem to be consistent with experimental findings in neuroscience and medicine.

Making a computer system understand complex image scenes is challenging. Complex imagescenes often have multiple objects, which are not isolated but related to each other in different aspects. Identifying certain object categories may not be enough to understand complex scenes. Categories have multiple granularities. We need such knowledge to capture semantic correlation thoroughly. In addition, objects have numerous interactions/relationships. We need to localize these objects, recognize scene environments, and figure out their interaction- s/relationships. In computer vision, recognizing what the categories are, where the objects are, and how objects interact to each other is often formulated as the classification, segmentation, and relationship recognition problem.

Existing approaches often tackle all these formulations in supervised settings. Despite theirtremendous progress, we identify three major limitations. 1) Human annotation is too time- and labor-consuming to scale up to real-world scenarios. 2) The sets of human labels are pre-selected arbitrarily, providing limited/biased perspectives to understand images. 3) Such supervised methods conduct inference in terms of discrete labeling. They isolate labels from each other, ignoring the similarity/dissimilarity among each other. Also, they can only put images to the known labels seen during training and fail to recognize novel images sampled from unknown labels during testing.

In this dissertation, we address the issues of current supervised approaches by replacingdiscrete labeling with grouping and using minimal human labels. Specifically, we tackle the recognition problem from four perspectives. 1) We address weakly-supervised semantic segmentation, where partial semantic pixel labels are used. 2) We address unsupervised semantic segmentation, where only low-level edge detections are used. 3) We address unsupervised concurrent image classification and segmentation in a single framework, where our model does not use any human labels. 4) We address unsupervised human-object recognition, where semantic and instance pixels labels, no relationship labels, are used. This dissertation explores more general and robust approaches to understanding the highly-complex and fast-changing real-world scene.

Deep learning has attracted much attention from the ecological community for its capability of extracting and generalizing patterns from data sets with highly complicated structures, such as images, audios, and motion signals. However, despite the promising cases, deep learning is complicated in terms of application and has shortcomings when applied to real-world ecological data sets. In this dissertation, we focus on: 1) demystifying the hidden mechanisms of deep learning in terms of wildlife recognition, 2) identifying the challenges of deep learning applications in wildlife recognition, and 3) proposing a generic recognition framework that can be practically deployed in the fields.

In the first chapter, we examine how deep learning recognizes wildlife through Convolutional Neural Network feature deconstruction and interpretation. The objective is to demystify aspects of artificial intelligence and facilitate wildlife recognition research.

The second chapter identifies three major challenges to automatic wildlife recognition through an avian recognition case study and provides preliminary solutions addressing each challenge. This chapter aims to increase awareness in the ecological community of these challenges, bridge the gap between ecological applications and state-of-the-art computer science, and open doors to future research.

In the third chapter, we propose a hybrid recognition system of machine learning and human-in-the-loop that overcomes two challenges discussed in the second chapter: imbalanced data distribution and continuous data expansion. Moreover, with the self-updating mechanism of our approach, the system can be practically deployed in the fields.

Purpose

Methods

Results

Conclusions

Translational relevance

Purpose

Methods

Results

Conclusions

Translational relevance