UC Riverside

Over the past few years deep learning has demonstrated impressive performance on many important practical problems such as image, video, and audio classification. This work develops three novel applications for automated stem cell classification, automated sports analytics, and a novel framework for defending deep learning models from white and black box adversarial attacks. In the field of stem cell classification, it is very expensive and time consuming to generate data which is very intrusive and not easy to obtain. This work leverages an ensemble of generative networks to create a large dataset of synthetic human embryonic stem cell (hESC) images which are exclusively used for training deep learning classifiers. In order to verify that the data distribution of the synthetic images is similar to that of the real-world images, the quality of the synthetic images are validated at a pixel level and high dimensional feature level with respect to the real-world data. Experimental results show that the classifiers trained on the synthetic dataset are able to achieve high performance when evaluated on real-world data and can be used as a tool for annotating more data saving hours of manual labor.

In the field of automated sports analytics, it is very important to analyze every minute detail in order to generate reliable statistics for every individual player. This work develops a novel framework for automatically generating the tactical statistics of soccer players directly from a video. The proposed approach empirically shows that high-level features learned from specific soccer matches do not necessarily generalize across all soccer matches and it is not feasible to obtain datasets for every single match. To solve this, the proposed approach develops a match-specific application that uses previously recorded videos of teams to learn fine-grained features that can generalize across other matches played by the same respective teams. Although generative networks have had huge success in augmenting existing datasets which improve the performance of deep learning classifiers, this work shows that they often overlook minute details when generating new data which is very important in sports analytics and can cause the performance of the classifiers to drop. This work proposes a novel generative architecture that learns to generate synthetic images with fine-grained structures which further improves our system to generate accurate tactical statistics for the players. Various ablation studies are performed to show the improvement in performance and significance of the results across different soccer matches.

Despite their outstanding performance, these models are vulnerable to adversarial manipulation of their input which could lead to poor performance. These adversarial manipulations are carefully crafted perturbations that are so subtle that a human observer does not even notice the modification at all, but can cause deep learning models to predict incorrect results. In order to address this vulnerability, this work proposes a novel white box defense algorithm that uses generative networks with Probabilistic Adversarial Robustness to neutralize adversarial examples by concentrating the sample probability to adversarial-free zones. Although, our proposed defense achieves state-of-the-art classification accuracy, this is not a reliable metric to determine if an image is ``adversarial-free''. This is a foundational problem for online image verification applications where the ground-truth of the input image is not known and hence we cannot validate the performance of the classifier or know if the image is ''adversarial-free'' or not. To address this problem, this work proposes a novel framework that uses an ensemble of individual defenses whose performance is continuously validated in a loop using Bayesian uncertainties and does not require any information about the black box classifier such as its architecture, parameters, or training dataset. Unlike existing defense mechanisms that requires knowing the ground-truth of the input data and modifying/re-training the black box classifier which is not feasible in online applications, our defense is designed in the first place to provide proactive protection to any existing deep learning based model. Evaluation on various public benchmark datasets including autonomous driving and face biometrics datasets shows that our defense can consistently detect adversarial examples and purify them against a variety of attacks with different ranges of perturbations.