As High Performance Computing (HPC) has grown considerably and is expected to grow even more, effective resource management for distributed computing sys- tems is motivated more than ever. As the computational workloads grow in quantity, it is becoming more crucial to apply efficient resource management and workload scheduling to use resources efficiently while keeping the computational performance reasonably good. The problem of efficiently scheduling workloads on resources while meeting performance standards is hard. Additionally, non-clairvoyance of job dimen- sions makes resource management even harder in real-world scenarios. Our research methodology investigates the scheduling problem compliant for HPC and researches the challenges for deploying the scheduling in real world-scenarios using state of the art machine learning and data science techniques.

To this end, this Ph.D. dissertation makes the following core contributions: a) We perform a theoretical analysis of space-sharing, non-preemptive scheduling: we studied this scheduling problem and proposed scheduling algorithms with polyno- mial computation time. We also proved constant upper-bounds for the performance of these algorithms. b) We studied the sensitivity of scheduling algorithms to the accuracy of runtime and devised a meta-learning approach to estimate prediction accuracy for newly submitted jobs to the HPC system. c) We studied the runtime prediction problem for HPC applications. For this purpose, we studied the distri- bution of available public workloads and proposed two different solutions that can predict multi-modal distributions: switching state-space models and Mixture Density Networks. d) We studied the effectiveness of recent recurrent neural network models for CPU usage trace prediction for individual VM traces as well as aggregate CPU usage traces. In this dissertation, we explore solutions to improve the performance of scheduling workloads on distributed systems.

We begin by looking at the problem from the theoretical perspective. Modeling the problem mathematically, we first propose a scheduling algorithm that finds a constant approximation of the optimal solution for the problem in polynomial time. We prove that the performance of the algorithm (average completion time is the constant approximation of the performance of the optimal scheduling. We next look at the problem in real-world scenarios. Considering High-Performance Computing (HPC) workload computing environments as the most similar real-world equivalent of our mathematical model, we explore the problem of predicting application runtime. We propose an algorithm to handle the existing uncertainties in the real world and show-case our algorithm with demonstrative effectiveness in terms of response time and resource utilization. After looking at the uncertainty problem, we focus on trying to improve the accuracy of existing prediction approaches for HPC application runtime. We propose two solutions, one based on Kalman filters and one based on deep density mixture networks. We showcase the effectiveness of our prediction approaches by comparing with previous prediction approaches in terms of prediction accuracy and impact on improving scheduling performance. In the end, we focus on predicting resource usage for individual applications during their execution. We explore the application of recurrent neural networks for predicting resource usage of applications deployed on individual virtual machines. To validate our proposed models and solutions, we performed extensive trace-driven simulation and measured the effectiveness of our approaches.

The emerging field of Cloud Computing provides elastic on-demand services over the Internet or over a network. According to the International Data Corporation (IDC), cloud computing has two major issues: i) architecture issues, such as a lack of standardization, a lack of customization; and ii) users’ data privacy. In this study we focus on these issues.

We are facing an increasing demand for migration of varieties of traditional databases and computation services to cloud computing environments, i.e., database-as-a-service. Although each service offers a new feature, it escalates standardization and customization issues due to the lack of standardization between cloud vendors and service customization because each cloud-based service has its own features, requirements and outputs. In the first part of this study, we propose a cloud architecture based on a Service-Oriented Architecture (DCCSOA) that enhances our ability to do standardization and customization in the cloud. The proposed architecture uses a single layer, which is called Dynamic Template Service Layer (DTSL), that provides the following operations and advantages:

i) enables a single service layer to interact with all native cloud services (e.g., IaaS, PaaS, SaaS and any cloud-based services);

ii) provides a standardization for existing services and future services in the cloud;

iii) customizes native cloud services based on users’ group requests.

The second part of this study focuses on users’ data privacy preservation on the proposed architecture. Users’ data privacy can be violated by the cloud vendor, the vendor’s authorized users, other cloud users, unauthorized users, or external malicious entities. Encryption of data on client side is one of the solutions to preserve data privacy in the cloud; however, encryption methods are complex and expensive for mobile devices to encrypt and decrypt each file, such as smart phones. We propose a novel light-weight data privacy method (DPM) by using a chaos system for mobile cloud users to store data on multiple clouds. The proposed method enables mobile users to store data in the clouds while it preserves users’ data privacy.

We consider different technologies to deploy our proposed data privacy preservation method on DCCSOA, including the mobile devices, the Internet-of-things (IoT), and Graphic Processing Unit (GPU)-based computing. We also consider different use case scenarios for the proof of concept, including data privacy preservation for users’ photos in smart phones, sensitive electronic health records protection in the cloud, and data privacy preservation for cloud-based databases.

We evaluate both the proposed dynamic architecture and the proposed data privacy preservation method. Our experimental results show that on the one hand DCCSOA enhances standardization by offering a flexible cloud architecture and minimizing the modification on the native cloud services; on the other hand, DPM achieves a superior performance over regular encryption methods in regard to computation time.

With progress in enabling autonomous cars to drive safely on the road, it is time to ask how should they be driving. This dissertation focuses on learning the desired objective function for autonomous cars with the goal of personalizing autonomous driving: drive following the passenger’s preferences across diverse environments. Traditionally autonomous cars have been trained using expert demonstrations, with an implicit assumption that the demonstrations are truly representative of optimal driving. Personalizing autonomous driving under this assumption would mean using Inverse Reinforcement Learning (IRL) to learn the objective function latent in the user’s own demonstration and then adopt the user’s own driving style. In this thesis, we question this assumption and propose algorithmic solutions for personalizing driving styles without demonstration data. Through user studies in a simulated driving environment, we first show that people do not want their autonomous cars to drive like them: they want a significantly more defensive car. Next we formalize driving preference as reward functions and propose several algorithms to learn them interactively from an alternative form of human guidance: Preference-based Learning. In Preference-based reward learning we show users several trajectory pairs sequentially and ask them to indicate their preference in each pair. This has been shown to be effective for learning reward functions in absence of demonstrations. Simple preference is, however, far less informative than all the demonstration data. The key contribution of this thesis is an algorithmic framework that leverages computational models of human behavior to enable learning from richer preference queries where response to each query contains more information than just a comparison. We propose different forms of rich preference queries. We ask people not only what they prefer, but also why they prefer. We design new queries to learn more complex reward functions that can potentially represent preferences in non-stationary environments. We introduce reward dynamics as a mixture of reward functions and parameters that govern how preferences change in response to the dynamics of the environment. We develop a unified formalism for treating all forms of human guidance as observations about the true preferences and use this formalism to derive objective functions for actively generating rich queries. We show empirically through simulations and also with user studies that richer preference queries can learn driving preference more accurately than comparison-alone queries. We also discover that richer queries not only speed up preference learning in practice but also offer more transparency into the decision-making algorithms of the autonomous car, thus enhancing people’s trust in the system. Although the human- robot system of choice in this thesis is autonomous car, our algorithmic solutions apply to personalizing other human-robot systems where the robot is a dynamical system that should match human preference and where demonstrations are unavailable due to complexity of robot operation or disparity between preferences and demonstrations.

Deep learning has made remarkable progress over the past decade across various fields, such as Computer Vision, Natural Language Processing (NLP), and Speech Recognition, driving innovation and advancements of various applications. One key challenge is to improve deep learning models' generalization, which refers to the capability of a model to handle unseen data. Several factors contribute to the challenge of generalization in deep learning models, including limited data availability, overfitting tendencies, and the inherent complexity of the models themselves.

The phenomenon of adversarial samples is symptomatic of the limitation in the generalization capability of deep learning models. Adversarial samples refer to data instances manipulated from their originals, often appearing visually similar and imperceptible to human senses, yet causing incorrect predictions from deep learning models. This phenomenon presents a substantial concern for security-sensitive domains like medical diagnosis, autonomous driving, and anomaly detection, where model reliability is crucial.

The development of various adversarial defense methods in recent years, such as adversarial training, noise reduction, and gradient masking, emphasizes the considerable efforts to enhance the robustness and reliability of deep learning models. Meanwhile, as innovative adversarial attacks continue to evolve, they effectively expose the vulnerabilities inherent in deep learning models, thereby raising challenges for existing defense methodologies.Although adversarial defense research has made advancements, the root cause of the vulnerability in deep learning models is still not fully understood. Additionally, there is a pressing need for defense mechanisms that offer comprehensive protection and high resilience against a wide range of adversarial attacks. The research presented in this dissertation aims to contribute to the enrichment of knowledge within the research community by providing deeper insights into adversarial attacks and defense mechanisms. It endeavors to develop novel defense methods that are robust and reliable when protecting deep learning models against adversarial threats. Through this work, we seek to advance the field of adversarial defense and contribute to the development of more effective defense strategies.

In this dissertation, we introduced three novel defense mechanisms aimed at enhancing the robustness of deep learning models against adversarial attacks. In the first work, we tackled the issue of image blurring in traditional Variational Autoencoder (VAE)-based generative networks by focusing on improving high-fidelity data reconstruction. Additionally, this work optimized the model's decision-making strategy through a Bayesian update, allowing a model to incorporate multiple sources of supporting evidence for the final decision. The second study proposed a new generative network structure coupled with a new two-step noise reduction approach designed to effectively filter out adversarial noise. The third method introduced a new noise reduction mechanism called VQUNet. This method features a learnable quantization of latent features and a hierarchical network structure for high-fidelity data reconstruction. VQUNet's unique design significantly enhances the data reconstruction quality after the filtering process, while effectively regularizing adversarial perturbation within the network, thereby improving its resilience against adversarial attacks.

Extensive experimental investigations demonstrated that the proposed methods provided superior robustness to the targeted deep learning models. They exhibited superior performance over other state-of-the-art noise-reduction-based defense methods, achieving prediction improvement with a notable margin over existing methods under adversarial attacks across both Fashion-MNIST and CIFAR10 datasets. The experimental analysis underscored the effectiveness, resilience, and robustness of the proposed methods against adversarial attacks. These findings offered valuable insights into the development of effective defense strategies, shedding light on the mechanisms and principles for future research.

The Expectation-Maximization (EM) Algorithm is well-known in the literature of machine learning and has been widely used for training of probabilistic and some non-probabilistic models, such as mixture of Gaussians and K-means, respectively. Despite the vast volume of research on application of the EM algorithm for training probabilistic models, there has been little attempt toward usage of the EM algorithm for non-probabilistic models. In this dissertation, various piecewise constant models, and their learning procedures in the literature are reviewed. For each model, the EM-based optimization of reviewed model is proposed. The EM algorithms proposed in this dissertation have the same spirit as the original EM algorithm. For each model, the proposed EM algorithm is properly modified to fit the non-probabilistic nature of the model. The EM algorithm was originally designed to fit the modular structure of any intelligent model, such as neural networks or mixture models. In this dissertation, it is shown how with the EM algorithm it is possible to approach a piecewise constant model as a modular structure and optimize the model based on each module of the structure. The optimization procedure consists of two steps, Expectation/assignment step and Maximization/update step. More specifically, in the EM algorithm, for each module of the structure, a maximization/minimization problem has to be solved. The parameters of optimization problem for each module are provided by the expectation step for that module. In this dissertation, it is shown that such optimization problems are NP-hard and can often be approximated through a proper surrogate objective function. We proposed novel surrogate functions. The proposed EM-based approach is applied to several piecewise constant models, such as prototype nearest neighbor. Further, the convergence guarantee and computational complexity of the developed EM algorithms are presented for each model. Finally, through extensive experiments we show that the proposed EM-based algorithms have superior or similar performance when compared with several other similar state-of-the-art models and algorithms. Additionally, the proposed approach for optimizing the piecewise constant models provides an in-depth interpretability for training procedures. We specifically applied the proposed optimization algorithm to synthetic reduced nearest neighbor for classification, adversarial label-poisoning, robust synthetic reduced nearest neighbor and synthetic reduced nearest neighbor for regression.

Neural networks generally require large amounts of data to adequately model thedomain space. In situations where the data are limited, the predictions from these models, which are typically obtained from stochastic gradient descent (SGD) minimization algorithms, can be poor. In addition, the data is commonly corrupted due to poor imaging appatus. In these cases, the use of more sophisticated optimization approaches and model architectures becomes crucial to increase the impact of each training iteration. Second-order methods can capture curvature information, providing a more informed guess on the direction and step length. However, they require vast amounts of storage and can be computationally time demanding. To address the computational issue, we propose an optimization algorithm that uses second-derivative information, exploiting curvature information for avoiding saddle points. We utilize a Hessian-free approach where we do not explicitly store the second-derivative matrix, by applying a conjugate gradient method. The algorithm uses a trust-region method, which does not require the Hessian to be positive definite. We present numerical experiments which demonstrate the improvement in classification accuracy using our proposed approach over a standard SGD approach. We propose using a limited-memory symmetric rank-one quasi-Newton approach which further addresses the time and space computational complexity. The approach allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified adaptive regularized using cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions with suitable regularization choices and investigate the performance of our proposed and compare our approach to state-of-the-art first-order and other quasi-Newton methods. To incorporate the benefits of an exponential moving average algorithm to a quasi-Newton approach, we propose a quasi-Adam approach. Judicious choices of quasi-Newton matrices can lead to guaranteed descent in the objective function and improved convergence. In this work, we integrate search directions obtained from using these quasi-Newton Hessian approximations with the Adam optimization algorithm. We provide convergence guarantees and demonstrate improved performance through an extensive experimentation on a variety of applications. Finally, to mitigate the issue of data corruption, we propose a variety of architectures for various applications in image processing. We propose a blind source signal separator, which involves separating image signals which have been superimposed by a common observing apparatus. We propose novel deep learning architectures for low photon count image denoising, which contains Gaussian noise in a low-photon count setting. Then we propose a novel architecture for lowphoton count and downsampled imaging, where the signal is interfered with some Gaussian noise, Poisson noise and then downsampled. Finally, we propose a novel adversarial detection method for white-box attacks using Radial basis function and Discrete Cosine Transforms.