Modern neural architectures present strong performance on various tasks, but their abilities are compromised when faced with the tasks requiring robust reasoning or involving abstract concepts. This dissertation starts with a study of the empirical representation power of two popular sequential neural networks, Long Short-Term Memory (LSTM) and Transformer network, in learning context-free grammar. The experiment results exhibit that the performance of both models is affected by the types of supervision and model regularization setting. Specifically, the LSTM and the Transformer network struggle to capture the grammatical structures, particularly the grammars with deeper recursions, when solely trained on input sequences in the generative language model setting. The performance gap between settings suggests two improvements when the input data structure is available. One approach is to supervise the model with intermediate states, while another is to regularize the models’ hidden states to enforce the its decomposition to store the state and to simulate the stack, mimicking a pushdown automaton, on separate segment.Complementing the study on sequential neural networks with prototypical grammars, several real-world applications are presented to showcase how the controlling the model complexity and intermediate state can enhance the performance. For the item retrieval problem in recommendation system, another transformer-based sequential model is proposed to capture the multiple interests of a user. Unlike the existing multi-interest frameworks, the proposed model is unbiased towards the site-wide popular items at the cost of incorporating a non-differentiable clustering method inside the neural network. The non-differentiable challenge are solved by two-stage training method, borrowing the idea from K-means clustering algorithm. In time series prediction tasks, a continuous convolutional neural network (CCNN) is proposed to handle non-uniformly sampled time series without preset interpolation kernels. CCNN can directly apply to the auto-regression tasks, and also works with temporal point process front-end to predict the time interval to the future event in the event-based time series. Lastly, a progressive optimization system, HISS , is designed to simplify the expressions in the intermediate representation of Halide compiler. HISS is built end-to-end without any human knowledge of the mathematical equivalences. Multiple training techniques, existing symbolic transformations of the expressions, and the evolving neural network modules, contribute to the HISS ’s performance that exceeds conventional searching method and human-heuristic methods. These three applications illustrate the benefits and techniques of applying the aforementioned philosophy, which suggests designing and supervising the neural networks to reflect the desired properties of neural networks. In conclusion, by presenting an empirical study method for evaluating the representation power of neural networks, which can be extended to broader areas, and the demonstrating how to design and overcome the challenges associated with symbolic properties in various real-world applications, this dissertation contributes to the advancement of the neural symbolic systems.

Recently, autonomous vehicle development ignited competition among car makers and technical corporations. Low-level automation cars are already commercially available. However, the high automated vehicle where the vehicle drives by itself without human monitoring is still at infancy. Such autonomous vehicles (AVs) rely on the AV system to ensure safety. The AV system consists of two key components: data centers and onboard systems. Data centers are responsible for training deep neural network models, which will be used in the onboard system. It is necessary for data centers to train models efficiently within limited periods. The AV onboard system act as human drivers, as it monitors surroundings and plans a route for the AV to drive. To ensure safety, the AV onboard system needs to make timely and appropriate driving decisions. Moreover, a safety validation stage for the onboard system is also required to guarantee AVs' safe operations.

To address the above mentioned challenges, this dissertation proposes three designs. In Chapter 2, this dissertation proposes a processing-in-memory architecture to accelerate deep neural network training stage, which reduces the data movement overhead and improves energy efficiency. In Chapter 3, this dissertation presents the safety score, a latency based safety metric, and the latency model, that represents the correlation between perception latency and surrounding obstacle distribution; then it presents the resource management scheme to optimize safety and performance. In Chapter 4, this dissertation presents Suraksha, a generalized safety validation framework that includes a set of safety metrics and driving scenarios; it also employs Suraksha to perform a case study, where the perception module is studied and safety effects can be collected and analyzed by changing selected perception parameters.

Deep Neural Networks (DNNs) are increasingly adopted in various fields due to their unprecedented performance. Yet, the computation overhead of DNN evaluation and training continues to grow exponentially. Enormous system-level advancements have recently been witnessed to improve the efficiency of DNN computation; many efficient DNN algorithms are proposed to reduce the cost of DNN computation. However, this is far from optimal. Most current DNN computation systems do not fully exploit efficient ML algorithms, while ML algorithms fail to consider novel systems for deployment.

This thesis focuses on designing efficient DNN algorithms and DNN computation systems to enable fast DNN training and evaluation. Instead of solely focusing on creating either new DNN algorithms or systems, we propose a co-design approach by exploring DNN models that are system-aware and DNN computation systems that are algorithm-aware. By leveraging such a co-design approach, we effectively advance the Pareto-frontier between task accuracy and efficiency of DNN execution in various application domains.

We present a set of works that explore the co-design method. Firstly, we present an algorithm-aware DNN compiler for quantized DNN. By leveraging the weight repetition feature of this efficient DNN algorithm, we can greatly reduce the computation overhead of DNN inference on both CPU and GPU. %This work illustrates how algorithm-aware system design can help in pushing the Pareto-frontier. Secondly, we discuss a hardware-aware DNN algorithm with enhanced model parallelism. We observe that previous works design efficient DNNs for single-device platforms. When customizing the DNN design for a multi-device system, we can reduce the DNN inference latency by a large margin while previous models can hardly be parallelized across multiple devices. Thirdly, we present a hardware-friendly transfer learning framework for natural language processing tasks. The existing transfer learning frameworks have a lot of computation redundancy when deploying on the existing systems. By reusing the computation of different transfer learning models, we can greatly reduce the computation overhead as well. Lastly, we introduce a novel training method to reduce the computation cost of DNN training and DNN design process. The key idea is to initialize the large models using small pretrained weights. The implicit knowledge in the pretrained models facilitates faster convergence of the large models. Besides, changing only the initialization phase of training means no extra computation overhead will be introduced to the existing training systems. Also, this new training method can be applied to accelerate the design process of system-aware DNN models.

As Moore's Law is slowing down, the computational capacity of current DNN systems is plateauing. This thesis sheds light on how to overcome this limitation by designing domain-specific DNN algorithms and computation systems.

The emerging Non-Volatile Memory (NVM) technologies are reforming the computer architecture. NVM holds advantages includes a byte-addressable interface, low latency, high capacity, and in-memory computing capability. However, data-intensive applications today demand compound features rather than just better performance. For instance, big data applications would require high availability and reliability. The neural network applications require scalability and power efficiency. Despite all the advantages of NVM, simply attaching the NVM to the memory hierarchy are unable to meet these demands. The decoupled reliability schemes among NVM and other devices fail to provide sufficient reliability. The vulnerability against overheating and hardware underutilization limit the performance and scalability of the in-memory computing NVM.Using the NVM for the data-intensive application requires redesign and customization. In this thesis, we focus on discussing the architecture designs that enable NVM for data-intensive applications. Our study includes two major types of data-intensive applications – big data applications and neural network applications. We first conduct a characteristic study against the persistent memory applications. Persistent memory implements over the NVM-based main memory and guarantees crash consistency. We explore the performance interaction across applications, persistent memory system software, and hardware components. Based on our characterization results, we provide a set of implications and recommendations for optimizing persistent memory designs. Second, we propose Binary Star for the generic data-intensive applications, which coordinates the reliability schemes and consistent cache writeback between 3D-stacked DRAM last-level cache and NVM main memory to maintain the reliability of the memory hierarchy. Binary Star significantly reduces the performance and storage overhead of consistent cache writeback by coordinating it with NVM wear leveling. For neural network applications, our first design explores the thermal effect over one representative NVM – resistive memory (RRAM). We find heat-induced interference decreases the computational accuracy in the RRAM-based neural network accelerator. We propose HR3AM, a heat resilience design, which improves accuracy and optimizes the thermal distribution. Results show that HR3AM improves classification accuracy and decreases both the maximum and average chip temperatures. Lastly, we present Mirage to improve parallelism and flexibility for pipeline-enabled RRAM-based accelerators. Mirage is a hardware/software co-design that addresses the data dependencies and inflexibility issues of existing accelerators. Our evaluation shows that Mirage achieves low inference latency and high throughput compared to state-of-the-art RRAM-based accelerators.

Heterogeneous systems emerge to enhance application performance by distributing tasks across various domain-specific accelerators and memory devices. However, adopting such systems often results in unanticipated performance characteristics in modern workloads, unforeseen vulnerabilities introduced by new architecture designs, and programming challenges utilizing novel architectures.

This dissertation explores a cross-stack approach to building heterogeneous systems. We start from the architecture level, studying the performance characteristics of emerging heterogeneous memory architectures. This led to our system-level study integrating diverse types of memory devices at a large scale, which facilitates efficient training of large machine learning models. These explorations revealed vulnerabilities in emerging memory architectures and led to our development of architectural side-channel attacks and cross-stack defenses to secure heterogeneous systems. Finally, to facilitate the adoption of heterogeneous systems, we proposed a novel programming model that provides a unified interface for heterogeneous computing.

Failures are unavoidable in engineered systems such as autonomous vehicles, butthe latency of recovering a failed component degrades the performance of autonomous vehicles. We proposed a scheme to reduce the time of recovering autonomous vehicles from failures. Using Linux kernel features and container technology, we containerize functional components of autonomous vehicles and periodically take checkpoints. After detecting a failure, we recover the failed component to a previous state, and we notify the rest of the system to coordinate with the recovery. We test our method using the Robot Operating System (ROS), a widely-used middleware for robots and autonomous driving vehicles. Our initial experimental results show that we reduced the recovery time of a practical pointcloud processing component in autonomous driving by 94%.

Memory systems are evolving in a multitude of ways in state-of-the-art computer systems and in computer architecture. The changes range from the low-level underlying technology used to implement memory, to the high-level ways in which memory systems are structured and organized. These changes have tremendous implications on the fundamental performance of computer systems because of how integral a role memory plays in the design of these systems. Traditionally, the memory hierarchy has been the bottleneck for how well computer systems perform and the resources they consume. Computer architects face significant challenges in meeting the growing memory demands and efficiently integrating these new technologies. Computer architecture must adapt to these changes in a high-performance, cost-effective way. Modern problems require modern solutions.

This thesis proposes efficient architecture solutions for two primary contemporary developments in memory: persistent memory and memory networks. Persistent memory is a subset of nonvolatile memory with similar properties of storage systems. As such, it comes with not only many benefits, but also a number of challenges inherent to both storage systems but to new technologies in general. Similarly, memory networking shares similar properties to network on-chip as well as computer networking design. Persistent memory and memory networks can go hand in hand to implement high performance computing or database systems to meet the demands of current applications.