UC Santa Barbara

With the end of Moore’s Law and the rise of compute- and data-intensive deep learning (DL) applications, the focus on arduous new processor design has shifted towards a more effective and agile approach: Intelligent Software to maximize the performance gains of DL hardware like GPUs. There are several highlights of such intelligent software design. First, it would maximize the execution efficiency of existing and emerging DL algorithms on powerful platforms like GPUs. Second, it would promote the adaptiveness of systems to handle a diverse range of inputs. Third, it would maintain sufficient portability and scalability across a diverse range of platforms, such as mobile devices and high-performance clusters.

In this thesis, I will first highlight the importance of software innovation to bridge the gap between the increasingly diverse DL applications and the existing powerful DL hardware platforms. The second part of my thesis will recap my research work on DL system software innovation, focusing on 1) Precision Mismatch between DL applications and high-performance GPU units like Tensor Cores (e.g., QGTC [PPoPP ’22] and APNN-TC [SC ’21]), to improve the efficiency of quantized deep learning on powerful GPU platforms, and 2) Computing Pattern Mismatch between the sparse and irregular DL applications, such as Graph Neural Networks, and the dense and regular tailored GPU computing paradigm (e.g., GNNAdvisor [OSDI ’21] and MGG [OSDI ’23]), to highlight system adaptability and scalability. Finally, I will conclude this thesis with my vision and future work for building efficient, scalable, and secure DL systems.