UC Irvine

In this thesis, we study efficient attention in the dominant transformer architecture from a theoretical perspective. We then use this theoretical understanding to guide the design of models for problems in time-series forecasting, image processing, and video understanding. In Part I, we begin with the motivation for and an introduction to modern machine learning, starting with recurrent neural networks for processing sequential data and continuing to the current transformer-based approach. We discuss how image processing has progressed from relying mainly on convolutional neural networks, which respect image geometry, to the current paradigm of transformers in computer vision. We provide an overview of the chapters in this dissertation and explain how they connect to the core research question. We then present a unified theoretical analysis of the attention mechanism, laying the theoretical groundwork for the remainder of the dissertation. In Part II, we consider applications to time-series forecasting. Chapter 3 presents the main work on developing FWin as an efficient model and comparing it with many state-of-the-art methods. In particular, we combine window attention with the Fourier transform as an efficient global mixing operator. Chapter 4 then validates the proposed FWin method on a more challenging dataset involving dengue cases in Singapore, where the data are nonstationary and quite limited. This demonstrates that FWin is robust enough to adapt to other applications rather than only perform well on standard benchmarks with well-known properties. In Part III, we develop a mathematically inspired efficient attention mechanism for image processing. We use arithmetic averaging as a global approximation of full attention while using window attention to mimic the local and recurrent nature of Mamba. This allows our proposed SEMA method to combine two well-known mechanisms in computer vision, retain their advantageous properties, and complement their weaknesses. We verify our design against state-of-the-art models on several challenging benchmarks. Following this success, we extend SEMA into a video model in the next chapter. We design a split space-time transformer in which SEMA processes the spatial component and standard softmax attention processes the temporal component. This design is efficient and scales to high-resolution videos. We conclude the dissertation by discussing the limitations of the current line of research and the importance of the works presented in this thesis. We also point toward future research directions in efficient machine learning and discuss their importance for real-world applications.