Understanding and predicting the dynamics is one fundamental problem that supports various real-world applications. Deep learning dynamical models such as recurrent neural networks (RNNs) and Transformer show powerful expressiveness in modeling sequential data. However, pure deep learning models lack appropriate inductive bias for dynamics, which limits their potential for more accurate dynamic predictions.

This dissertation aims to enhance deep neural networks' capability of modeling dynamics. My research starts by injecting physical law as prior knowledge into deep nets, with the finding that such prior knowledge shapes the predicted trajectory desirably and therefore achieves more accurate forecasting. However, such physical law is not available for more general and complicated dynamics, such as retail time series, and energy consumption sequence. To this end, we propose to use the Fourier series instead of task-specific rules as a more general inductive bias to capture the periodicity. Unfortunately, either specific physical law or general periodic series still just learns the association between historical observations and the future series. However, answering counterfactual questions like ``Would the community protection be better had a different group of people gotten vaccinated first?'' is one key problem for decision-making in dynamical systems. A dynamical is naturally represented by a graph, where units are nodes and the interactions among them are edges. The second part of my research focuses on how to answer causal questions on graphs and then extend to general dynamical systems.