The problem of forecasting the behavior of a complex dynamical system through analysis of observational time-series data becomes difficult when the system expresses chaotic behavior and the measurements are sparse, in both space and/or time. Despite the fact that this situation is quite typical across many fields, including numerical weather prediction, the issue of whether the available observations are `sufficient' for generating successful forecasts is still not well understood.

The goal of this study is to develop the assimilation schemes using the time series observation. Several approaches to Lorenz 96 chaotic dynamics and geophysical fluid dynamics have been proposed.

Chapter 1 and 2 contain the problem statement of data assimilation and description of two nonlinear models, the Lorenz 96 model, a toy model commonly used in data assimilation, and the shallow water model, a fast but relatively realistic model in geophysical fluid dynamics. The details of the simulations are provided.

In Chapter 3, we will systematically compare the data assimilation methodology in literature. We will focus on the advantages and disadvantages of each method.

In Chapter 4, the time delayed nudging method with the time delayed observation data is presented. We show that in certain circumstances, it provides a sizable reduction in the number of observations required to construct accurate estimates and high-quality predictions. In particular, we find that this estimate of 70\% can be reduced to about 33\% using time delays, and even further if Lagrangian drifter locations are also used as measurements.

Chapter 5, we apply the sequential adversarial method on Lorenz 96 model. We find that the adversarial method can highly reduce the task burden of the parameter tuning in the data assimilation algorithm. We obtain a substantial improvement in the traditional sequential data assimilation scheme.

Chapter 6 will be the conclusion and discussion on future work.

The determination of the hidden states of coupled nonlinear systems is frustrated by the presence of high-dimensionality, chaos, and sparse observability. This problem resides naturally in a Bayesian context: an underlying physical process produces a data stream, which -- though noisy and incomplete -- can in principle be inverted to express the likelihood of the underlying process itself. A large class of well-developed methods treat this problem in a sequential predict-and-correct manner that alternates information from the presumed dynamical model with information from the data.

One might instead formulate this problem in a temporally global, non-sequential manner, which suggests new avenues of approach within an optimization context, but also poses new challenges in numerical implementation. The variational annealing (VA) technique is proposed to address these problems by leveraging an inherent separability between the convex and nonconvex contributions of the resulting functional forms. VA is shown to reliably track unobservable states in sparsely observed chaotic systems, as well as in minimally-observed biophysical neural models.

Second, this problem can be formally cast in continuous time as a Wiener path integral, which then suggests classical solutions derived from Hamilton's principle. These solutions come with their own difficulties in that they comprise an unstable boundary-value problem. Accordingly, a further technique called Hamiltonian variational annealing is proposed, which again exploits an existing separability of convexity and nonlinearity, this time in a an enlarged manifold constrained by underlying symmetries.

A running theme in this thesis is that the optimal estimate of a nonlinear system is itself a dynamical system that lives in an unstable, symplectic manifold. When this system is recast in a variational context, instability is manifested as nonconvexity, the central idea being that when this nonconvexity is incorporated in a systematic and gradual way, the classical solutions can be tracked reliably.

Characterizing the many physical properties of biological neurons has been historicallydifficult. The use of conductance models describing how properties of these systems change with time can be combined with measured voltage traces to help characterize immeasurable properties of a neuron. This is accomplished using data assimilation, which formulates the inference of these properties as probability maximization using nonlinear optimization. Because measurement noise and model error are inevitable in the study of complex systems, the methods used in this dissertation are designed to cope with unknown processes. This dissertation starts with an overview of data assimilation by formulating the problem xiii data assimilation attempts to solve. I then introduce two methods, nudging and variational annealing, which I use to characterize the properties of a neuron in the Zebra Finch songbird system, HVCX. A key result from this experiment is that there are statistical differences in estimated model parameters for neurons from different birds. Current artificial neural network models are unmanageably large (billions of parameters) and need massive amounts data and computational power to train. The insect olfactory system is a biological network which is capable of learning and identifying new odors quickly in the presence of interference. These properties make it attractive model to explore as a classifier for machine learning tasks. I give an overview of the insect olfactory system by describing its key properties. I use these properties to create a simplified classification system using a winnerless competition network as a pre-processor to a support vector machine. I demonstrate this networks classification capabilities using classes designed to resemble the stimulus an insect receives in the presence of odors. One key result from this experiment is that our network is capable of identifying a mixtures of previously seen odors.

Dynamical systems are an incredibly broad class of systems that pervades every field of science, as well as every aspect of daily life. Not only are they pervasive, but they often exhibit complex behavior resulting from microscopically simple interactions. Examples of such systems are the weather, our brains, animal population, and even home prices – to name but a few. As such, predictions of these systems pose a towering yet necessary challenge, and this work aims at making at dent in this effort. To the extent that models are available, they are useful in that constraints are automatically satisfied, and a mechanistic understanding of the system naturally follows. Part one of this work addresses this case by leveraging our knowledge of the physical model. However, it is often the case that the model is not known, so an effective surrogate model is desired. Part two proceeds in this vein, where the availability of large amounts of data is utilized in constructing surrogate models. Though the theory of dynamical systems still applies to the constructed surrogate model, this approach disregards the physics of the underlying system and has a machine learning flavor.

Traditionally, characterizing many properties of biological or silicon neural systems has been expensive, laborious, or impossible. Conductance models describing how properties of these systems change with time can be used with accessible data, such as measured voltage traces, to help characterize inaccessible properties such as ionic currents or transistor mismatch. This is accomplished using variational methods which formulate an inference problem about these properties as nonlinear optimization. Because measurement noise and model error are inevitable in the study of complex systems, the method is designed to cope with unknown processes. Conductance models are overparameterized, causing the inference problem to remain underdetermined, which can result in a proliferation of widely separated sets of estimated model parameters producing accurate predictions. Additionally, real world data will be approximated by a model in a number of ways, leading to an additional contribution to this model identifiability problem. This dissertation probes and overcomes some of the difficulties encountered in the analysis of real world data in individual biological and silicon neurons. One key result is the characterization of a neuromorphic silicon neuron followed by emulation of a biological neuron on the silicon substrate. Another key result is a data mining approach which discovers statistical differences in estimated model parameters, despite underdeterminacy, in an Alzheimer's strain of neurons in mice compared to healthy controls.

Complex systems are found at the heart of the most pressing intellectual challenges of the 21st century. Weather and climate, ecological and neurobiological networks, many body quantum systems, and even the operation of human societies can be described as the interaction of simple components from which collectively emerge distinct phenomena. In this thesis, key concepts from dynamical systems theory and machine learning are employed to analyze complex systems in both neurobiological contexts and weather prediction. The first of these concepts is synchronization, from which data assimilation is derived in order to describe the optimal interpolation between an imperfect physical model and sparse/noisy data. Data assimilation is applied to biophysical problems such as the design of VLSI circuits for neuromorphic computing, as well as the estimation of a biological neuron's physical characteristics from voltage and calcium fluorescence measurements. Generalized synchronization is then deployed as the framework to examine the operation of recurrent neural networks, particularly the reservoir computing (RC) architecture. RC operates through the synchronization of a high dimensional artificial network together with chaotic input data, and is able to produce state of the art forecasting performance on a number of benchmark systems in numerical weather prediction. Invariant quantities such as the Lyapunov exponent spectrum and the fractal dimension of the RC are calculated and shown to be directly related to the input dynamics of the data. The concluding remarks unify both RC and data assimilation together into a machine learning forecasting cycle that produces forecasts from sparse and noisy data with no physical model.