Symplectic numerical integrators for Hamiltonian systems form the paramount class of geometric numerical integrators, and have been very well investigated in the past forty years. By preserving the symplecticity of the flow of Hamiltonian systems, symplectic integrators generate discrete solutions which enjoy many desirable qualitative properties such as excellent long-time near-energy preservation over exponentially-long time intervals, and as a result typically exhibit superior numerical stability and better long-time fidelity to the continuous dynamics they are resolving.

The purpose of this dissertation is to establish, explore and exploit connections between symplectic numerical integration and two prominent disciplines of scientific computing: accelerated optimization for machine learning applications, and structure-preserving dynamics learning. By leveraging the well-established theory of symplectic integration, we aspire to design new algorithms for accelerated optimization and dynamics learning with superior numerical and computational properties.

Most machine learning algorithms are designed around the minimization of a loss function or the maximization of a likelihood function. Due to the ever-growing size of the datasets, obtaining efficient optimization algorithms which can be executed on parallel and distributed processing architectures is of critical importance. Numerous accelerated optimization algorithms limit to differential equations as the step size goes to zero, and the objective function typically converges to its optimal value at an accelerated rate along the trajectories of these limiting differential equations. As a result, the optimization problem can be replaced by the problem of evolving dynamics governed by appropriately defined differential equations. In Part II, we construct novel optimization algorithms via symplectic integration of carefully designed Hamiltonian systems which converge to the minimizer at an accelerated rate, both on normed vector spaces and Riemannian manifolds.

Identifying accurate and efficient models for dynamical systems based on observed trajectories is crucial for predicting and controlling their future behavior. Incorporating the structure underlying the dynamics in deep learning architectures has proven to be an efficient approach to learning structured dynamical systems. In Part III, we design novel deep learning architectures incorporating the geometric structure of nearly-periodic Hamiltonian systems and controlled Lie group Hamiltonian systems to learn structure-preserving surrogate evolution maps.

A variational framework for accelerated optimization was recently introduced on normed vector spaces and Riemannian manifolds in ^[1] and ^[2]. It was observed that a careful combination of time-adaptivity and symplecticity in the numerical integration can result in a significant gain in computational efficiency. It is however well known that symplectic integrators lose their near-energy preservation properties when variable time-steps are used. The most common approach to circumvent this problem involves the Poincaré transformation on the Hamiltonian side, and was used in ^[3] to construct efficient explicit algorithms for symplectic accelerated optimization. However, the current formulations of Hamiltonian variational integrators do not make intrinsic sense on more general spaces such as Riemannian manifolds and Lie groups. In contrast, Lagrangian variational integrators are well-defined on manifolds, so we develop here a framework for time-adaptivity in Lagrangian variational integrators and use the resulting geometric integrators to solve optimization problems on vector spaces and Lie groups.

The main idea of a geometric integrator is to adopt a geometric viewpoint of the problem and to construct integrators that preserve the geometric properties of the continuous dynamical system. For classical mechanics, both the Lagrangian and the Hamiltonian formulations can be described using the language of geometry. Due to the rich conservation properties of mechanics, it is natural to study the construction of numerical integrators that preserve some geometric properties, such as the symplectic structure, energy, and momentum maps. Such geometric structure-preserving numerical integrators exhibit nice properties compared to traditional numerical methods. This is especially true in galaxy simulations and molecular dynamics, where long time simulations are required to answer the corresponding scientific questions. Variational integrators have attracted interest in the geometric integration community as it discretizes Hamilton's principle, as opposed to the corresponding differential equation, to obtain a numerical integrator that is automatically symplectic, and which exhibits a discrete Noether's theorem. Besides classical mechanics, such an approach has also been applied to other fields, such as optimal control~\cite{junge2005discrete,leyendecker2010discrete}, partial differential equations~\cite{marsden1998multisymplectic}, stochastic differential equations~\cite{bou2009stochastic}, and so on.

In this thesis, we consider generalizations of geometric integrators that are adapted to three special settings. One is the case of stiff systems of the form, $\dot{q} = Aq + f(q)$, where the coefficient matrix $A$ has a large spectral radius that is responsible for the stiffness of the system, while the nonlinear term $f(q)$ is relatively smooth. Traditionally, exponential integrators have been used to address the issue of stiffness. In Chapter~\ref{exp}, we consider a special semilinear problem with $A=JD$, $f(q)=J\nabla V(q)$, where $J^T = -J, D^T=D$, and $JD=DJ$. Then, the system is described by $\dot{q} = J(Dq+\nabla V(q))$, which naturally arises from the discretization of Hamiltonian partial differential equations. It is a constant Poisson system with Poisson structure $J_{ij}\frac{\partial}{\partial x_i}\otimes \frac{\partial}{\partial x_j}$, and Hamiltonian $H(q) = \frac{1}{2}q^TDq + V(q)$. Two types of exponential integrators are constructed, one preserves the Poisson structure, and the other preserves energy. Numerical experiments for semilinear Possion systems obtained by semi-discretizing Hamiltonian PDEs are presented. These geometric exponential integrators exhibit better long time stability properties as compared to non-geometric integrators, and are computationally more efficient than traditional symplectic integrators and energy-preserving methods based on the discrete gradient method.

The other generalization is to Lie groups. When configuration manifold is a Lie group, we would like to utilize the group structure rather than simply regard it as embedded submanifold. This is particularly useful when codimension of the embedding is large. For the rigid body problem, the configuration space is $\mathbb{R}^3\rtimes SO(3)$, which is a Lie group. \citet{LeMcLe2005} were the first to directly use the Lie group structure of the rotation group to construct a Lie group variational integrator. In contrast, most prior approaches used the unit quaternion representation of the rotation group and applied symplectic integrators for constrained systems with the unit length constraint. In Chapter~\ref{quater}, we adopt the approach used in constructing Lie group variational integrators for rigid body dynamics on the rotation group and applied it to the unit quaternion representation. A Lie group variational integrator in the unit quaternion representation is derived, and it can be shown that our method is related to the RATTLE method applied to the rotation representation by the projection from unit quaternions to rotation matrices. The numerical results for our Lie group quaternion variational integrator are presented. The integrators constructed in Chapter~\ref{quater} are only second-order, and in Chapter~\ref{polar}, variational integrators of arbitrarily high-order on special orthogonal group $SO(n)$ are constructed by using the polar decomposition. It avoids the second-order derivative of the exponential map that arises in the traditional Lie group variational integrator method. Also, a reduced Lie--Poisson integrator is constructed. The resulting algorithms can naturally be implemented using fixed-point iteration. Numerical results are given for the case of $SO(3)$.

The last generalization is to control systems. We studied the problem of uncertainty propagation and measurement update for systems that are partially unobservable. We construct a method that satisfies the chain property that the unobservable subspace remains perpendicular to the measurement $dh$ during propagation. We characterize the unobservable subspace in terms of the group-invariance of the control system, and obtain a reduced control system on the observable variables. By decomposing the system explicitly into unobservable and observable parts $(x_N, x_O)$, the chain property can be naturally satisfied. Also, we propose a reduced Bayesian framework, where the update from the measurement is only applied to the observable variables $x_O$. In Chapter~\ref{geometric_reduce}, we consider a planar robot model, which has one odometry sensor and one camera. Odometry is used for propagation and the camera is used for measurement. In this model, the two-dimensional position as well as the orientation are all unobservable. We applied our technique to this model and performed numerical simulations. We tested this on straight line, circle, and general trajectories and found that the reduced Kalman filter that we proposed outperforms the classical Kalman filter and modifications that were proposed in the literature. In particular, it estimates the angle quite well, and as a result, yields a better estimate of the position as well.

Uncertainty quantification seeks to provide a quantitative means to understand complex systems that are impacted by uncertainty in their parameters. The polynomial chaos method is a computational approach to solve stochastic partial differential equations (SPDE) by projecting the solution onto a space of orthogonal polynomials of the stochastic variables and solving for the deterministic coefficients. Polynomial chaos can be more efficient than Monte Carlo methods when the number of stochastic variables is low, and the integration time is not too large. When performing long-term integration, however, achieving accurate solutions often requires the space of polynomial functions to become unacceptably large. This dissertation presents an alternative approach, where sets of empirical basis functions are constructed by examining the behavior of the solution for fixed values of the random variables. The empirical basis functions are evolved over time, which means that the total number can be kept small, even when performing long-term integration. We introduce this method of empirical chaos expansion, and apply it to a number of model equations, demonstrating that the computational time scales linearly with the final integration time. That is not the case for polynomial chaos in general, since achieving accuracy for long-term integration usually requires larger polynomial bases, causing a nonlinear scaling with the final integration time. We also present an analytical method that uses the dynamics of the SPDE to predict the evolution of the empirical basis functions and demonstrate how it can be applied to evolve the empirical basis functions without needing to resample realizations of the original SPDE.

Variational integrators are a class of geometric structure-preserving numerical integrators that are based on a discretization of Hamilton’s variational principle. We construct, analyze and investigate the applications of variational integrators to multisymplectic partial differential equations and to adjoint systems.

The variational structure of multisymplectic PDEs encodes both the conservation laws admitted by these systems via Noether’s theorem and multisymplecticity, a covariant spacetime generalization of symplecticity. We develop variational integrators for these systems which preserve these properties at the discrete level, in both the Lagrangian and Hamiltonian settings. In the Lagrangian setting, we utilize compatible finite element spaces to develop these variational integrators and utilize their preservation of the de Rham complex to define discrete geometric structures associated to these integrators and naturally relate them to their continuous counterparts. In the Hamiltonian setting, we utilize a discrete Type II variational principle, based on the notion of a Type II generating functional for multisymplectic PDEs, to construct structure-preserving variational integrators for multisymplectic Hamiltonian PDEs.

Adjoint systems are ubiquitous in optimization and optimal control theory since they allow for efficient computation of sensitivities of cost functionals in optimization problems and arise as necessary conditions for optimality in optimal control problems via Pontryagin’s maximum principle. Adjoint systems admit a fundamental quadratic conservation law which is at the heart of the method of adjoint sensitivity analysis; this conservation law arises from the symplectic geometry of these systems. We develop a geometric theory for continuous and discrete adjoint systems associated to ordinary differential equations and differential-algebraic equations, by investigating their underlying symplectic and presymplectic structures, respectively. We develop a Type II variational principle for such systems at the continuous level. Subsequently, we discretize this variational principle to construct variational integrators for adjoint systems which preserve the quadratic conservation law at the discrete level and thus, allow for sensitivities of cost functions to be computed exactly. We further extend this framework to the Lie group setting and develop a variational integrator based on novel continuous and discrete Type II variational principles on cotangent bundles of Lie groups.

This dissertation explores Hamiltonian variational integrators. Variational integrators are a common class of symplectic integrators, which have primarily been analyzed and constructed by discretizing Hamilton's principle. Hamiltonian variational integrators are derived by discretizing Hamilton's phase space principle and have not been studied as thoroughly. In this dissertation new error analysis theorems and other related results extend the theory of Hamiltonian variational integrators. It is shown that these two formulations of variational integrators are not always numerically equivalent, even when they analytically represent the same map. Numerical simulations show there can be important differences between these two formulations of variational integrators, particularly for averaging methods

Next, a new class of variational integrators is developed based on the Taylor method combined with an augmented shooting method. A symmetric and more computationally efficient version is also developed, as well as a comparison of Lagrangian and Hamiltonian formulations of the integrator. Error analysis results are presented, in addition a proof is given of a sufficient condition for the equivalence of a Hamiltonian and Lagrangian variational integrator. Numerical simulations are presented, as well as a discussion on the role of automatic differentiation in the implementation of Taylor variational integrators.

The last topic focuses on an adaptive framework for symplectic integrators. The Poincare transformation is used to construct an extended Hamiltonian system, which allows for variable step sizes. However, it is shown that the resulting Hamiltonian is often degenerate, and the only plausible framework for variational integrators is to use Hamiltonian variational integrators. Furthermore, the degeneracy of the Hamiltonian is discussed with regards to error analysis and the invertibility of the discrete Legendre transforms. A few monitor functions are considered, and numerical simulations demonstrate the significant gains in efficiency when using adaptive variational integrators.

In this dissertation, I will discuss and explore the various theoretical pillars re- quired to investigate the world of discretized gauge theories in a purely classical setting, with the long-term aim of achieving a fully-fledged discretization of General Relativity (GR). I will start with a brief review of differential forms, then present some results on the geometric framework of finite element exterior calculus (FEEC); in particular, I will elaborate on integrating metric structures within the framework and categorize the dual spaces of the various spaces of polynomial differential forms PrΛk(Rn). After a brief pedagogical detour on Noether’s two theorems, I will apply all of the above into discretizations of electromagnetism and linearized GR. I will conclude with an excursion into the geodesic finite element method (GFEM) as a way to generalize some of the above notions to curved manifolds.