Fitting multivariate autoregressive (AR) models is fundamental for analysis of time-series data in a wide range of applications in science, engineering, econometrics, signal processing, and data-science. This dissertation considers the problem of learning a $p$-lag multivariate AR generalized linear model (GLM). In this model, the state of the time-series at each time step, conditioned on the history, is drawn from an exponential family distribution with the mean parameter depending on a linear combination of the last $p$ states. The problem is to learn the linear connectivity tensor from a single observed trajectory of the time-series. We provide non-asymptotic error bounds on the regularized Maximum Likelihood estimator in high dimensions.

We focus on the sparse tensor setting, which arises in applications where there exists a limited number of direct connections between variables. For such problems, $\ell_1$-regularized maximum likelihood estimation (or M-estimation more generally) is often straightforward to apply and works well in practice. The M-estimator can be posed as a convex optimization problem and hence can also be solved efficiently.

However, the statistical analysis of such methods is difficult due to the feedback in the state dynamics and the presence of a non-linear link function, especially when the underlying process is non-Gaussian. Our main result in Chapter 3 provides a bound on the mean-squared error of the estimated connectivity tensor as a function of the sparsity and the number of samples, for a class of discrete multivariate AR($p$) GLMs, in the high-dimensional regime. Importantly, the bound indicates that, with sufficient sparsity, consistent estimation in cases where the number of samples is significantly less than the total number of free parameters. Towards proving the main result, we present a general framework to establish the Restricted Strong Convexity (RSC) property for time-averaged loss functions often seen in time-series analysis. We also derive new concentration inequalities of functions of discrete non-Markovian random variables. These intermediate results may be of independent interest to the reader.

Modern machine learning techniques rely heavily on iterative optimization algorithms to solve high dimensional estimation problems involving non-convex landscapes. However, in the absence of knowing the closed-form expression of the solution, analyzing statistical properties of the estimators remains challenging in most cases. This dissertation provides a framework, called Multi-layer Vector Approximate Message Passing (ML-VAMP), for analyzing optimization-based estimators for a broad class of inverse problems. This framework is based on new developments in random matrix theory. Importantly, it does not rely on convex analysis and applies more broadly to non-convex optimization problems.

The ML-VAMP framework enables exact analysis in a certain high dimensional asymptotic regime for several problems of interest in machine learning and signal processing. In particular, the following problems have been explored in some detail,- Reconstruction of signals from noisy measurements using deep generative models, - Generalization error of learned one-layer and two-layer neural networks, \label{prob:nn} to demonstrate the analytical capabilities of the framework.

Using this framework we can analyze the effect of important design choices such asdegree of overparameterization, loss function, regularization, initialization, feature correlation, and a mismatch between train and test data in several problems of interest in machine learning.