Markov chain Monte Carlo (MCMC) methods are highly desirable when the sampling distribution is intractable. Among all MCMC methods, the fundamental one is the Metropolis-Hastings algorithm. Despite its extensive application in approximating any distribution, the Markov chain often suffers from slow mixing, which then causes insufficient estimation. We address this issue by proposing modifications to the Metropolis-Hastings algorithm that, under specified conditions, induces substantial improvements in jump distances and statistical efficiency while preserving the overall quality of convergence. This dissertation starts with an introduction of the MCMC methods and continues by proposing the Efficient Conditional Metropolis-Hastings algorithm (ECMH) and a variation of ECMH under a uniform setting (ECMHu). We further investigate their properties through a series of models, including a Bivariate normal model, a Bayesian random effects model, and a Bayesian dynamic spatiotemporal model. Simulation results are compared across all algorithms.

Spatiotemporal processes are ubiquitous in the environmental and physical sciences. The complexity of these processes and a large number of observations preclude the use of traditional models such as partial differential equations, integrodifference equations, and covariance based space-time models. Alternatively, the spatiotemporal hierarchical Bayesian models are ideal in this case as it can conditionally specify the components in the model and eventually link them together through Bayes' Theorem. However, the complex and high-dimensional nature of these models prevents the direct evaluation of the posterior distribution. Instead, we can apply MCMC methods to draw samples from the posterior distribution and make Bayesian inferences. In fact, MCMC methods have revolutionized such modeling by allowing for more realistic and complicated models. As a novel application of the MCMC methods, we propose several spatiotemporal Hierarchical Bayesian models to understand the dynamic of post-fire chaparral recovery with data collected from the Angeles National Forest. This dissertation continues to investigate a particular spatiotemporal process of galaxy formation and evolution, in which the environment (cosmic web) plays a major role. However, the relation between galaxies and environment is not well understood. To this end, we propose a multi-step approach of representing galaxy formation trees as feature vectors and classifying along with galaxy properties to the environment.

Variance estimation in the context of high dimensional Markov Chain Monte Carlo (MCMC) is an interesting topic in research. Practical implications of estimating variance are limited due to inherent systematic bias both in univariate and multivariate settings. Recent advancements in high dimensional covariance matrix estimation in the MCMC setting, including works on Lugsail Batch Means (LUG-BM), have proven to improve the bias properties. Using spectral theory of estimation, we can further improve upon the bias and variance properties of the estimators. Finite sample properties of variance estimators should be studied in detail using statistical properties of the sampling bias, while accounting for the sampling error. The direction of this bias is crucial in finite sample applications. Mean Square Error (MSE) has been traditionally used to assess the quality of estimation. However, using alternate asymmetrical loss functions is recommended as they are more natural to use in applications where constructing optimal variance estimators in finite samples is required. Normality assumptions are required to make efficient use of these techniques and a careful analysis should be done to ensure the assumptions for normality are met.

Markov chain Monte Carlo (MCMC) simulations are commonly employed for estimating features of a target distribution, particularly for Bayesian inference. A fundamental challenge is determining when these simulations should stop. This dissertation begins by introducing relevant MCMC basics and discussing several existing techniques to terminate an MCMC simulation: the convergence diagnostics, using the effective sample size (ESS) as a stopping rule, and the fixed-width stopping rule (FWSR).

This dissertation continues by proposing the relative FWSRs that terminate the simulation when the width of a confidence interval is sufficiently small relative to the size of the target parameter. Specifically, we introduce two sequential stopping rules: the relative magnitude and the relative standard deviation FWSR in the context of MCMC. In each setting, we develop conditions to ensure the simulation will terminate with probability one and the resulting confidence intervals will have the proper coverage probability. The results are applicable in such MCMC estimation settings as expectation, quantile, or simultaneous multivariate estimation. We investigate the finite sample properties through a variety of examples, and provide some recommendations to practitioners.

New challenges present when the relative FWSRs are applied to terminate high-dimensional MCMC simulations. To this end, we propose using a modified relative standard deviation FWSR that terminates the simulation when the computational uncertainty is small relative to the posterior uncertainty. Further, we show this stopping rule is equivalent to stopping when the effective sample size is sufficiently large. Such a stopping rule has previously been shown to work well in settings with posteriors of moderate dimension. We further illustrate its utility in high-dimensional simulations while overcoming some current computational issues. As examples, we consider two complex Bayesian analyses on spatially and temporally correlated datasets. The first involves a dynamic space-time model on weather station data and the second a spatial variable selection model on fMRI brain imaging data. The results show the modified sequential stopping rule is easy to implement, provides uncertainty estimates, and performs well in high-dimensional settings.

As a novel application, we propose using Bayesian model selection on linear mixed-effects models to compare multiple treatments with a control. A fully Bayesian approach is implemented to estimate the marginal posterior inclusion probability for each treatment, along with the model-averaged posterior distributions. It automatically traverses the model space and identifies subsets of predictors with nonzero fixed-effects coefficients; that is, it locates the model with the highest posterior probability. The resulting marginal inclusion probabilities provide a straightforward measure of the differences between treatments and the control. Default priors are proposed for model selection and a component-wise Gibbs sampler is developed for posterior computation. The proposed method is shown to work well using simulated data and the experimental data from a longitudinal study of mouse weight trajectories.

Markov chain Monte Carlo (MCMC) methods are often used in Bayesian analysis to approximate expectation with respect to a certain distribution. Monte Carlo standard error (MCSE) can be used to determine the desired number of dependent samples, as well as to construct confidence intervals of MCMC estimates. It is usually unknown and various techniques have been suggested to estimate MCSE. A fundamental problem for these techniques is to choose an appropriate bandwidth. Previous research shows that a bandwidth proportional to $n^{1/3}$ is optimal for certain estimators. The proportional constant of $n^{1/3}$ however is unknown. As a result, $n^{1/3}$ is suggested although sub-optimal due to the missing proportional constant. $n^{1/2}$ was also considered to account for the constant but its asymptotic performance is worrisome.

Existing literature mostly considered the above issues under univariate setting but Bayesian analysis normally involves multiple parameters. Computation time is a major challenge to estimate multivariate MCSE, where large amount of dependent samples are involved. Therefore multivariate estimators of MCSE that delivers fast and accurate estimation is desirable.

This dissertation addresses the above two problems. I consider a family of estimators and established conditions under which their mean squared consistency exist. The results have a direct application in bandwidth selection and also suggests a bandwidth proportional to $n^{1/3}$. The proportional constant can be obtained based on the proof of mean squared consistency. I further suggest to approach the proportional constant with a pilot estimate. The suggested bandwidth shows superior performances compared with $n^{1/3}$ or $n^{1/2}$. The above results are established under multivariate setting which not only covers the long-standing univariate bandwidth selection problem, but also brings up the multivariate question with a solution.

To tackle the computational problem in multivariate setting, I propose a family of new estimators and prove strong consistency of these estimators. The new estimators are fast to compute and have comparable performances to spectral variance estimators with slightly inflated variance.

Markov chain Monte Carlo (MCMC) is a sampling technique that allows for estimating features of intractable probability distributions. Output analysis of MCMC samples aims to assess the quality of the sampler and the resulting estimates. We provide an example based overview of current best practices using visualizations and the estimation of features of interest using Markov chain central limit theorems. Most features of interest are functionals of expectations or quantiles. The estimation of quantiles has thus-far been limited to univariate theory and marginal limiting distributions, while the estimation of expectations enjoys multivariate approaches through a joint limiting distribution. In this work, we provide an extension to jointly estimate combinations of functionals of expectations and quantiles through a joint limiting distribution for the Monte Carlo error. We use this limiting distribution to establish a procedure for finding sets of simultaneous intervals forming confidence regions of approximately correct coverage probabilities. These simultaneous intervals motivate a class of visualizations for Monte Carlo errors for a broad class of estimation procedures for which a multivariate normal limiting distribution holds. Finally, we consider MCMC sample size through sequential stopping rules which terminate simulation once the Monte Carlo errors become suitably small. We develop a general sequential stopping rule for combinations of expectations and quantiles from Markov chain output and provide a simulation study to illustrate the validity.