In the year 2000, Gerber and Green published the results of a field experiment which examined the impact of electoral campaigns on voter participation. Since this landmark work, more than one hundred similar studies have appeared in the political science literature. These randomized controlled trials are usually conducted within get-out-the-vote (GOTV) drives seeking to increase voter turnout. The surge in GOTV experiments was partly due to a statistical innovation that preceded Gerber and Green’s publication, the average treatment effect for the treated (ATT), which allowed researchers to compare directly those treated to a similar group that was assigned to the control.

In this research we focus on settings common to many social science field experiments such as those of GOTV studies, where participants may comply, or not, with the treatment protocol assigned by the experimenter. For experiments with binary outcomes, we show that each individual in the study may be classified as one of a finite number of distinct types. We call these behavioral types because they characterize the individual’s complete reaction, their measured response and how they receive treatment, to the assignment of each possible experimental group. In this context, the data is generated by randomly allocating these various behavioral types to the different levels of treatment. Thus, the model is parameterized by the unknown proportions of the different behavioral types so that many statistical aspects of the experiment, such as commonly studied average treatment effects, may be written as a function of these proportions.

Viewing the data as generated by these behavioral types changes the analysis of the experiment in two ways. First, it changes the perspective on what is being estimated. Instead of finding a particular treatment effect, the ultimate goal can be seen as estimating proportions of behavioral types. With this frame of reference, the effect of a certain treatment will be most accurately represented as the fraction of the experimental sample for which the treatment has an effect. Second, by clarifying the underlying data generating process, a behavioral-types approach directs the resulting statistical analysis.

We use a well cited example to introduce behavioral types before providing formal definitions. We present the ATT as a case study for how to apply a behavioral-types approach for a design known to many social science researchers. The understanding of the data generating process allows us to evaluate the bias and variance of the ATT estimator, and we show the variance depends on the choice of the sampling assumptions. We then provide rigorous definitions of a behavioral type and of restrictions which reduce the number of behavioral types in a population to a number where the proportions of each type may be estimated. We present three experimental designs and present a strategy to identify the proportions of each type and elucidate how treatment effects may be found from the proportions.

A behavioral-types approach is well suited to multi-treatment experiments because it distills often complex designs into an estimation problem of a manageable number of types. We apply the behavioral types approach to four published social science field experiments involving multiple levels of ordered treatment. For each, we show how the interpretations and the statistical analyses differ with a behavioral-types approach, and can lead to different conclusions. Through the applications we illustrate how behavioral types provides insight into a range of experimental designs, such as those with spillover effects or partial ordering of treatment levels.

For two of the four applications we further examine the issue of joint significance by constructing multi-dimensional confidence regions for the proportion of behavioral types. We find that normal approximation methods perform poorly, but the shortcomings can be corrected by the bootstrap. However, even the bootstrap regions may not attain the desired coverage levels, so we adjust our regions using a double bootstrap. We discuss other methods that merit further exploration.

Methods for clustering health care claims into episodes of care are important tools for data analysis in evaluating health care outcomes and methods of payment. In this research, we implement two statistical methods (1) using an ensemble classifier, Random Forests, to estimate the strength of relationship in pairs of Medicare claims, and (2) using a sequence model and an Expectation Maximization (EM) algorithm.

Previous researchers into episode of care clustering have implemented other methods, some based on decision rules and others based on statistical methods. Other research in natural language processing, particularly conversation disentanglement, has developed methods that were inspirational for the methods that we implemented in this research.

We acquired two sets of Medicare claims data, one containing claims from 2006 and 2007 for 1.9 million patients (a 5 percent sample), and the other containing claims from 2007 and 2008 for 250 thousand patients. We tested our two statistical methods on claims for 50 randomly selected patients who were age 65 and older, using independent annotations by three licensed nurse practitioners. We found that the method using Random Forests outperformed the sequence model and achieved accuracy comparable to the nurse practitioner annotators. Future research into episode of care clustering might incorporate more extensive data on clinical relationships and create a more flexible representation of episode clusters, such as hierarchies and phases of care.

Many problems in statistics are inherently discrete. When one of these problems also contains an optimization component, integer programming may be used to facilitate a solution to the statistical problem. We use integer programming techniques to help solve problems in the following areas: optimal blocking of a randomized controlled experiment with several treatment categories and statistical auditing using stratified random samples.

We develop a new method for blocking in randomized experiments that works for an arbitrary number of treatments. We analyze the following problem: given a threshold for the minimum number of units to be contained in a block, and given a distance measure between any two units in the finite population, block the units so that the maximum distance between any two units within a block is minimized. This blocking criterion can minimize covariate imbalance, which is a common goal in experimental design. Finding an optimal blocking is an NP-hard problem. However, using ideas from graph theory, we provide the first polynomial time approximately optimal blocking algorithm for when there are more than two treatment categories. In the case of just two such categories, our approach is more efficient than existing methods. We derive the variances of estimators for sample average treatment effects under the Neyman-Rubin potential outcomes model for arbitrary blocking assignments and an arbitrary number of treatments.

In addition, statistical election audits can be used to collect evidence that the set of winners (the outcome) of an election according to the machine count is correct---that it agrees with the outcome that a full hand count of the audit trail would show. The strength of evidence is measured by the p<\italic>-value of the hypothesis that the machine outcome is wrong. Smaller p<\italic>-values are stronger evidence that the outcome is correct. Most states that have election audits of any kind require audit samples stratified by county for contests that cross county lines. Previous work on p<\italic>-values for stratified samples based on the largest weighted overstatement of the margin used upper bounds that can be quite weak. Sharper $p$-values than those found by previous work can be found by solving a 0-1 knapsack problem. We also give algorithms for choosing how many batches to draw from each stratum to reduce the counting burden.

Mathematical models are often used to forecast interesting scientific phenomena. These models may provide a good approximation for the system being studied, but they are rarely perfect. Whether due to scientific misunderstandings or the necessity of numerically approximating the solution to the model, the mathematical models fail to exactly replicate the true dynamics of the system. This "model error" leads to forecast errors and degradation of forecast skill. The primary goal of this dissertation is to develop methods to correct a mathematical model with observed data, improving the accuracy of forecasts generated by the model. Correcting for model error is particularly difficult when the system is high-dimensional and when the state transition model is nonlinear. The methods developed in this dissertation are robust to both of these difficulties.

The problem of estimating model error may be phrased as a parameter estimation problem within a filtering context. Many parameter estimation methods have been developed for the particle filter (PF), a filter with strong theoretical underpinnings and well understood large sample properties. Unfortunately, the PF fails when the state dimension is large. Instead, scientists who filter high-dimensional state-spaces typically use the ensemble Kalman filter (EnKF). This algorithm is effective in practice, but its large sample properties remain to be proven and methods for parameter estimation exist only in special cases. We make a newfound connection between these two algorithms: the EnKF is a regularized PF with additional approximations that have yet to be fully justified from a theoretical standpoint. Building on this connection, we introduce a novel algorithm that combines the best features of the EnKF and the PF. The EnKF is used to filter the high-dimensional state while an auxiliary PF is used to estimate the low-dimensional parameters; we call this filter the EnKF-APF. Using the Lorenz 2005 system, we demonstrate that the true parameter values are better captured with the EnKF-APF algorithm than with the EnKF.

Having developed a robust method for parameter estimation within a filtering context, we return to the question of model error. Following previous work in the literature, we focus on a linear correction to the mathematical model. If the state has dimension d, the number of parameters in the linear correction is on the order of d^2. The EnKF-APF algorithm does not allow for the estimation of such high-dimensional parameters, and even if it did, an enormous amount of data would be required to provide good estimates. To make the problem tractable, we assume the problem of interest exhibits a large degree of spatial structure and exploit this special structure to reduce the dimensionality of the linear correction. We propose to correct the model error with a low-rank linear correction, inspired by methods used to model multivariate spatial processes in the geostatistics literature. We demonstrate our method on the Lorenz 2005 system by applying the EnKF-APF algorithm to estimate the parameters of the low-rank linear correction in both a batch and online manner. Although the model error in this test is nonlinear, we demonstrate that the proposed low-rank linear correction provides better forecasts.