- Main
Design and Inference of Randomized Experiments for Modern Applications
- Wu, Yuhang
- Advisor(s): Zheng, Zeyu
Abstract
Randomized experiments are widely used across various fields and are often referred to by different names, such as online controlled experiments or A/B tests. With the rapid development of big data and machine learning, along with the increasing demand for applications in scenarios such as online platforms, there is growing interest in new challenges and tools in randomized experiments that differ significantly from classical approaches. In this dissertation, we discuss several problems on design and inference of randomized experiments for modern applications.Chapter 2 discusses my work on randomized experiments under non-stationarity. Non-stationarities can often arise nonparametrically in key metrics in experiments. In this work, we develop a nonparametric stochastic model to capture non-stationarities in experiments, build a limiting regime to facilitate asymptotic analysis, provide a new estimator and a new design of experiments. We show that our estimator achieves semi-parametric variance lower bound, and when our new design is integrated to standard experiments to generate outcomes, even simple estimators can be semi-parametrically efficient.Chapter 3 discusses my work on joint statistical inference and convex optimization for experiments. We focus on scenarios that the treatment plan is not a single plan, but instead encompasses an infinite continuum of plans indexed by a continuous treatment parameter. The experimenter not only needs to provide valid inference, but also needs to handle the task of optimizing treatment parameter. We find that classical stochastic optimization algorithms fails to provide valid inference, and we fix this issue by providing a new algorithm that can handle dual tasks of inference and optimization.Chapter 4 discusses my joint work with Chenyu Zhang and Professor Nian Si on debiasing randomized experiments in two-sided platforms. For two-sided platforms, while randomized experiments are commonly employed on the user-side to optimize platform features, seller-side experiments present unique challenges due to interference effects that can bias treatment effect estimation. In this work, we introduce a novel estimator to adjust for interference effects via a shrinkage mechanism. Under the multinomial logit model, we theoretically establish that our estimator reduces bias compared to naive methods.Chapter 5 discusses my work on randomized experiments with constraints on subpopulations. In this work, we formulate this scenario as a problem of best arm identification with fairness constraints on subpopulations. The BAICS problem aims at correctly identify, with high confidence, the arm with the largest expected reward from all arms that satisfy subpopulation constraints. We analyze the complexity of this problem using tools in best arm identification, and solve it by proposing a new algorithm that achieves the optimal asymptotic sample complexity.