UCLA

Online experimentation, most commonly implemented as A/B testing, evaluates product changes by randomly assigning users to different versions of a system and comparing their performance on key business metrics. Many commonly used metrics are ratio metrics, such as click-through rate (CTR). To improve statistical sensitivity, variance reduction (VR) methods are widely adopted in practice. However, ratio metrics are nonlinear functions of unit-level outcomes and often exhibit substantial heterogeneity in unit-level contributions. These features limit the applicability and performance of classical variance reduction methods developed for mean outcomes. This thesis first provides a review of VR methods for ratio metrics and then proposes a stratification-based VR method designed for settings with heterogeneous unit-level contributions. Simulation results show that the proposed method achieves strong variance reduction while remaining unbiased. Overall, this work provides a clear and practical framework for improving statistical efficiency in large-scale online experiments.