UC Berkeley

Simulations play a crucial role in the modern scientific process. Yet despite (or due to) their ubiquity, the Data Science community shares neither a comprehensive definition for a high-quality study nor a consolidated guide to designing one. Inspired by the Predictability - Computability - Stability (PCS) framework for 'veridical' Data Science, we propose six MERITS that a Data Science simulation should satisfy. Modularity and Efficiency support the Computability of a study, encouraging clean and flexible implementation. Realism and Stability address the conceptualization of the research problem: How well does a study Predict reality, such that its conclusions generalize to new data/contexts? Finally, Intuitiveness and Transparency encourage good communication and trustworthiness of study design and results. Chapter 1 of this manuscript elaborates these quality criteria and presents a conceptual framework and guidelines for designing simulations that satisfy them. Chapters 2 and 3 describe two projects in which we leverage in-depth, MERITS-driven simulation studies to solve real-world problems in the respective fields of Metabolomics (analyzing mass spectra belonging to complex plant-exudate samples) and Political Science (scrutinizing the efficacy of redistricting rules for the curtailment of gerrymandering). With these contributions, we seek to (a) distill and enrich the best practices of simulation across disciplines into a cohesive recipe for trustworthy, veridical Data Science; and to (b) leverage simulation studies embodying those ideals to solve real-world problems in the applied sciences.