Proceedings of the Annual Meeting of the Cognitive Science Society

Researchers typically have a fair amount of freedom when it comes to data processing and analysis selection. In many instances, there isn't one correct way to, for example, deal with outliers, which gives rise to a multitude of reasonable analysis pathways, each with its own outcome. Computational advances provide researchers with a unique opportunity to view the impact of such researcher degrees of freedom on the results from a study. Multiverse analyses involve the computational analysis of all these potential pathways, which can demonstrate the robustness of a particular phenomenon, or the lack thereof. However, even though multiverse analyses are less susceptible to biases compared to the typical single-pathway approach, it is still possible to selectively add or omit pathways. To address this, we propose a more principled approach to conducting multiverse analyses, and illustrate how it can be applied using the Semantic Priming Across Many Languages project.