Department of Economics

Numerous solutions have been proposed to address the replication crisis, in which numeroushigh-profile empirical research studies cannot be replicated by other research teams. One possibleexplanation is that researchers have the option to adjust their data analyses after viewingthe results, inflating false positive rates. One popular solution is study preregistration, the practiceof developing the data analysis plan before the data is collected. However, preregistrationsonly alleviate replication problems if researchers are held accountable to their analysis plans.Across two related studies, we explore the effectiveness of preregistration in its current form.In Study 1, we audit recent preregistered publications from a major psychology journal andobserve deviations in 19 of 32 papers. In Study 2, we simulate the effects of generic deviationson the false-positive rate. We find that deviations that run more or more varied tests causelarger changes, tripling the false-positive rate in the most extreme case. We note that auditingpreregistrations requires an inconsistent amount of time depending on their length and format,which we suspect contributes to the enforcement issues we observe. We suggest that researchersand journals alike adopt the asPredicted.org template for preregistrations.