Proceedings of the Annual Meeting of the Cognitive Science Society

We show that publishing results using the statistical signif-icance filter—publishing only when the p-value is less than0.05—leads to a vicious cycle of overoptimistic expectationof the replicability of results. First, we show analytically thatwhen true statistical power is relatively low, computing powerbased on statistically significant results will lead to overesti-mates of power. Then, we present a case study using 10 exper-imental comparisons drawn from a recently published meta-analysis in psycholinguistics (J ̈ager et al., 2017). We show thatthe statistically significant results yield an illusion of replica-bility. This illusion holds even if the researcher doesn’t con-duct any formal power analysis but just uses statistical signifi-cance to informally assess robustness (i.e., replicability) of re-sults.