Proceedings of the Annual Meeting of the Cognitive Science Society

Laypeople and experts struggle to reason accurately about scientific evidence. Here, we tested whether laypeople noticed methodological flaws related to participant sampling and selection. Across four studies (n = 1,031), we investigated whether lay readers could identify sampling and selection errors related to non-random assignment and limited generalizability in news-style summaries of scientific research, and whether individual differences in thinking dispositions predicted success. Although participants were sensitive to errors when explicitly highlighted in texts, they did not notice most flaws (the exception being biased samples) when comparing articles or judging individual articles. Even when participants identified an article with a methodological flaw, few could articulate the specific issue, suggesting a lack of conceptual understanding. Across tasks, actively open-minded thinking consistently predicted better error detection as reflected by article ratings and free responses, while the effects of cognitive reflection and numeracy were less robust.