Proceedings of the Annual Meeting of the Cognitive Science Society

What sorts of graphical formats best convey effect size and degree of certainty of a finding? Confidence intervals are commonly used to show uncertainty, yet lay people and experts fail to correctly interpret their meaning. There has been a recent push to present individual data points rather than only presenting aggregated summary statistics (e.g., means, confidence intervals, lines of best fit). But it is unclear how well people can aggregate raw data presented in a graphical format. Across two studies, we presented participants with hypothetical study outcomes of two independent groups in three graph styles: dot plots, mean with 95% confidence interval (CI) plots, combined plots, and bee plots. We asked participants to make judgments about the effect size using the Common Language Effect Size or Bayes Factors. Participants were more likely to underestimate effect sizes and Bayes Factors for dot plots and bee plots compared to mean + 95% CI plots and combined plots. These findings suggest that people have trouble making statistical inferences when presented with raw data points in graphs.