UC Davis

The past decade has brought to light many questions and concerns about the validity of psychological research. In chapter 1, I argue that the field of social and personality psychology must reckon with the wave of doubts about the credibility of our research that emerged during the replication crisis and credibility revolution in the 2010s. To do so, we must take stock of the state of the field and empirically evaluate whether self-correction has occurred before declaring the crisis to have passed. I propose an agenda for metascientific research and review approaches to empirically evaluate and track where we are as a field (e.g., analyzing the published literature, surveying researchers). I describe one such project, SPPSPSSPP, underway in our research group and emphasize the need for empirical evidence to evaluate the credibility of research in social and personality psychology.

Validity is a critical component of research quality, and one that is both paramount and complicated for a field to assess. In chapter 2, I introduce a tool (seaboat.io) to aid researchers and reviewers in identifying potential threats to the validity of empirical research. This tool was developed through an iterative consensus-based process of eliciting expert feedback to select potential validity threats that are most common and most serious in psychological science. Reviewers can visit seaboat.io to identify validity threats relevant to the research they are evaluating and generate a report that can be shared alongside peer traditional review reports or used in post-publication peer review.

In chapter 3, I investigate researchers’ in social and personality psychology perceptions of the state of the field and of the published literature. To explore how researchers perceive the field to have changed over time, I compare their perceptions of articles published in 2010, just before the advent of the replication crisis, vs. articles published a decade later. I also examine researchers’ perceptions of their own work, what qualities they consider important when evaluating research quality, and explore individual differences among researchers’ perceptions. Overall, these findings indicate that researchers perceive the quality of the published literature to have improved in many ways over the last decade, where significant strides are thought to have been made, and what weaknesses remain.