UC Santa Barbara

In the social sciences, data is often generated from responses to self-report survey instruments. Thus, researchers are often concerned with investigating survey instrument quality, more commonly known as validity. Evidence of instrument quality is typically presented quantitatively using correlational methods such as confirmatory factor analysis (CFA), regression, and coefficient alpha. Rarely do researchers engage in extensive pretesting of survey items or present evidence that survey items are understood as intended by the population of interest. Despite potential ambiguity in score meaning, these assessments are commonly used in high stakes settings such as measuring treatment efficacy in medical trials or evaluating the suitability of applicants for careers.

In this three-paper dissertation, I highlight some flaws with the current approaches to scale validation and present two new methods that will hopefully lead to improvements in survey quality. I begin by introducing the Response Process Evaluation method; a standardized framework for iteratively pretesting multiple versions of survey items and generating individual item level validity reports. Next, I discuss the improper over-generalization of a set of approximate fit index cutoff values for CFA models, and introduce a simulation-based, model-specific alternative called Dynamic Fit Index (DFI) cutoffs which have been made easily accessible in a new point-and-click software. I conclude by reviewing several of the most used scales in education and psychology and examining the types of validity evidence presented in defense of their use. My goal is to encourage researchers to think more critically about validity evidence and consider revisiting best practices in validation.