UC Santa Barbara

Given the inherently subjective nature of moral content analysis (MCA), high intercoder reliability (ICR) statistics reported in extant MCA literature have been questioned. This paper argues that greater precision in identifying morally relevant content cues via multidimensional coding schemes can minimize the likelihood of subjective coder interpretations being applied during the coding task, thereby yielding high ICR statistics. Accordingly, and derived from Moral Foundations Questionnaire - 2 (MFQ-2), we offer the seminal version of the Moral Foundations Content Codebook (MFCC-1) which provides a rule-based framework for the identification of morally relevant cues, specifically in visual content. Our findings support the claim that a multidimensional MCA codebook, as opposed to a unidimensional one such as the coding manual for Model of Intuitive Morality and Exemplars (MIME), yields comparatively greater ICR statistics. However, these still remain below thresholds for minimum acceptable reliability. In our post-hoc analyses, we find that acceptable reliability for MFCC-1 can also be achieved but only after constraint adjustment procedures that take into account the degree of coder disagreement (i.e., slight versus severe). Additionally, we observe enhancement of discriminant, predictive, and external validities when using ratings as derived from MFCC-1. We discuss these findings and provide explanations inspired from the cognitive bias literature that speculatively rationalize why coders using unidimensional MCA codebooks, in general, can be expected to perform worse than their counterparts.