UCLA

Annotated data are central to text-as-data methods in political science. To resolve annotator disagreement, researchers typically rely on the majority vote, collapsing multiple judgments into a single label. While effective for objective tasks, this approach is less well suited to subjective classification tasks, where disagreement often reflects meaningful variation in human interpretation rather than measurement error. This thesis introduces a soft-label approach that treats annotator disagreement as an informative signal rather than noise. Instead of assigning a single label to each text, soft labels represent each document’s annotations as probability distributions over categories, preserving heterogeneity in human judgment. Using simulations that vary the number of annotators, categories, and training sample sizes, I show that soft-label models outperform majority-vote models under conditions of subjectivity and class complexity. A case study of stance detection toward China in U.S. news further shows improved performance across both conventional classification metrics and distribution-sensitive evaluation measures.