UC Berkeley

Reading comprehension (RC) is often viewed as a multi-faceted, multi-layered construct (Broek & Espin, 2012; Duke, 2005; Graesser & McNamara, 2011; Perfetti & Stafura, 2014), which manifests through complex interactions among three broad factors: the reader, the passage, and the task, all residing in a particular socio-cultural context (RAND Reading Study Group, 2002). Drawing on the item difficulty modeling paradigm, this study examined how these three factors as well as their interactions affected comprehension difficulty. Specifically, the study used explanatory item response models (De Boeck & Wilson, 2004) to analyze a vertically-scaled item response matrix from an operational online assessment, which included a wide range of readers (n=10,547) as well as of passages (n=48), covering grades 1 through 12+. Analyses indicated that it is text features, as measured by computational text analyzers, rather than task features as coded by human raters, that explained over half the variance in item difficulty, after controlling for student general vocabulary knowledge. Specifically, sentence length, word frequency, syntactic simplicity, and temporality (i.e., the extent to which the text has time markers) were found to significantly affect comprehension difficulty in both model building and cross validation analyses. Further, small but significant interaction effects were found, indicating that these textual effects were moderated by student general vocabulary knowledge as well as task demands as captured by item types. In general, readers with higher vocabulary knowledge benefitted more from traditional textual affordances (e.g., shorter sentences, familiar words) than their peers with lower vocabulary knowledge, especially when questions asked them to recall specific localized information without accessing the source passage. However, a reverse effect was found with temporality: passages with more time markers helped low vocabulary readers, while it was low temporality passages that helped high vocabulary readers. The implications of these findings as well as their limitations are discussed as they relate to the measurement of RC and to instructional practice.