In social and behavioral sciences, assessment instruments that are made up of multipleitems (e.g., educational tests, clinical scales) are often used to measure latent variables. Item response theory (IRT) has been employed increasingly frequently in the analysis of these assessment data. Respondents in these fields (e.g., students, patients) often belong to two or more than two non-nested groups simultaneously, yielding the so-called item level data with cross-classified structures. The present study is motivated by the need to appropriately model these item-level data with cross-classified structures, which are prevalent in education and allied disciplines such as psychology.

To achieve the research goal, this study introduces a cross-classified multilevel IRTmodel. The proposed model takes into account both item properties and cross-classified structures. This study also proposes to apply the Metropolis-Hastings Robbins-Monro (MH-RM; Cai, 2008a, 2010b, 2010a) algorithm to address the computational complexities in estimating multilevel latent variable models with crossed random effects, of which the proposed model is a special case. Simulation studies are conducted to evaluate the performance of the MH-RM algorithm in terms of estimating the proposed model. The cross-classified multilevel IRT model is also applied to student evaluations of teaching (SET) collected in a large public university.

This study fills the gap in the IRT literature by introducing the cross-classified multilevelIRT model and an associated estimation algorithm. From a substantive perspective, it also improves the current practice in social and behavioral sciences by introducing an approach that properly models item-level data with cross-classified structures.

Respondents in social and behavioral studies often belong to two or more non-nestedhigher-level groups of aggregation simultaneously, yielding the so-called cross-classified data structure. For example, in education, students belong to the schools they attend and the neighborhoods they live in, and there exists no exact nesting between the schools and neighborhoods. The cross-classified multilevel model (CCMM; Goldstein, 1994; Rasbash & Goldstein, 1994) was introduced as an extension of the standard multilevel model to accommodate the prevalent cross-classified data. The CCMM has been mainly applied in education to study the impacts of various contexts on certain outcomes, such as the influence of schools and neighborhoods on smoking behaviors among adolescents (Dunn, Richmond, Milliren, & Subramanian, 2015). However, applications of the CCMM in other fields are relatively scant and little-known. One potential reason for this lack of applications could be the limited availability of software programs that allow the easy fit of the CCMM.

To advocate more applications of the CCMM in a broader spectrum, in this article, wefirst show the connections between the CCMM and several widely used psychometric models, including the random effect item response theory (IRT) model (Van den Noortgate, De Boeck, & Meulders, 2003), the model for rater effects (e.g., Murphy & Beretvas, 2015), the multitrait-multimethod (MTMM) model (Campbell & Fiske, 1959), and the generalizability theory (G-theory) model (Shavelson & Webb, 1991). Then we review a few modern applications of the CCMM, such as its applications to meta-analyses and social network analysis (SNA).

To address the issue of software programs, we introduce a flexible and efficient Rpackage PLmixed (Jeon & Rockwood, 2017), and show how the above-mentioned related models and applications of the CCMM can be estimated with PLmixed and other existing R packages. Finally, we conclude that the CCMM would be applied more broadly with the support of computer software such as PLmixed.

Background

Methods

Results

Conclusions