In social and behavioral sciences, assessment instruments that are made up of multipleitems (e.g., educational tests, clinical scales) are often used to measure latent variables. Item response theory (IRT) has been employed increasingly frequently in the analysis of these assessment data. Respondents in these fields (e.g., students, patients) often belong to two or more than two non-nested groups simultaneously, yielding the so-called item level data with cross-classified structures. The present study is motivated by the need to appropriately model these item-level data with cross-classified structures, which are prevalent in education and allied disciplines such as psychology.

To achieve the research goal, this study introduces a cross-classified multilevel IRTmodel. The proposed model takes into account both item properties and cross-classified structures. This study also proposes to apply the Metropolis-Hastings Robbins-Monro (MH-RM; Cai, 2008a, 2010b, 2010a) algorithm to address the computational complexities in estimating multilevel latent variable models with crossed random effects, of which the proposed model is a special case. Simulation studies are conducted to evaluate the performance of the MH-RM algorithm in terms of estimating the proposed model. The cross-classified multilevel IRT model is also applied to student evaluations of teaching (SET) collected in a large public university.

This study fills the gap in the IRT literature by introducing the cross-classified multilevelIRT model and an associated estimation algorithm. From a substantive perspective, it also improves the current practice in social and behavioral sciences by introducing an approach that properly models item-level data with cross-classified structures.

Test forms are often modified in order to accommodate various special populations or situations where administration of the original test forms is infeasible. While this practice aims to promote fairness for all, the goal can only be met if they are coupled with a systematic method for obtaining comparable scores across test forms. Numerous psychometric barriers stand in the way. For example, the limited sample size of the students taking modified test forms can prevent the usage of standard calibration and linking procedures. One particular area in which these issues are pronounced is the English language proficiency (ELP) assessment for students who are blind or have low vision, and consequently take modified Braille test forms developed under English Language Proficiency Assessment for the 21st Century (ELPA21).

To address these bottlenecks, this study proposes a method for moderated item calibration. A unified cross-classified random effects model that jointly utilizes item response data to the original test form and judgmental data provided by expert raters is developed to revise item parameters for scoring modified test forms. Estimation of this new model is performed using the Metropolis-Hastings Robbins-Monro algorithm (MH-RM; Cai, 2008, 2010a, 2010b). The method is programmed in R (R Core Team, 2018) and its implementation strategies are discussed in detail. The proposed model is applied to Braille test forms in ELPA21, and its performance is compared to the common practice. Simulation study validates the modeling framework and provide guidance for future data collection and study design. More generally, this study is significant because of its broad adaptability to 1) any special population for whom direct item calibration or standard linking is not feasible, and 2) any operational or research setting where field testing cannot be conducted because of resource/sample size constraints.

In item response theory (IRT), the underlying latent variables are typically assumed to be normally distributed. If the assumption of normality is violated, the item and person parameter estimates can become biased. Therefore, it is necessary in practical data analysis situations to examine the adequacy of this assumption in an effective manner. There is a recent surge of interest in limited-information overall goodness-of-fit test statistics for IRT models (see e.g., Cai, Maydeu-Olivares, Coffman, & Thissen, 2006; Joe & Maydeu-Olivares, 2010; Cai & Hansen, 2013), but their appropriateness for diagnosing latent variable distributional fit has not been studied.

The approach undertaken in this research is to use summed score likelihood based indices. The idea itself is not new (see e.g., Ferrando & Lorenzo-Seva, 2001; Hambleton & Traub, 1973; Lord, 1953; Ross, 1966; Sinharay, Johnson, & Stern, 2006; Thissen & Wainer, 2001), but this study recasts the problem using the framework of limited-information goodness of fit testing. The summed score based indices can be viewed as a particular form of reduction of the full underlying multinomial that are potentially sensitive to the latent variable distributional misspecifications.

Results from a pilot study (Li & Cai, 2012) show that summed score likelihood based indices enjoy high statistical power for detecting latent variable distributional assumption violations, and are not sensitive (correctly) to other forms of model misspecification such as unmodeled multidimensionality. Meanwhile, the limited-information overall fit statistic M2 (Maydeu-Olivares & Joe, 2005) has relatively low power against latent variable non-normality. However, technically the statistical indices proposed by Li and Cai (2012) don't follow an exactly chi-squared distribution. They proposed a heuristic degrees of freedom adjustment, but more rigorous justifications could be developed along the lines of Sattora-Bentler type moment adjustment popular in structural equation modeling (Satorra & Bentler, 1994). In IRT, the moment adjustment approaches have been used by Cai et al. (2006) and Maydeu-Olivares (2001).

The major methodological contributions of my dissertation come from simulation studies that examine the calibration and power of the moment adjusted test statistics across various conditions: number of items, sample size, item type, generating latent variable distribution, and the values of generating item parameters. The performance of these fit statistics are also compared with M2. Simulation study results show that the proposed moment-adjusted statistics improve upon the unadjusted statistics in the null and alternative conditions, especially when generating item parameters are dispersed. Finally, performance of the indices is illustrated with empirical data from educational and psychological assessment development projects.

Cognitive diagnosis models (see, e.g., Rupp, Templin, & Henson, 2010) have received increasing attention within educational and psychological measurement. The popularity of these models may be largely due to their perceived ability to provide useful information concerning both examinees (classifying them according to their attribute profiles) and test items (describing the particular attributes that are relevant to or required in order to achieve a certain response). However, the validity of such information may be undermined when diagnostic models are misspecified.

This research focuses on one aspect of model misspecification: violations of the local item independence assumption. Potential causes of dependence are examined, with a particular focus on those causes unrelated to the attributes a diagnostic test is intended to measure. Ignoring such dependencies, as is the standard practice in fitting traditional diagnostic models, may lead to biased estimates of model parameters and misclassification of examinees.

An alternative to traditional diagnostic models is presented, in which random effects are included in order to account for these nuisance dependencies. This approach is already well-established in item factor analysis, serving as the basis for the testlet response model (Wainer, Bradlow, & Wang, 2007), random intercept item factor model (Maydeu-Olivares & Coffman, 2006), item bifactor model (Cai, Yang, & Hansen, 2011), and two-tier item factor model (Cai, 2010), among others.

The resulting hierarchical diagnostic item response model maintains the desirable properties of traditional diagnostic models (e.g., the classification of examinees with respect to fine-grained cognitive attributes), while allowing for greater complexity in the underlying response process. Importantly, the model may be estimated efficiently — even for models with a large number of nuisance variables — using an analytical dimension reduction technique described by Gibbons and Hedeker (1992).

In recent years, the importance of formative assessments has been emphasized within educational measurement. This type of assessment often includes multiple correlated sub-domains and a hierarchical structure among the proficiencies. In this dissertation, several multidimensional CAT procedures are investigated to improve the measurement aspects of diagnostic testing and to better match the psychometric models to the test structure.

Five factors are manipulated with higher-order IRT models and hierarchical IRT models: (1) the different correlation conditions between two primary factors (low, medium, and high), (2) the number of group factors per primary factor (two and four), (3) the number of items (40, 80 and 160), (4) the item selection method (MFI and Bayesian), and (5) the proficiency score estimation method (MLE and EAP). Three outcome measures, including correlations between true and estimated proficiency scores, Root Mean Square Error (RMSE) of estimated proficiency scores, and Standard Errors (SE) are computed totaling 192 different conditions.

As expected, the correlation between true and estimated proficiency scores increase while RMSE and SE decrease when the test length correlation between two primary factors increase under different correlations among the factors, different item selection methods and different scoring methods. In overall, the higher-order IRT model CAT has an advantage over the hierarchical IRT model CAT when we need scores for the primary factors. On the other hand, if test designers are interested in more specific group factors, hierarchical IRT models outperformed the higher-order IRT models.

This study undertakes a comprehensive comparison of item selection methods and proficiency scores estimation in several multidimensional IRT models in conjunction with a CAT. The item selection and proficiency score estimation methods are negligible across the four multidimensional IRT CAT algorithms. However, the Bayesian item selection method has smaller RMSEs and SEs than the MFI method in specific cases and the EAP scoring method outperforms the MLE method, especially for short test length in this study.

Woods and Lin (2009) proposed a unidimensional item response theory (IRT) model where the distribution of the latent variables is estimated using a semi-nonparametric (SNP, Gallant & Nychka, 1987) density. Estimation of the latent variable density can reduce bias in parameter estimates that results from misspecifying the form of the density (Woods & Thissen, 2006; Woods & Lin, 2009). However, application of the Woods and Lin (2009) model is restricted to the unidimensional setting. To address this limitation, the present research generalizes the Woods and Lin (2009) model to multidimensional IRT (MIRT). The resulting model, the SNP-MIRT model, may also be considered a generalization of the "standard" MIRT model, which specifies a normal density for the latent variables.

A secondary focus of this research concerns a new proposal for calculating student growth percentiles (SGP, Betebenner, 2009). In Betebenner (2009), quantile regression (QR, Koenker & Bassett, 1978; Koenker, 2005) is used to estimate the SGPs. However, a shortcoming of the original methodology is that measurement error in the score estimates, which always exists in practice, leads to bias in the SGP estimates (Shang, 2012). One way to address this issue is to estimate the SGPs using a modeling framework that can directly account for the measurement error. MIRT is one such framework, and the one utilized here. To maximize the generality of the approach, as well as guard against misspecification of the latent variable density, the SNP-MIRT model is used. SNP-MIRT estimates, in turn, are used with the calibrated projection linking methodology (Thissen, Varni, et al., 2011; Thissen, Liu, Magnus, & Quinn, 2014; Cai, in press-a, in press-b) to produce SGP estimates.

Preliminary simulation studies are conducted to investigate the fidelity of the SNP-MIRT model and SGP estimation implementations. The simulation study for the SNP-MIRT model focuses on recovery of the shape of the data-generating latent variable density. The simulation study for the proposed SGP method focuses on comparing the accuracy of the QR, standard MIRT, and SNP-MIRT approaches. Finally, empirical applications are provided to illustrate the new methods.