UCLA

In model selection, we seek a balance between goodness-of-fit and generalizability, for which model complexity is key. Fitting Propensity (FP) has been suggested as an ideal measure of complexity that refers to a model’s inherent flexibility to fit diverse patterns of data, all else being equal. Assessing the FP of item response theory (IRT) models requires random and uniform sampling of all item response patterns for a set of items and fitting the sampled data to one or more models repeatedly many times. The model fit information across the replications is summarized for each model and examined. In the case of multiple models, comparisons between models are also made. Computational issues due to the high-dimensional discrete space involved in the generation of random datasets have rendered it infeasible to investigate FP for more than a handful of dichotomously scored items under the conventional full information (FI) approach of the multinomial framework.This study turns to limited information (LI) methods as an alternative, capitalizing on the fact that IRT models can be realized as contingency tables using marginal probabilities. LI methods use information from only the lower-order, usually univariate and bivariate, margins of IRT models as opposed to full response patterns. Thus, they not only significantly reduce the number of response probabilities to be generated in the first place but can also make model estimation computationally simpler. The computational gain afforded by the proposed LI approach opens doors for investigating the FP of more complex IRT modeling schemes which traditionally require many more response patterns. A data-generating algorithm founded on classical literature on sampling contingency tables with fixed margins along with sequential importance sampling (SIS) of contingency tables is introduced for random and uniformly distributed sampling across all univariate and bivariate margins of items. To estimate the data consisting of solely the lower-order margins, a pairwise marginal maximum likelihood (PMML) estimator tailored to fit a wide variety of IRT models is introduced. Lastly, the feasibility of the proposed LI data generation algorithm and estimation approach to assess the FP of IRT models is tested under various combinations of data sampling and estimation methods.