Conversation-Based Assessment (CBA) represents a relatively new method of measuring student cognitive skills(e.g., science reasoning) through adaptive dialogues with automated characters. This approach leverages the openness ofnatural language with the interactivity of spoken dialogue to engage students in verbal reasoning and constructive processes(i.e., cognition). These two dimensions differentiate CBA from other assessment items (e.g., multiple choice and essays) byallowing for more freedom in responses along with the ability to adapt and follow-up on particular threads of information.The conversational exchange affords a rich data stream that can provide additional explanatory evidence of students’ cognitionexhibited through conversational content and dialogue paths. The current work, built on the AutoTutor dialogue engine, willdiscuss the affordances and constraints of CBA along with how this approach may complement and enhance other methods ofmeasuring of cognitive skills.

This article discusses estimation of multilevel/hierarchical linear models that include cluster-level random intercepts and random slopes. Viewing the models as structural, the random intercepts and slopes represent the effects of omitted cluster-level covariates that may be correlated with included covariates. The resulting correlations between random effects (intercepts and slopes) and included covariates, which we refer to as “cluster-level endogeneity,” lead to bias when using standard random effects (RE) estimators such as (restricted) maximum likelihood. While the problem of correlations between unit-level covariates and random intercepts is well known and can be handled by fixed-effects (FE) estimators, the problem of correlations between unit-level covariates and random slopes is rarely considered. When applied to models with random slopes, the standard FE estimator does not rely on standard cluster-level exogeneity assumptions, but requires an “uncorrelated variance assumption” that the variances of unit-level covariates are uncorrelated with their random slopes. We propose a “per-cluster regression” (PC) estimator that is straightforward to implement in standard software, and we show analytically that it is unbiased for all regression coefficients under cluster-level endogeneity and violation of the uncorrelated variance assumption. The PC, RE, and an augmented FE estimator are applied to a real data set and evaluated in a simulation study that demonstrates that our PC estimator performs well in practice.