Journal of Writing Assessment

Advances in automated writing evaluation technology have shifted the aims of the tools from reliably and holistically scoring and ranking essays to providing formative and analytical feedback to users for improving writing. This study uses a quasi-experimental design to test the ability of one automated writing evaluation program to improve college student writing. Using a comparative judgment model of assessment, four college writing instructors evaluated pairs of essays with one per pair treated by the program and selected the better of each pair. The essays treated by the automated evaluation program significantly underperformed the null hypothesis of 50%. Results suggest the automated evaluation program fails to improve student writing in the eyes of instructors. Theories and implications for why are discussed.