Proceedings of the Annual Meeting of the Cognitive Science Society

Question-asking, an essential yet understudied activity, holds significant implications for fields such as learning, creativity, and cognitive development. The quality, and complexity in particular, of the questions are recognized as crucial factors affecting these fields. Previous research explored question complexity through Bloom's taxonomy, but measurement remains challenging. Recent advancements have enabled automated scoring of psychological tasks but have not been applied to open-ended question complexity. Here, we address this gap by employing large language model (LLM) techniques to predict human ratings of open-ended question complexity. Our results reveal that our LLM-generated complexity scores correlated strongly with human complexity ratings in both the holdout-responses (r = .73) and holdout-item set (r = .77), whilst also exceeding baseline methods tested. The research emphasizes the significance of LLMs in psychological research and their potential in automating question complexity assessment. This study also highlights exciting possibilities for usage of LLMs in education and psychology.