Proceedings of the Annual Meeting of the Cognitive Science Society

Many theories of decision making have proposed that diverse psychological mechanisms (e.g., mechanisms related to affect, goals, or social factors) shape people's decisions under risk and uncertainty. How best to study such mechanisms during decision making? In a preregistered study (N = 699), we tested whether six experimentally induced mechanisms (incidental anger, cognitive load, lack of knowledge, goal pursuit, descriptive social norms, and time pressure) leave linguistic traces during decision making and can thus be identified with large language models (LLMs). To this end, participants completed four decision-making tasks while recording verbal decision protocols (VDPs). We then inferred the presence of psychological mechanisms in VDPs using "speech-to-psych", a newly developed analysis pipeline leveraging different LLM pipelines. Across analytic methods and decision contexts, psychological mechanisms as induced by experimental conditions could be identified at above-chance level (average area under the curve = .61). Feature importance scores revealed that this identification frequently relied on linguistic markers associated with the targeted psychological mechanism. These results indicate that at least some psychological mechanisms leave detectable linguistic traces and that LLM-based analyses could serve as a powerful measurement tool for advancing process-level understanding of decision making.