Proceedings of the Annual Meeting of the Cognitive Science Society

Large, curated databases have catalyzed key breakthroughs in science—notably the Protein Data Bank's role in shaping AlphaFold. A similar trend is emerging in cognitive science, with the Psych-101 dataset being instrumental in the development of the first foundation model of cognition. However, Psych-101 lacks representative coverage, misses critical metadata, and remains a static dataset with no scope for scaling. We propose that LLM-based agentic workflows can address these concerns. In this work, we first developed a framework to formalize claims within research articles and programmatically validate them against their associated datasets. We then built PSYCHE (Pipeline for SYstematic Curation of Human Experiments), an automated agentic system that iterates based on validator feedback. When applied to studies within Psych-101, PSYCHE produces a dataset with superior metadata coverage and higher claim accuracy compared to the original, bringing us a step closer to the creation of a scalable, 'living' database of psychology tasks.