Proceedings of the Annual Meeting of the Cognitive Science Society

Event knowledge representations in Large Language Models (LLMs) are explored through the lens of temporal connectives. Temporal connectives like before and after play an important role in sentence comprehension by linking events in the sentence to real-world event order. Their position in a sentence can create different outcomes for event order interpretation and sentence comprehension, and they have therefore been studied extensively in experimental research. Importantly, previous studies have found a human preference for chronological ordering in sentence comprehension. The current study investigates whether LLMs can recognize event order and reflect the human-like processing patterns of temporal connectives across three languages: English, Norwegian and Greek. The aim of the study is to shed light on how sentence structure and event sequence influence LLM predictions, with implications for both cognitive modeling and the knowledge representations learned by LLMs. Results vary by language and show that some models do have event representations and reflect human-like patterns of temporal ordering. The results highlight that language models for some languages (English) are better optimized for cognitive modeling than others (Greek) and underscore the need for more cognitively motivated evaluation benchmarks to assess the models being used in cognitive science research.