Proceedings of the Annual Meeting of the Cognitive Science Society

Surprisal values from large language models (LLMs) have been used to model the amplitude of the N400. This ERP component is sensitive not only to contextual word expectancy but also to semantic association, such that unexpected but associated words do not always induce an N400 increase. While LLMs are also sensitive to association, it remains unclear how they behave in these cases. Moreover, another ERP component, the P600, has shown graded sensitivity to plausibility-driven expectancy, while remaining insensitive to association; however, its relationship to LLM surprisal is not well researched yet. In an rERP analysis, we evaluate surprisal values of two unidirectional transformers on their ability to model N400 and P600 effects observed in three German ERP studies isolating the effects of association, plausibility, and expectancy. We find that surprisal predicts an N400 increase for associated but implausible words, even when no such increase was observed in humans. Furthermore, LLM surprisal accounts for P600 effects elicited by violations of selectional restrictions, but captures neither P600 effects from more subtle script knowledge violations nor graded P600 modulations. The results of our investigation call into question the extent to which LLM surprisal offers an accurate characterisation of the functional generators of either the N400 or P600.