Proceedings of the Annual Meeting of the Cognitive Science Society

Understanding how different sources of our semantic knowledge contribute to language comprehension, especially under unpredictable conditions, remains a challenge. Using vision-language CLIP, and language-only GPT2 representations, we examined whether these information sources differentially predict single-trial EEG responses to expected (i.e., high cloze probability) and unexpected (i.e., near-zero cloze probability) words in constraining sentence contexts. We also assess whether these effects vary across semantic processing by analyzing 100-ms intervals from 200-700 ms post word onset. GPT2 provided stronger fits for both expected and unexpected words from 300 to 500 ms. CLIP explained variance beyond GPT2 from 300 ms for expected, and from 400 ms for unexpected words. For the unexpected words, CLIP performed better than GPT2 in the 500-600 ms interval. Results show that both pure distributional and vision-informed language semantic information explain unique variance in EEG responses, with systematic differences due to word predictability as well as processing time.