Proceedings of the Annual Meeting of the Cognitive Science Society

When humans listen to natural speech, their brain responses become time-locked to certain features of the auditory stimulus. The study of this phenomenon is typically limited to acoustic and linguistic representations of speech. This work examines the contributions of pre-lexical, lexical, and supra-lexical representations of speech as regressors in univariate and multivariate TRF models to predict EEG responses during passive listening of narrative stimuli. Pre-lexical features include acoustic properties and word segmentation. Lexical features incorporate emotive dimensions, sentiment analysis, and sensorimotor dimensions of words derived from both human ratings and transformer-based models. Finally, supra-lexical features include syntactic and semantic characteristics of words. The main finding of our work is that excluding acoustic representations, word surprisal remains the best tracked feature. Overall, our results suggest that LLMs capture multimodal aspects of word meaning and support a view of speech comprehension as an intrinsically predictive process.