LLMs Electrified: Early and deep layers differentially correlate with the N400 and P600 in language comprehension
Abstract
Recent research has examined the extent to which large language models (LLMs) can model the temporal dynamics of online language comprehension, as indexed by the N400 and P600 components of the event-related potential (ERP) signal. To better understand whether the internal representations of LLMs are consistent with distinct stages of comprehension, we employ representational similarity analysis (RSA) on a German ERP study, that found the N400 to be sensitive to association and expectancy, and the P600 to be sensitive to expectancy alone. We find that earlier layers show a stronger correlation with association, whereas deeper layers are more strongly correlated to expectancy. Similarly, correlations to the N400 are stronger at earlier and intermediate layers, consistent with its sensitivity to association, while correlations with both the N400 and P600 continue to increase in deeper layers, reflecting the influence of expectancy on both components. These results are consistent with independent stages of processing proposed by neurocognitive theories, such as Retrieval-Integration theory, suggesting that LLMs may contribute to our understanding of the temporal dynamics of language comprehension - as indexed by ERPs - at a more mechanistic level.