Proceedings of the Annual Meeting of the Cognitive Science Society

The semantic memory literature has recently seen the emergenceof predictive neural network models that use principles ofreinforcement learning to create a “neural embedding” of wordmeaning when trained on a language corpus. These models havetaken the field by storm, partially due to the resurgence ofconnectionist architectures, but also due to their remarkablesuccess at fitting human data. However, predictive embeddingmodels also inherit the weaknesses of their ancestors. In this paper,we explore the effect of catastrophic interference (CI), long knownto be a flaw with neural network models, on a modern neuralembedding model of semantic representation (word2vec). We usehomonyms as an index of bias depending on the order in which acorpus is learned. If the corpus is learned in random order, the finalrepresentation will tend towards the dominant sense of the word(bankà money) as opposed to the subordinate sense (bankàriver). However, if the subordinate sense is presented to thenetwork after learning the dominant sense, CI produces profoundforgetting of the dominant sense and the final representationstrongly tends towards the more recent subordinate sense. Wedemonstrate the impact of CI and sequence of learning on the finalneural embeddings learned by word2vec in both an artificiallanguage and in an English corpus. Embedding models show astrong CI bias that is not shared by their algebraic cousins.