Proceedings of the Annual Meeting of the Cognitive Science Society

Predictive neural networks, such as word2vec, have seenimpressive recent popularity as an architecture to learndistributional semantics in the fields of machine learning andcognitive science. They are particularly popular because theylearn continuously, making them more space efficient andcognitively plausible than classic models of semantic memory.However, a major weakness of this architecture is catastrophicinterference (CI): The sudden and complete loss of previouslylearned associations when encoding new ones. CI is an issuewith backpropagation; when learning sequential data, the errorsignal dramatically modifies the connection weights betweennodes—causing rapid forgetting of previously learnedinformation. CI is a huge problem for predictive semanticmodels of word meaning, because multiple word sensesinterfere with each other. Here, we evaluate a recentlyproposed solution to CI from neuroscience, elastic weightconsolidation, as well as a Hebbian learning architecture fromthe memory literature that does not produce an error signal.Both solutions are evaluated on an artificial and naturallanguage task in their ability to insulate a previously learnedsense of a word when learning a new one.