Both network (graph) and vector space are two representational forms commonly used in modeling knowledge structures and processes, but their relationship has not been extensively explored. In this paper, we conceptualize and formally show that these two types of models can be related in terms of both structure and process. In particular, the traditional and `higher order' cosine similarity in the vector space is mathematically equivalent to intersecting activation spread on a network after traversing direct and indirect paths. Inspired by this equivalence, we transfer graphical techniques to vector space modeling and demonstrate that the `higher-order' information embedded in the vector space can be used to create more powerful representations and accelerate learning in neural networks. Our result may have profound implications for both cognitive representational theory and machine learning practice.

Semantic space models based on distributional information and semantic network (graphical) models are two of the mostpopular models of semantic representation. Both types of models succeed at modeling or explaining various tasks. Bothtypes of models also have limitations. Spatial models have difficulties representing indirect semantic relations, whilegraphical models have lacked a theoretical account for the construction of their semantic network. In this article, wedevelop the Distributional Graph Model. The new model resembles semantic space models in the way that it is a repre-sentation of semantic memory obtained from statistical learning on a linguistic corpus. But like other graphical models,it is able to capture indirect semantic relatedness as well. Using an artificial language specifically designed to test differ-ent types of syntagmatic and paradigmatic relationships, we show that the Distributional Graph Model demonstrates thebenefits of both graphical and spatial distributional models.

An important aspect of language comprehension is learning and generalizing complex lexical relations. For instance, having learned that the phrase preserve cucumbers predicts vinegar and that preserve berries predicts dehydrator, one should be able to infer that the novel phrase preserve peppers is more compatible with vinegar, because pepper is more similar to cucumber. We studied the ability to perform such (compositional) generalization in distributional models trained on an artificial corpus with strict semantic regularities. We found that word-encoding models failed to learn the multi-way lexical dependencies. Recurrent neural networks learned those dependencies but struggled to generalize to novel combinations. Only mini GPT-2, a minified version of the Transformer GPT-2, succeeded in both learning and generalization. Because successful generalization in our tasks requires capturing the relationship between a phrase and a word, we argue that mini GPT-2 acquired hierarchical representations that approximate phrase structure. Our results show that, compared to older models, Transformers are architecturally advantaged to perform compositional generalization.

A critical part of language comprehension is inferring omitted but plausible information from linguistic descriptions of events. For instance, the verb phrase ‘preserve vegetable’ implies the instrument vinegar whereas ‘preserve fruit’ implies dehydrator. We studied the ability of distributional semantic models to perform this kind of semantic inference after being trained on an artificial corpus with strictly controlled constraints on which verb phrases occur with which instruments. Importantly, the ability to infer omitted but plausible instruments in our task requires compositional generalization. We found that contemporary neural network models fall short generalizing learned selectional constraints, and that a graph-based distributional semantic model trained on constituency-parsed data and equipped with a spreading-activation procedure for calculating semantic relatedness, achieves perfect performance. Our findings shed light on the mechanisms that give rise to compositional generalization, and using graphs to model semantic memory.