Proceedings of the Annual Meeting of the Cognitive Science Society

Distributional models (such as neural network language mod- els) have been successfully used to model a wide range of lin- guistic semantic behaviors. However, they lack a way to dis- tinctly represent different kinds of semantic relations within a single semantic space. Here, we propose that neural network language models can sensibly be interpreted as representing syntagmatic (co-occurrence) relations using their input-output mappings, and as representing paradigmatic (similarity) rela- tions using the similarity of their internal representations. We tested and found support for this hypothesis on four neural net- work architectures (SRNs, LSTMs, Word2Vec and GPT-2) us- ing a carefully constructed artificial language corpus. Using this corpus, we show that the models display interesting but understandable differences in their ability to represent these two kinds of relationships. This work demonstrates distribu- tional models can simultaneously learn multiple kinds of re- lationships, and that systematic investigation of these models can lead to a deeper understanding of how they work.