Compositionality makes linguistic creativity possible. By combining words, we can express uncountably many thoughts;by learning new words, we can extend the system and express a vast number of new thoughts. Recently, a numberof studies have questioned the ability of neural networks to generalize compositionally (Dasgupta, Guo, Gershman &Goodman, 2018). We extend this line of work by systematically investigating the way in which these systems generalizenovel words.In the setting of a simple system for natural language inference, natural logic (McCartney & Manning, 2007), we systemat-ically explore the generalization capabilities of various neural network architectures. We identify several key properties ofa compositional system, and develop metrics to test them. We show that these architectures do not generalize in human-likeways, lacking inductive leaps characteristic of human learning.

With a series of theoretically-informed tests, Dankers, Bruni, and Hupkes (2021) investigated how compositional the behavior of neural networks that are trained on fully natural data is. Focusing on neural machine translation (NMT), one of their key findings is that models appear to be modulating poorly between local and global behavior, where local changes in the input often affect the output in an unwanted manner. While their study is based exclusively on the behavior of the models, we take one step further and investigate how this non-locality manifests itself within the model. We develop metrics to quantify internal locality on the encoder side of the model, focusing on the attention mechanism. We find strikingly different patterns in models trained on different amounts of data that go beyond what could be observed behaviourally and present a range of experiments showing how local and global behavior is modulated within different setups.