Proceedings of the Annual Meeting of the Cognitive Science Society

Recurrent Neural Networks (RNNs) trained on a languagemodeling task have been shown to acquire a number of non-local grammatical dependencies with some success (Linzen,Dupoux, & Goldberg, 2016). Here, we provide new evidencethat RNN language models are sensitive to hierarchical syntac-tic structure by investigating the filler–gap dependency andconstraints on it, known as syntactic islands. Previous workis inconclusive about whether RNNs learn to attenuate theirexpectations for gaps in island constructions in particular orin any sufficiently complex syntactic environment. This papergives new evidence for the former by providing control studiesthat have been lacking so far. We demonstrate that two state-of-the-art RNN models are are able to maintain the filler–gapdependency through unbounded sentential embeddings and arealso sensitive to the hierarchical relationship between the fillerand the gap. Next, we demonstrate that the models are ableto maintain possessive pronoun gender expectations throughisland constructions—this control case rules out the possibil-ity that island constructions block all information flow in thesenetworks. We also evaluate three untested islands constraints:coordination islands, left branch islands, and sentential subjectislands. Models are able to learn left branch islands and learncoordination islands gradiently, but fail to learn sentential sub-ject islands. Through these controls and new tests, we provideevidence that model behavior is due to finer-grained expecta-tions than gross syntactic complexity, but also that the modelsare conspicuously un-humanlike in some of their performancecharacteristics.