We evaluate whether BERT, a widely used neural network forsentence processing, acquires an inductive bias towards form-ing structural generalizations through pretraining on raw data.We conduct four experiments testing its preference for struc-tural vs. linear generalizations in different structure-dependentphenomena. We find that BERT makes a structural general-ization in 3 out of 4 empirical domains—subject-auxiliary in-version, reflexive binding, and verb tense detection in embed-ded clauses—but makes a linear generalization when tested onNPI licensing. We argue that these results are the strongest ev-idence so far from artificial learners supporting the propositionthat a structural bias can be acquired from raw data. If this con-clusion is correct, it is tentative evidence that some linguisticuniversals can be acquired by learners without innate biases.However, the precise implications for human language acqui-sition are unclear, as humans learn language from significantlyless data than BERT.

Language models have great potential as cognitive models for studying human language acquisition, but current models are far less data-efficient than human learners. Children acquire language from 100 million words or less, but large language models are trained on trillions of words. We discuss the prospects for improving language models' developmental plausibility through a meta-analysis of results from the 2023 BabyLM Challenge. BabyLM was a competition that invited participants to train a language model on a 100 million-word corpus including transcribed speech and child-appropriate texts. Results from over 30 submissions showed that new machine learning techniques and increased training iterations yielded models that outperformed leading large language models in grammar, language understanding, and linguistic generalization, while cognitively plausible approaches such as curriculum learning were less effective. We discuss the implications of these and other findings for computational cognitive modeling and explore ideas to ensure future competitions' contributions to cognitive science.