Glossa Psycholinguistics

1.1 A unified account

One way to account for the puzzling asymmetry between subject and object gaps in islands and non-islands might be to stipulate a specific constraint against subject gaps in islands. However, researchers have long noted that the That-Trace Effect and the Island Boundary-Gap Effect are suspiciously similar (Chomsky 1980), at least on the surface. They both involve an unacceptable subject gap immediately following an overt clausal function word, like ‘that’ (2) or ‘why’ (3a). Drawing on this surface similarity, many attempts have been made to attribute the That-Trace Effect and the Island Boundary-Gap Effect to the same underlying constraint.

For instance, McDaniel et al. (2015) offer an account based on the idea that processing pressures during production eventually become grammaticized over the course of language evolution (Hawkins 2014). They stipulate that clause-initial function words like that and whether demarcate separate planning units for the language production system. They then argue that connecting clauses with a long-distance dependency is more difficult immediately after the beginning of a new planning unit. Over the course of language evolution, this difficulty results in the grammaticization of a constraint against gaps immediately after a clause-initial function word.

Perhaps the most famous attempt at unification is a linguistic constraint proposed by Chomsky (1981) known as the Empty Category Principle (ECP). According to the ECP, which is understood to be a general principle that applies in all languages, gaps are prohibited from appearing immediately after clause-initial function words like that, when, because, etc. The specifics of this proposal require extensive knowledge of Government and Binding (Chomsky 1981), a prominent theoretical framework from the 1980s, but are not critical for the present purposes (see Chomsky 1981; Lasnik & Saito 1984; Kayne 1981 for further detail). Like McDaniel et al.’s account, the ECP also explains the ungrammaticality of the That-Trace Effect and the Island Boundary-Gap Effect as the result of violating a single underlying constraint.

In spite of the elegance of a unified explanation of the That-Trace Effect and the Island Boundary-Gap Effect, there is no a priori reason to believe that the two should be the same. Indeed, there are several key differences between the two. While the That-Trace Effect is conditioned on the presence of a specific complementizer, ‘that,’ the Island Boundary-Gap Effect is conditioned on any type of island boundary, which may or may not involve a complementizer. Island boundaries can be marked with a complementizer (or complementizer-like function word; the particular analysis does not matter here) like if or because (4a). But they can also be marked with a relative pronoun or wh-operator like who, why or when (4b), or even a wh-operator involving an entire referential phrase like which of these staplers (4c). Not only are these latter examples not complementizers, they also occupy a different syntactic position within the sentence,¹ as shown in Table 1.

Table 1

Illustrations of the distinct positions that complementizers and relative pronouns occupy.

Complementizer (that)	Wh- operator (why)
“…thought that the patient took the pill”	“…wondered why the patient took the pill”

Thus, the similarity between the That-Trace Effect and the Island Boundary-Gap Effect is superficial, at best. This relatively weak evidence has nonetheless been used to motivate the assumption, which is pervasive throughout the literature, that the two effects are the same. This assumption is clearly appealing: it reduces the number of stipulative constraints in the grammar. But it only does so by one. Without stronger evidence in favor of unification, it is not clear that such a small reduction is sufficient to justify such a consequential assumption.

The present aim is to experimentally assess whether the two effects are the same, and in so doing, demonstrate a novel approach to questions about equivalence. As will be seen in Experiments 1 and 2, the type of experimental paradigms and statistics standardly used in experimental syntax impose a familiar constraint: it is easy to design a study intended to detect differences between constructs, but showing that two constructs are the same is trickier. To do so, Experiment 3 leverages psychometric methods to flip the usual logic of the experimental design, so that a significant result can provide evidence of equivalence rather than difference.

1.3 The present experiments

Here, three experiments with large sample sizes aim to provide further evidence for unifying the two effects. Experiment 1 (N = 161) examines acceptability while deconfounding the subject position in wh-islands from the position immediately after clause-initial function words. Experiment 2 (N = 189) aims to validate that the types of stimuli from Experiment 1, which have not been previously used as stimuli in experimental research, show prototypical properties of islands. While the findings of Experiments 1 and 2 are consistent with a unified theory of the That-Trace Effect and the Island Boundary-Gap Effect, the experiments are unable to definitively demonstrate that the effects are the same due to inherent limitations on the joint logic of the standard experimental design and Null Hypothesis Significance Testing.

Experiment 3 (N = 104) circumvents these limitations using a novel approach in experimental syntax. Leveraging psychometric methods, it directly tests for equivalence by analyzing individual differences in the That-Trace Effect and the Island Boundary-Gap Effect to determine whether the two correlate beyond what would be expected due to extraneous shared properties.

2.2 Results

Results (summarized in Table 3 and depicted in Figure 1) showed a significant simple main effect of COMPLEMENTIZER. Because the factors were treatment coded, this effect reflects the fact that the Subject,Null condition was better than the Subject,That condition (t(76.51) = 4.55, p < .001); model comparison confirmed the effect significantly contributed to model fit (χ²(1) = 18.185, p < .001). There was also a significant simple main effect of GAP POSITION, reflecting the fact that the Object,That condition was better than the Subject,That condition (t(399.78) = 2.2, p = .03; model comparison: χ²(1) = 4.839, p = .028). Critically, the interaction was significant, reflecting the fact that in islands, subject gaps are only worse than object gaps in the presence of an overt clause-initial function word (t(135.628) = –3.15, p < .01; model comparison: χ²(1) = 9.803, p = .002).⁵

Figure 1

Experiment 1 results: Mean z-scores per condtion. Bars denote standard error.

2.3 Discussion

Experiment 1 tested the hypothesis that the Island Boundary-Gap Effect derives from a specific constraint on subject gaps in islands, in which case it would reflect a different constraint from whatever gives rise to the That-Trace Effect. This hypothesis would predict that subject gaps are always less acceptable than the corresponding object gaps in islands. However, the results contradict this prediction. Instead, subject gaps are only worse when immediately preceded by a clause-initial function word. This is consistent with the idea that the Island Boundary-Gap Effect may be the same as the That-Trace Effect.

One potential loose end is that the Experiment 1 stimuli were atypical of stimuli commonly tested in island literature in that they had an extra level of embedding. In order to confirm that the findings of Experiment 1 can be brought to bear on the question at hand, it should be verified that these stimuli do in fact behave like islands. This was the aim of Experiment 2.

3.2 Results

Results, summarized in Table 5 and depicted in Figure 2, showed a significant simple main effect of ISLANDHOOD, reflecting the fact that sentences with islands were better than sentences without islands when the gap appeared in the matrix position (i.e., outside the island; t(57.58) = 2.27, p = .03); model comparison confirmed that the effect significantly contributed to model fit (χ²(1) = 5.064, p = .024). There was also a significant simple main effect of GAP EMBEDDING, reflecting the fact that in non-islands, embedded gaps were less acceptable than matrix gaps (t(66.25) = –11.57, p < .001; model comparison: χ²(1) = 55.868, p < .001). Crucially, the characteristic island interaction was also significant, indicating that the difference in acceptability between a gap in an embedded versus a matrix clause is larger for island sentences than for non-island sentences (t(604.4) = –3.73, p < .001; model comparison: χ²(1) = 13.068, p < .001).

Figure 2

Experiment 2 results: Mean z-scores per condition. Bars denote standard error.

3.3 Discussion

Experiment 2 demonstrates that the sentences tested in Experiment 1 do in fact behave like islands, showing the defining statistical interaction between ISLANDHOOD and GAP EMBEDDING. However, two aspects of the current findings are atypical with respect to the commonly found pattern in the literature. First, when gaps were in matrix position, the Island condition was rated higher than the Non-island condition. Second, while significant, the difference between the Non-island,Embedded condition and the Island,Embedded condition was not as big as is commonly observed (see, for instance, the many results in Sprouse et al. 2016).

The first atypicality – that islands were more acceptable than non-islands in Matrix conditions – may reflect the increased syntactic diversity of the clause types in the sentence. It has previously been demonstrated that increasing the heterogeneity of syntactic properties of complex sentences can increase their acceptability. For instance, (9a) contains three full Determiner Phrases (the mouse, the cat, and the dog), and is highly unacceptable. However, (9b), which has the same global structure but contains one quantified nominal (someone), one local pronoun (I), and one full Determiner Phrase (the mouse), is much more acceptable. This is thought to be the result of the fact that mutual distinctiveness among aspects of a stimulus can facilitate working memory processes (Bever 1974).

In the Experiment 2 stimuli, the types of nominal elements did not vary systematically across conditions as in (9), but the types of clause structures did. In Island conditions, stimuli consisted of two declarative clauses and one embedded interrogative clause, whereas Non-island stimuli consisted of three declarative clauses. It stands to reason that in these highly demanding sentences, structural diversity may have facilitated working memory in the same way that nominal distinctiveness aids working memory in (9), resulting in the higher acceptability of the Island,Matrix condition relative to the Non-island,Matrix condition.

An anonymous reviewer pointed out the second atypical feature of the Experiment 2 results: that the difference between the two embedded conditions was smaller than is often reported in the literature. This is potentially concerning because, if the size of this difference is indeed smaller than for the types of island structures that have been previously studied, it may mean that these stimuli are somehow different from “ordinary” island violations. If that were true, then the conclusion of Experiment 1 – that in islands, subject gaps are not inherently worse than object gaps – may not extend to islands in general.

However, there are at least three reasons not to be overly concerned about this discrepancy. First, if the structural diversity explanation given above is correct, then the Island,Embedded condition is worse than the Non-island,Embedded condition in spite of an acceptability boost the former receives from containing mutually distinctive clause types. Without this boost, the difference would be bigger.

Second, the absolute size of a difference in z-scores should not generally be compared across experiments. This is because a unit in z-scores represents one standard deviation in the distribution of ratings of all stimuli, and therefore depends on the spread of acceptabilities in the particular stimulus set. Thus, in a hypothetical experiment with the same critical items but different fillers, the size of this difference could be larger or smaller.

Third, and most importantly, the ratings in the Island,Embedded condition exhibit signs of a floor effect. That is, differences that we might have expected to be bigger are difficult or impossible to detect because there simply wasn’t enough room at the bottom of the Likert scale to distinguish between, e.g., very unacceptable and very, very unacceptable. For instance, the floor fillers were included to establish a floor in the study. Thus, if the floor fillers were rated lower than the Island,Embedded fillers, then it suggests there was in fact room to distinguish even lower degrees of acceptability. But this was not the case. The mean z-score for floor stimuli was –0.521(S.E.: 0.055), and the mean z-score of the Island,Embedded condition was –0.482(S.E.: 0.049). An uncorrected (i.e., liberal) t-test shows that the difference is not significant (t(1056.1) = –1.331, p = .184). Thus, if the floor items did in fact establish a floor in this experiment, then there is no evidence that the Island,Embedded condition was not also effectively at floor.

If the Island,Embedded condition was in fact confounded by a floor effect, this does not change the interpretation of the Experiment 2 results. In fact, any floor effect in this condition would only have made the interaction smaller, meaning that the critical prediction was borne out in spite of another potential effect that worked against it, making the interaction even more credible.

Thus, despite these atypical features, the results of Experiment 2 verify that these stimuli behave like normal islands. This finding suggests that the conclusion of Experiment 1 – that subject gaps are not inherently worse than object gaps in islands – likely generalizes to the types of island structures more commonly tested in the experimental literature.

While the conclusion of Experiment 1 is consistent with the Island Boundary-Gap Effect and the That-Trace Effect resulting from the same underlying violation, it does not mean that this must be the case. Indeed, with a standard experimental approach such as those in Experiments 1 and 2, it is not generally possible to show that two constructs are the same in the Null Hypothesis Significance Testing framework.

As outlined in the introduction, one tactic for circumventing this limitation is to look for a relationship between individual differences in the That-Trace Effect and the Island Boundary-Gap Effect. In such an analysis, a significant result can constitute evidence for equivalence.

4.2 Analysis

In addition to not randomizing trial order, two other psychometric practices were adopted. First, no trials were removed from the data prior to analysis. This is because excluding trials means that individuals’ mean ratings reflect the means of different items, which could create artifactual individual differences.

Second, Experiment 3 analyzed raw ratings rather than z-scores. This is because z-scores remove individual differences by design. To calculate the z-score, the mean of all of a participant’s ratings is subtracted from each of that participant’s ratings to set each participant’s mean rating to 0. Each of the participants’ mean-adjusted ratings are then divided by the standard deviation of all of that participant’s ratings, setting the standard deviation of each participant’s ratings to 1. Typically, this is done to eliminate differences in the use of the Likert scale. While different uses of the scale are certainly a concern in Experiment 3, z-scoring can have the unintended consequence of reducing the individual differences we aim to study.

To see this, consider a participant who likes TTE stimuli more than the average person. For the sake of argument, let us also momentarily assume that there are no individual differences in the acceptability of the other conditions, nor in the use of the Likert scale. The participant who likes TTE stimuli more will have a higher overall mean rating (i.e., across all stimuli) than a participant who likes TTE stimuli less. In calculating these participants’ z-scores, a higher mean will therefore be subtracted from the ratings of the participant who likes TTE stimuli more, and a lower mean will be subtracted from the ratings of the participant who likes the TTE less, thus reducing the difference between these two participants’ ratings of TTE stimuli and obscuring the individual differences. Now, individual differences in other conditions and in the use of the Likert scale probably do exist, so the picture is somewhat noisier. But if these individual differences are not correlated with the individual differences in the TTE and IBGE (and there is no reason to expect that they are), then, across a large enough sample size, individual differences in other conditions and in the use of the Likert scale should cancel out, and the same logic holds.⁸

To understand whether the That-Trace Effect and the Island Boundary-Gap Effect derive from the same constraint, it is not enough to show a correlation between participants’ mean ratings of the two critical conditions, because a correlation could be driven by any number of extraneous things. Thus, rather than simply correlating ratings of these conditions with one another, each critical condition was modeled with a separate multiple linear regression in which the control conditions served as covariates. In Model 1, participants’ mean ratings of TTE items were modeled as a function of participants’ mean ratings of IBGE items, as well as Control_TTE items and Control_IBGE items. In Model 2, participants’ mean ratings of IBGE items were modeled as a function of participants’ mean ratings of TTE items, Control_IBGE items, and Control_TTE items.⁹

In the particular type of multiple regression employed here (R’s default lm() function), multicollinearity (when two or more of the regressors are correlated) is dealt with by ignoring the shared variance (McElreath 2020; Ch. 6). Thus, by including the control conditions as covariates, it is ensured that any significant relationship between the two critical terms reflects individual differences in properties shared only by those two conditions.

This has a similar effect to simply subtracting each participant’s mean rating of Control_TTE stimuli from their mean rating of TTE stimuli and correlating that difference with the difference between IBGE and Control_IBGE stimuli. While subtraction may be somewhat more intuitive, multiple regression offers a few additional benefits, including producing statistics indicating whether these “subtractions” remove meaningful variance.

To make this point more concrete, consider two individuals who map the same degree of subjective unacceptability to different points along the Likert scale. One may assign this degree of unacceptability a 4, and the other may assign it a 6. This baseline difference in ratings will likely mean that for each type of structure, the first person’s ratings will be lower than the second person’s. This alone is enough to drive a correlation between individual differences in two structures. One might therefore expect that all of the independent variables should significantly predict the dependent variables in Models 1 and 2.

However, because this variance is shared by all of the independent variables, the model cannot tell which of them to attribute it to, so it attributes it to none. Thus, individual differences that are common to all of a participant’s acceptability ratings, such as those that might arise from differences in the use of the Likert scale, cannot drive a significant effect in the present analyses. If the IBGE term is significant in Model 1, it means that individual differences in the Island Boundary-Gap Effect correlate with individual differences in the That-Trace Effect above and beyond what can be explained by individual differences present in the ratings of Control_TTE and/or Control_IBGE.

Another benefit of this analysis approach is that the models have built-in tests of two key concepts in psychometric research: reliability, or the accuracy of a measure, and validity, or the degree to which a measure reflects the construct it is meant to measure (Cronbach & Meehl 1955). Without high validity and reliability, individual differences are impossible to distinguish from naturally occurring noise.

The data can be said to have high reliability if a significant relationship is found between TTE and IBGE because this would indicate that acceptability ratings were an accurate enough measure to reveal individual differences in a shared underlying constraint, despite other differences between these two conditions.

If acceptability ratings have high enough validity, then the distinct properties of the two control conditions should lead to different patterns of significance in the two models. In Model 1, where the dependent variable is TTE, a significant effect of Control_TTE is expected, but no effect of Control_IBGE because, although it also shares some structural properties with TTE, all of these properties are also shared by Control_TTE. Similarly, in Model 2, Control_IBGE is expected to be significant, but Control_TTE is not because the former shares all of the same properties with IBGE as the latter.

One final difference between this analysis and that of Experiments 1 and 2 is that Experiment 3 employs a fixed-effects-only model. This is because there are no random effects to feed the model: random effects for participants are intended to remove individual differences, so these are not included, and random effects for items require that the model have access to item-specific data, but the present models analyzed aggregate statistics (i.e., participants’ condition means).

4.3 Results

Mean ratings for each condition are shown in Figure 3. Because particular lexicalizations were not counterbalanced across items (i.e., the sentence about a pirate kidnapping a princess only ever appeared in the Non-island,Overt condition), any differences between conditions may reflect differences in the acceptability of the particular sentences in a condition. Comparing condition means therefore does not necessarily reveal anything about the true differences between the structures’ acceptability, so no such statistical analysis was performed.

Figure 3

Experiment 3 results: Mean ratings per condition. Bars denote standard error.

Nonetheless, the pattern is what would be expected had the study included counterbalancing and randomization, perhaps indicating that any lexicalization-specific acceptability differences between conditions were relatively small. The TTE items were rated worse than the Control_TTE stimuli, as expected due to the That-Trace Effect. In islands, this pattern reversed, and IBGE stimuli were rated better than Control_IBGE stimuli. This is not surprising given that the Control_IBGE condition contained one more clausal embedding than the IBGE condition, making it longer and more complex.

Results of both models are summarized in Table 7. Both revealed a strong positive relationship across participants between the strength of the That-Trace Effect and the Island Boundary-Gap Effect (t(89) = 6.44, p < .001; see Figure 4). Note that this is the same effect in both models (and as such has the same t-value). The difference between Models 1 and 2 is in the relationships between the dependent variable and the control conditions. In Model 1, TTE was significantly predicted by Control_TTE (t(89) = 3.82, p < .001), but not by Control_IBGE. In Model 2, the pattern was flipped, and IBGE was significantly predicted by Control_IBGE (t(89) = 7.562, p < .001), but not by Control_TTE.¹⁰

Figure 4

Experiment 3 results: Dots represent individuals; bilinguals have blue interiors. In order to show just the relationship between the That-Trace Effect and the Island Boundary-Gap Effect, the TTE and IBGE values here were both residualized on both control conditions. The dark grey line represents the best-fit line and density plots appear in light grey. A Pearson’s correlation for the data in this plot reveals a strong, significant relationship: r = .564, t(91) = 6.511, p < .001).

4.4 Discussion

The main analysis revealed a significant relationship between TTE and IBGE. That is, a person who finds the That-Trace Effect to be especially bad is more likely to also find the Island Boundary-Gap Effect especially bad. This relationship was present even though the models removed variance associated with control conditions, indicating that it could not have been driven by some extraneous individual differences such as different uses of the Likert scale.

It also could not have been driven by individual differences in how people rate ungrammatical structures in general, because Control_IBGE, which is ungrammatical due to an island violation, did not significantly predict TTE in Model 1. The results therefore support the idea that the That-Trace Effect and the Island Boundary-Gap Effect reflect the same underlying violation.

5.1 Individual differences, reliability, and validity

In addition to demonstrating a novel approach to assessing equivalence between psychological constructs, Experiment 3 made another important contribution: the finding of individual differences in a syntactic representation. Contrary to the idealized notion that native speakers of the same language all have the exact same grammar, speakers in fact have slightly different grammars (Dąbrowska 2012). However, there is not much prior evidence for individual differences in syntax.

The clearest such data come from Han et al. (2007), who looked for evidence of different grammars among Korean speakers. Specifically, there are multiple syntactic positions that a verb might occupy due to the presence of a phenomenon known as verb raising. While in languages with verb-medial word order, like English, verb-raising results in a different surface word order, in verb-final languages like Korean, verb raising would not alter the surface word order. It is therefore possible that some Korean speakers might acquire a grammar with verb raising, while others acquire one without verb raising. Han et al. (2007) cleverly asked native Korean-speaking children and adults to interpret sentences with negation to determine whether the verb fell within the scope of negation, consistent with a non-verb raising grammar, or outside of the scope of negation, consistent with verb raising. Interestingly, results in both children and adults were bimodal: some speakers appeared to have acquired one grammar, while others had acquired a different one. (See also Dąbrowska 2008 for parallel work demonstrating bimodal individual differences in Polish morphology.)

One reason for the paucity of evidence for individual differences like these is the difficulty of controlling for confounds in this kind of research. For instance, individual differences in reading times in language comprehension tasks have been claimed to reflect individual differences in processing (Kidd et al. 2018). But it is trivially true that individuals’ mean reading times will differ from one another given the presence of noise in the data. To credibly demonstrate individual differences, one must also demonstrate reliability. This was achieved in Experiment 3 by showing that individual differences correlated between the TTE and IBGE conditions.

But even when reliable individual differences are identified, it is often not possible to attribute them to any particular source. Reading time differences could reflect differences in language processing mechanisms, as Kidd et al. (2018) argue; but they could also reflect differences in visual processing, attention, or even different computer hardware (Enochson & Culbertson 2015). In addition to reliability, one must therefore also demonstrate that the measure is valid. Validity was demonstrated in Experiment 3 by showing that these conditions correlated with related control conditions, but not with unrelated ones.

The present work demonstrates a clear case of individual differences in native speakers’ grammars. While individual differences in acceptability ratings are generally disregarded as noise, these findings show that they contain meaningful between-subjects variance. Individuals in Experiment 3 differed not only in how strongly they dislike the That-Trace Effect/Island Boundary-Gap Effect, consistent with decades of speculation along these lines (Chomsky & Lasnik 1977; Pesetsky 1982; Sobin 1987), but also in how much they (dis)like island violations and ordinary relative clauses, as evidenced by the significant relationships between the critical items and control items.

5.2 Why is there a That-Trace Effect at all?

The finding that the That-Trace Effect and the Island Boundary-Gap Effect derive from the same underlying constraint makes an intriguing typological prediction: that languages should either show sensitivity to both effects or neither (see Goodall (2022) for similar reasoning about island phenomena). Indeed, this prediction appears to be borne out in Spanish (Torrego 1984; Suñer 1991) and Norwegian (Kush & Dahl 2020), both of which lack the That-Trace Effect and the Island Boundary-Gap Effect. A promising avenue for future work will be to establish whether this pattern holds for a wider set of languages. Such a finding would provide important validation for the unified analysis.

But a fundamental question still stands: Why should a constraint that prohibits gaps after clause-initial function words exist at all? And why in so many languages? One might posit that something about the semantic or pragmatic structure of these sentences is hard to process or ill-formed, as has been suggested for islands (Goldberg 1995). But this cannot be the case, as a sentence that is identical to one with a That-Trace Effect violation except in that it has no that is perfectly grammatical and has the same meaning.

This is all the more puzzling when one considers the fact that the existence of this constraint imposes a limit on the expressive capacity of a language. That is, in cases where clause-initial function words are obligatory (as in islands), English provides no good way to form a subject relative clause, as exemplified by (10):

As it is quite common to talk about doctors and why they are running late, it stands to reason that being able to express the intended meaning of this sentence, and similar ones, could be useful for speakers of languages that disallow it like English and Wolof. Why, then, should this particular structure be prohibited?

This study did not directly investigate the reason for the constraint, but the finding of individual differences in Experiment 3 may provide a clue. That is, whatever the underlying reason for the That-Trace Effect, it must be something that can vary across individuals. One possibility consistent with this is that the That-Trace Effect is an artifact of domain-general processing difficulty, as argued by McDaniel et al. (2015).

Another possibility is that the That-Trace Effect simply reflects the low frequency with which English speakers hear the violation (Phillips 2013). A child learning English might surmise that, if it were allowed in English, she would hear it more frequently. Because she doesn’t, she implicitly learns that it is unacceptable, rates it as such when participating in a psycholinguistic experiment, and perpetuates this cycle by producing it infrequently. Children who happen to hear more of these structures early in life may go on to report that they are relatively more acceptable later in life, and children who hear fewer should show greater sensitivity. This would be consistent with the fact that the constraint appears to vary across languages (Maling & Zaenen 1978; Nicolis & Biberauer 2008; Chacón 2015; Kush & Dahl 2020), and it converges with recent ideas that a similar process may be at least partially responsible for the idiosyncratic variation in which structures constitute islands across languages (Goodall 2019).

5.3 Causal inference

This paper has relied heavily on a correlation to infer a causal relationship. This inference is not generally licensed given that, if A correlates with B, it could be either because A causes B or because some latent variable, C, causes both A and B. Here it is assumed that a latent variable – maybe a constraint like (11) – causes both the That-Trace Effect and the Island Boundary-Gap Effect.

(See Bresnan 1972; 1977; Chomsky & Lasnik 1977 for similar proposals.) It is this constraint that is understood to vary across individuals, leading to the correlation between individual differences in the That-Trace Effect and the Island Boundary-Gap Effect.

In this case, the conclusion that a latent variable causes the correlation derives not from the data, but from reasoning about the problem. That is, it is not clear how the That-Trace Effect, a property of one type of sentence, could cause a distinct effect in different kinds of sentences. Thus, only the latent variable explanation makes sense.

In some cases, however, reasoning may not provide a clear answer. For instance, visual and verbal working memory appear to use partially overlapping resources (Morey & Cowan 2005). It is therefore possible that some component of visual working memory relies causally on the verbal working memory architecture (or vice versa). Here, however, the standard experimental approach can suffice to demonstrate both equivalence and causation. This is because working memory can be manipulated. For instance, if a researcher reduces verbal working memory capacity – for example by varying memory load, noise, attentional demands, or with transcranial magnetic brain stimulation – then it is relatively straightforward to infer the underlying relational architecture.

For linguistic constructs like the ones investigated here, it is not obvious how one might manipulate a syntactic “rule” like that in (11) to test for causal consequences. While the psychometric approach exemplified in Experiment 3 cannot definitively establish causality, a correlation may be enough in cases where causality can be inferred from the logic of the problem. For language research, then, the Experiment 3 approach may constitute a particularly useful tool.

A. Supplementary Analysis: Trial exclusion criteria in Experiments 1 and 2

An anonymous reviewer wondered whether excluding trials in which participants responded incorrectly to a comprehension question for a sentence with an island violation might be overzealous given that the stimuli were ungrammatical. The analyses in Experiments 1 and 2 were therefore repeated, but with different exclusion criteria.

For Experiment 1, participants were removed if they responded to fewer than 60% of the unambiguous ceiling fillers correctly (32 exclusions) or if they did not rate ceiling fillers more than two standard deviations higher than floor fillers, as before (8 more exclusions). Also as above, trials were removed if they were responded to in less time than it would take to read at a rate of 1 word per 150 ms (71 critical trials) and ceiling trials were removed if the comprehension question was responded to incorrectly (103 trials). No critical trials were excluded as all of these involved island violations. A total of 2833 remaining trials from 121 participants were analyzed (as compared to 1967 trials from 106 participants in the main analysis). As before, z-scores were computed by participant. All other analysis details were the same as in the main analysis.

Condition means are shown in Figure 5. Model results, reported in Table 8, were qualitatively identical to those reported in the main analysis in the paper (Table 3), suggesting that the Experiment 1 results were relatively robust to particular choices of exclusion criteria.

Figure 5

Experiment 1 condition means without excluding island violation trials for accuracy.

For Experiment 2, participants were excluded if they responded correctly to fewer than 60% of comprehension questions to “unambiguous” stimuli (defined as ceiling fillers and the three critical conditions that did not involve an island violation: Matrix,Non-island, Matrix,Island, and Embedded,Non-island; 39 exclusions). Participants were also excluded if they did not rate ceiling fillers more than two standard deviations higher than floor fillers, as before (8 exclusions). Trials were excluded if they were read in less time than it would take to read at a rate of 1 word per 150 ms (20 critical trials). A total of 647 critical trials that did not involve island violations (19.1%) were excluded because the comprehension question was answered incorrectly. All other analysis details were the same as in the main analysis.

Condition means are shown in Figure 6. Again, model results, given in Table 9, were similar to those in the main analysis (Table 5), suggesting that the results of this experiment were also relatively robust.

Figure 6

Experiment 2 condition means without excluding island violation trials for accuracy.

B. Supplementary Analysis: Analyzing z-scores in Experiment 3

The Experiment 3 models reported in Table 7 analyzed raw ratings. As explained in Section 4.2, z-scoring the ratings from Experiment 3 could have had the unintended consequence of reducing individual differences in TTE and IBGE stimuli. Specifically, participants who like TTE and IBGE stimuli more than the average person will likely have higher overall mean ratings on average. (TTE and IBGE stimuli made up roughly 30% of all stimuli in Experiment 3.) Subtracting these higher means from their ratings will reduce their ratings of TTE stimuli more than it would for a person who finds the TTE worse than the average person. Indeed, people who like TTE stimuli less will likely have lower overall mean ratings, and subtracting these means from their ratings will increase their ratings relative to participants who like TTE stimuli more. The end result is a reduction in the difference between these two types of participants, obscuring the individual differences of interest.

One potential way to avoid this would be to calculate z-scores with the means and standard deviations of just filler trials. In this way, participants’ ratings of the critical and control conditions do not contribute to scaling their ratings. Such an approach comes with its own risks. For instance, participants’ mean and standard deviations are calculated from only 27 ratings (just filler trials), meaning that these estimates are likely to be noisier than in an ordinary z-score approach (which would estimate these statistics using all 67 trials’ ratings). The approach furthermore assumes that ratings of filler conditions are independent of ratings of critical and control conditions, which is probably not true. Sprouse (2009) demonstrated that participants in a binary forced choice task aim to give roughly the same number of each of the two types of response. It is easy to imagine that something similar happens with Likert scale rating tasks. For example, participants may aim to make their mean rating across stimuli a 5. If a participant strongly dislikes TTE stimuli, they may give fillers higher ratings to compensate for the lower ratings they assign to TTE items.

To determine whether anything hinged on the choice to analyze raw ratings in Experiment 3, the models reported in Table 7 were re-run using z-scores calculated using participants’ means and standard deviations calculated using just filler trials. Results, reported in Table 10, largely pattern with those reported in the main analysis.

Critically, as in the main analyses, IBGE significantly predicts TTE in Model 1 and TTE significantly predicts IBGE in Model 2. Also as in the main analyses, Control_IBGE z-scores significantly predicted IBGE z-scores in Model 2. The only difference in the pattern of significant/non-significant results between these analyses and the main ones reported in Table 7 is that Control_TTE did not significantly predict TTE in Model 1. Note that this effect is not necessary for concluding that individual differences in the That-Trace Effect correlate with Individual Differences in the Island Boundary-Gap Effect. Thus, the results of this analysis suggest that the findings of Experiment 3 are relatively robust to Experimenter degrees of freedom in data preprocessing.

C. Supplementary Analysis: Accounting for measurement error in Experiment 3

The Experiment 3 models analyzed participants’ means in each of the critical and control conditions, thus ignoring measurement error associated with the means. There exist models, commonly used in biological and economic research, which aim to take into account the uncertainty in the predictors, giving better estimates of the slopes (βs). However, it is important to note that measurement error in the independent variable(s) affects model results in a different way from measurement error in the dependent variable, the latter of which is a much more familiar problem in psycholinguistics. With increased measurement error in the dependent variable, the result is increased uncertainty in the estimated slope, but not bias in any particular direction. However, when there is measurement error in the independent variable(s), as in the present case, it causes dilution (or attenuation bias): the systematic underestimation of the slopes and corresponding increase in p-values (Gorrod et al. 2013; Bound et al. 2001; Riggs et al. 1978; Rosner et al. 1992). Thus, if an accurate estimate of the effect size is the goal, then it is very important to take measurement error in the independent variables into account (although this is almost never done in psycholinguistics). However, if the goal is simply to establish a relationship, as in the present case, ignoring measurement error in the independent variables is safe, and in fact a more conservative approach as the result will be an underestimation of the slope(s).

A clever suggestion by a reviewer was to analyze the Experiment 3 data by modeling individual differences as random effects in a linear mixed-effects model. The advantage of this approach is that it accounts for measurement error while simultaneously modeling item-based differences, thus removing another extraneous source of variance. A linear mixed effects model of raw ratings was fit with a four-level fixed effect term for CONDITION (levels: TTE, IBGE, Control_TTE, and Control_IBGE) with random effects for participants and items, and a random slope for condition nested within participants. The frequentist model would not converge, so the same model was fit in the Bayesian framework using the brms package (Bürkner 2017). The random effects matrix for participants was used as estimates of individual differences per condition.

These estimates were then plugged into the same simple linear models used in the manuscript (Table 7). The results, reported in Table 11, revealed a slightly different pattern of results from that reported in the paper. Critically, however, the relationship between the TTE and IBGE conditions remained positive and significant in both models.