Glossa Psycholinguistics

1. Introduction

The ability to use grammatical rules to combine lexical items (i.e., words or signs) into phrases and sentences forms the core of the human capacity for language (Chomsky, 2017; Friederici et al., 2017; Lenneberg, 1969). Within a sentence, individual phrases are hierarchically grouped into constituents, essentially, groups of lexical items that behave as units (Everaert et al., 2015). For example, in the English sentence in (1), the lexical items [the new and interesting book] form a complex constituent, which can be focused (i.e., made more prominent by moving it to the front) in a so-called cleft construction, as in (2):

This does not apply to the non-unit [new and interesting book in], which is why the construction in (3) is considered ungrammatical (conventionally indicated by an asterisk):

The theoretical necessity of such groupings of lexical items into constituents for adequate descriptions of linguistic structures and their relations to each other was recognised early, and by now is firmly established (Everaert et al., 2015; Graffi, 2001; Harris, 1946; Nida, 1948; Pike, 1943; Wells, 1947).

Research over the past decades has established that the grammatical structure of sign languages also relies on the hierarchical grouping of lexical items into constituents (Cecchetto, 2017; Kocab et al., 2023; Mathur & Rathmann, 2014; Sandler & Lillo-Martin, 2001, 2008; Tang & Lau, 2012). For example, in the German Sign Language (DGS) sentence in (4), only the complete constituent [BOOK NEW INTERESTING] can be topicalized (accompanied by a non-manual marker in the form of raised eyebrows, glossed as ‘top’) by moving it to the front, as in (5).

This is not true for [BOOK NEW] as a sub-unit of the constituent which is embedded in the complex adjective phrase, so that extraction results in (6), which is considered ungrammatical.

The basic principles upon which language-specific syntactic structures are created are, therefore, remarkably similar across modalities (i.e., speech and sign), which is why some researchers treat them as abstract and largely universal (Chomsky, 1995; Pfau et al., 2018; Sandler & Lillo-Martin, 2001, 2008).

In parallel to the theoretical relevance of linguistic competence independent of the modality of language use (Chomsky, 1965; van der Burght et al., 2023), early psycholinguistic research began to investigate how abstract linguistic representations might affect ongoing cognitive processes in comprehension. A paradigm that was applied in those research lines was the detection of extraneous sensory signals such as an auditory click embedded in sentences (henceforth, the click paradigm; see Cutler & Norris, 1979, for a detailed overview). In this paradigm, initially introduced by Ladefoged and Broadbent (1960), participants listen to speech signals in which a short click occurs, and are asked to report when it occurred, usually by marking its position (e.g., with a ‘/’) in a written version of sentence they heard, or by typing both what they heard and the click position. We refer to these modality responses as offline responses, as the responses were made after the speech signal was over or generally without time pressure. The click paradigm, in its offline version, was used to investigate the extent to which the linguistic units assumed from a competence perspective could find a correlate in the performance domain. For example, Fodor and Bever (1965) employed this paradigm to investigate “the psychological reality of linguistic segments”, as evidenced in the title of their work, to test whether syntactic constituents might function as perceptual units in speech comprehension. Based on the notion that perceptual units maintain unity despite interfering stimuli, they interpreted the fact that participants mislocated clicks occurring within constituents, “migrating” them towards the closest major constituent boundary, as evidence for constituents being a psychological or perceptual reality in speech perception. Similar results were observed in follow-up work by Garrett and colleagues (1966) and by Bever, Lacker, and Kirk (1969), who further showed that this effect was specific to boundaries between clauses and absent for boundaries between phrases.

Building on the results of Fodor and Bever (1965), Holmes and Forster (1972) further observed that, in the offline click paradigm, not only were migration effects present, but also a modulation of the accuracy according to the click position in the syntactic structure. In particular, they showed that, even independently from migration effects (or with mild migration effects), accuracy was higher for clicks located between clauses, slightly decreasing for clicks between phrases, and becoming even smaller for clicks presented within phrases. This finding converged with the result of a previous work employing an online response modality, in which participants were asked to respond as fast as possible via button press as soon as they heard the click (Holmes & Forster, 1970). In this study, reaction times (RTs) for click detection were the fastest for clicks located between clauses, intermediate for clicks located between phrases, and the slowest for clicks within a phrase. This pattern of results was most strongly observed in the first half of a sentence, while differences between conditions became less pronounced or disappeared in the second half of the sentence. Building on the results of these two studies, Holmes and Forster (1970, 1972) suggest that the click paradigm reflects the processing load during sentence comprehension. Accordingly, both the accuracy (Holmes & Forster, 1972) and RTs (Holmes & Forster, 1970) data would indicate that the processing load is higher within constituents – hence, the lowest accuracy and highest RTs observed when the click occurs in this position – intermediate within phrases, and smaller between clauses. Furthermore, this account of the click paradigm would also be able to account for the fact that RTs were generally faster in the second half of the sentence (Holmes & Forster, 1970), as, in this case, the processing load might be generally lower than in the first part of the sentence, due to the presence of linguistic predictions. Subsequent work observing that the RTs of detecting clicks at a clause boundary were modulated by syntactic complexity (e.g., subject relatives vs. object relatives) further corroborated the notion that the click paradigm can reflect processing load during sentence comprehension (Cohen & Mehler, 1996).

In the present work, focusing on sign rather than spoken language, we test whether the click paradigm, in its online response format, might reveal processing load during the comprehension of German Sign Language (DSG). Accordingly, we assume the interpretation put forward by Holmes and Forster (1970, 1972) linking click detection to current processing load. We, however, point out that the click paradigm has also been adopted to investigate linguistic dimensions beyond hierarchical structure (for an overview, see Cutler & Norris, 1979), and, in some of these instances (i.e., open vs. closed class status), the pattern of results was consistent with modulation driven by acoustic low-level features, rather than high-level linguistic ones (Cutler et al., 1993). At the same time, we believe that the results of Holmes and Forster (1970, 1972) are largely consistent with several studies in the fields of psycholinguistics (e.g., Bever, 1988; Miller, 1964; see Levelt, 1995, for an overview) and neurolinguistics (Dehaene et al., 2022; Friederici et al., 2017; Hale et al., 2022; Maran, Friederici, et al., 2022; see Zaccarella & Trettenbrein, 2021, for an overview), demonstrating the pertinence of hierarchical constituent structure during auditory and written language processing, as well as automatic behavioral (Berkovitch & Dehaene, 2019; Pyatigorskaya et al., 2023) and neural (Batterink & Neville, 2013; Hahne & Friederici, 1999; Pulvermüller & Shtyrov, 2003; Pyatigorskaya et al., 2025) effects driven by syntactic structure.

In contrast to the extensively studied auditory and written domains, the link between abstract syntactic structure and cognitive processing in sign language processing remains poorly unerstood. Research over the past decades has demonstrated that sign languages as well as spoken and written languages are both primarily processed in the brain’s left-hemispheric language network (Campbell et al., 2007; Emmorey, 2015; Trettenbrein, Papitto, et al., 2021), with similar structural connectivity in the left, but not the right, hemisphere, compared to hearing individuals (Trettenbrein, Klein, et al., 2025). On the one hand, electroencephalographic studies with evoked potentials have shown that deaf signers’ brains show sensitivity to grammatical violations similar to the evoked potentials known from studies of spoken and written language processing in hearing non-signers (Capek et al., 2009; Hänel-Faulhaber et al., 2014; Hosemann et al., 2018). On the other hand, functional magnetic resonance imaging studies aiming to identify the neural basis of syntactic processing and hierarchical structure in sign language so far have yielded conflicting results (Matchin et al., 2021; Moreno et al., 2018; for in-depth discussion, see Trettenbrein, Zaccarella, et al., 2025). Psycholinguistic work on sign languages took an interest in syntactic processing in sign language already early on (e.g., Grosjean & Lane, 1977; Klima et al., 1979), but then has often focused on lexical and semantic processing as well as modality-specific phenomena, such as iconicity (Baus et al., 2013; Bosworth & Emmorey, 2010; Carreiras et al., 2008; Gutiérrez-Sigut & Baus, 2021; Thompson et al., 2009). To the best of our knowledge, no recent psycholinguistic studies have investigated how processing during sign language comprehension is modulated by hierarchy or constituent structure.

Building on the results obtained with spoken language (Holmes & Forster, 1970, 1972), the present study is the first to probe whether the processing load is modulated by hierarchical constituent structure during sign language comprehension. Accordingly, we adapted the classical psycholinguistic click paradigm from the auditory-oral modality of spoken languages to the visuo-spatial modality of sign languages. Using short white flashes inserted into videos of DGS sentences as analogues to clicks in the auditory domain, we sought to determine whether signers’ speed of detection of extraneous perceptual signals is modulated by their position within the hierarchical structure of signed sentences. We performed a pre-registered online experiment in which deaf signers watched videos of DGS sentences and had to respond to the white flashes, which could occur at different points in the constituent structure, as quickly as possible via button press. Building on Cutler et al.’s (1993) approach in the spoken modality, to rule out the possibility that any potential observed effects are perceptual, rather than linguistic, in nature, we performed a second independent pre-registered online experiment, using the same stimuli and experimental paradigm, with a group of participants not knowing DGS (i.e., hearing non-signers).

To preview our results, we observed no effect of our syntactic manipulations in either experiment. Additional analyses revealed effects of topicalization and uncertainty for both groups. In Section 4, we provide two potential explanations of the observed findings. On the one hand, continuing to maintain the assumptions of Holmes and Forster (1970, 1972) regarding the click paradigm, we argue that the simultaneous and time-shifted presence of syntax-relevant cues (i.e., hands, mouthings, and non-manuals) in the sign stream might have made it difficult to observe a modulation of processing load by hierarchical structure, compared to the spoken modality tested in previous studies. On the other hand, building on van der Burght et al. (2023), we highlight the point that, while our study was built on the assumption that the click paradigm can reflect the perceptual unity of syntactic constituents and ongoing processing load in comprehension (Fodor & Bever, 1965; Garrett et al., 1966; Holmes & Forster, 1970, 1972), not all studies employing this paradigm support this assumption (Abrams & Bever, 1969; Cohen & Mehler, 1996; Cutler et al., 1993; Reber, 1973). Finally, we further argue that non-signers attend to some non-manual cues (i.e., eyebrow raising), despite their lack of sign language knowledge, because they resemble communicative gestures (Herrmann & Pendzich, 2014; Pendzich, 2020).

2. Materials and methods

The experimental design, hypotheses, planned analyses, as well as stimulus materials were pre-registered independently at the Open Science Framework for both experiments. The full text of the pre-registration, including stimulus materials for the main experiment with deaf signers, can be found here: https://doi.org/10.17605/OSF.IO/N29AZ. The full text of the pre-registration for the control experiment with hearing non-signers can be found here: https://doi.org/10.17605/OSF.IO/9FR25.

2.2 Stimuli

The paradigm for both experiments used short flashes inserted into videos of signed sentences as analogues to auditory clicks, which, following Holmes and Forster (1970), could occur in the first or second half of a syntactically complex DGS sentence consisting of a main and a subordinate clause (Figure 1). In this regard, the choice of short visual flashes as an equivalent of auditory clicks converges with the methodology employed by Geers (1978) to test lipreading in both signers and hearing participants. The exact point in time when the flash occurred (indicated by using a ‘/’) differed with regard to the syntactic structure of the sentence, so that a flash could occur after a major break in the constituent structure separating two clauses, after a minor break, or not at a break, as shown in (7)–(9) for the first half of the sentence and in (10)–(12) for the second half.

1. (7)

1. [NEXT MONTH WE HOLIDAY DRIVE CAN] / [OR WE FRIEND FAMILY TOGETHER HOUSE OLD RENOVATE]
1. ‘Next month we can go on vacation or we renovate the old house together with friends and family.’

1. (8)

1. [[IMAGINE] / [[CAR NEW] BIG]]] [IX₁ TWO BICYCLE ONE SCOOTER LOAD]
1. ‘If the new car is big as I imagine, I can load it with two bikes and a scooter.’

1. (9)

1. [THREE FRIEND+++ LIKE [[APARTMENT / BIG] MOVE]] [BUT WITHOUT PARENTS
2.                                                                      neg
3. _3bSUPPORT_3a IX_3a RENT PAY CANNOT]
1. ‘The three friends would like to move into a big apartment, but without their parents supporting them they cannot pay the rent.’

1. (10)

1. [TODAY TRAVEL+GUIDE AND TRAVEL+GROUP WAIT STAND UNTIL
2.                                                                         hn                                       hs
3. CLOUDS DARK DISAPPEAR] / [REASON IX_3pl WET WISH NOT]
1. ‘Today the travel guide and the travel group wait until the dark clouds have disappeared because they do not want to get wet.’

1. (11)

1. [SPORTS+HALL INSIDE CHILD+++ BALL PLAY RELAXED MIX] [[OR TOGETHER PRACTISE] / [FOR NEXT COMPETITION]]
1. ‘In the sports hall, the children casually play with the ball or practise together for the next competition.’

1. (12)

1. [LAST WEEK MY BROTHER CAR TIRE FLAT] [BUT LORRY [FROM / ENGLAND] HELP_3a]
1. ‘Last week my brother had a flat tire, but a lorry from England helped him.’

Stimuli were designed in such a way that there was no systematic bias of the phonological parameter ‘place of articulation’ for the signs immediately following a flash across conditions.

This yielded a 2 × 3 within-subjects design with the factors Position (first vs. second half) and Structure (major vs. minor vs. no break), where each combination of factors comprised six different sentences. Such a design naturally balances out potential technical differences (e.g., operating system, web browser, etc.) amongst participants in online experiments (Bridges et al., 2020). In addition to the six experimental conditions, six additional filler items (14.29%) in which no flash occurred were also included. These trials were included to reduce the possibility that later flashes are detected faster simply because they are more expected, due to the fact that no flash has yet occurred (see Reber, 1973). The stimuli were designed with the aid of deaf informants and recorded in the professional filming facilities of the SignLab at the University of Göttingen by one of the authors, who is a deaf native signer of DGS. To make the stimulus materials look as natural as possible, the signer produced mouthings and non-manuals spontaneously in the same manner as they would produce them in normal signed discourse (Pendzich, 2020; Trettenbrein, Pendzich, et al., 2021).

Figure 1: Structure and motion information for an example stimulus sentence.

Note. The top panel shows an example of the structural description of the constituent structure of a DGS sentence in our stimulus set with the extraneous visual signal located at a minor break in the second half of the sentence (i.e., part of the condition ‘minor-second’; video: https://doi.org/10.6084/m9.figshare.22573105). Representative frames for each individual sign are provided at the terminal nodes of the tree, together with English glosses. To illustrate: If this item had been part of the condition ‘major’, then the extraneous signal would have occurred between the signs MIX and OR. If it had been part of the condition ‘no break’, then the signal could, for example, have occurred between NEXT and COMPETITION. The abbreviations at the different nodes indicate different types of phrases (in order of occurrence in the diagram from left to right): Sentence (S), prepositional phrase (PP), noun phrase (NP), preposition (P), verb phrase (VP), verb (V), adverb (Adv), and conjunction (Conj). This DGS sentence roughly translates into English as ‘In the sports hall, the children casually play with the ball or practise together for the next competition’. The position in the diagram where the extraneous visual signal occurred is indicated by an orange flash. The smaller panel emerging from the position of the flash depicts individual frames in the video at this point in time and illustrates the two consecutive frames which were overlaid to achieve the visual manipulation. The bottom panel shows a plot of the total amount of motion of the signer occurring from frame to frame, as determined by the analysis of data from body-pose estimation performed on the stimulus video clip. The orange bar indicates the position in the video where the flash occurred.

All clips were annotated following a pre-defined coding scheme that was based on the so-called long view of the sign (Jantunen, 2015), before flashes were inserted using a fully automated video-editing procedure (see the file “add_clicks_to_stimuli.Rmd” in the online resources). In essence, this script takes an input video file, splits it into individual frames, then adds an overlay that constitutes the “visual click”, and ultimately recombines the individual files for every frame into a video file. More precisely, an initial set of annotations was created by one coder using ELAN (version 5.7-FX; Lausberg & Sloeties, 2009). A second coder then independently annotated 25% of the stimulus items randomly drawn from the set. This revealed a very high overlap between the annotations by the different coders (Hedges’ g for both the start and end of annotations: –0.02, 95% CI [–.02 –0.01]). Flashes were then inserted automatically, using a custom-made procedure which read the ELAN annotations into R (version 4.0.4; R Core Team, 2019) and added a half-transparent white overlay to two consecutive frames (i.e., for a duration of 80 ms [i.e., 2 frames at a rate of 25 frames per second] during playback, mimicking Holmes & Foster, 1970) at the end of the annotation of the sign occurring immediately before the desired location of the flash in the constituent structure of the sentence.¹ This procedure was designed to mimic the nature and placement of clicks used in the auditory study by Holmes and Forster (1970) as closely as possible in the visual domain.

To control our stimuli for a possible correlation between the manual and bodily motion of the signed sentences (e.g., articulatory pauses) and the probed constituent structure, we used automated motion-tracking to quantify the movement of the signer in the videos (Figure 1, bottom panel). This analysis is intended to capture aspects of sign language prosody encoded by the body and hands, but cannot capture more fine-grained aspects of prosody encoded by the face (Fenlon & Brentari, 2021; Sandler, 2010). We fit a body-pose model capturing the posture of the signer using OpenPose (version 1.7.1; Cao et al., 2019) to all stimulus clips and computed an index for the velocity of this model on a frame-to-frame basis, using the OpenPoseR package (version 1.0.4; Trettenbrein & Zaccarella, 2021) in R. Based on the manual annotations of our stimuli, we then extracted the motion information for the duration of the annotations of all signs articulated immediately before a flash occurred. This indicated that the average motion of signs immediately before the occurrence of a flash (measured in arbitrary units) was very similar across conditions and did not differ systematically: major-first (M = 282.58, SD = 66.73), minor-first (M = 288.56, SD = 74.44), no break-first (M = 261.82, SD = 86.51), major-second (M = 305.18, SD = 101.27), minor-second (M = 332.36, SD = 181.07), and no break-second (M = 303.66, SD = 112.38).

3. Results

3.1 Pre-registered analyses

This subsection only reports the results of the pre-registered analyses for both experiments.

3.1.1 Main experiment

The full model converged, yet, contrary to our hypothesis, neither the factor Structure nor the factor Position could explain the variance observed. Results for the group of deaf signers are plotted in Figure 2. See Table 1 for descriptive statistics. Details of the statistical analysis are provided as part of the Supplementary Materials, in Appendix Table 1.

Figure 2: Main experiment: Responses of deaf signers across conditions.

Note. Illustration of the results from the main experiment with deaf signers. Individual responses, their distribution, and means are plotted for all six experimental conditions resulting from the combination of the different levels of the factors Structure (major, minor, no break) and Position (i.e., first vs. second half of the video). From left to right: major-first, minor-first, no break-first, major-second, minor-second, and no break-second. Reaction times (RT) are given in milliseconds (ms). No statistically significant differences between conditions were observed (all p-values > 0.05).

Table 1

Main experiment: Descriptive statistics.

	First			Second
	Major	Minor	None	Major	Minor	None
Mean	644.00	647.33	640.77	621.08	617.34	636.17
Standard Deviation	161.56	125.11	125.05	141.24	137.84	142.47

• Note. Descriptive statistics (means and standard deviations) for the main experiment with deaf signers per experimental condition.

3.1.2 Control experiment

The full model converged. In line with our hypotheses, we observed no effect of Structure and no Structure × Position interaction. Results for the group of hearing non-signers are plotted in Figure 3. See Table 2 for descriptive statistics. Details of the statistical analysis are provided as part of the Supplementary Materials, in Appendix Table 2.

Figure 3: Control experiment: Responses of hearing non-signers across conditions.

Note. Illustration of the results from the control experiment with hearing non-signers. Individual responses, their distribution, and means are plotted for all six experimental conditions resulting from the combination of the different levels of the factors Structure (major, minor, no break) and Position (i.e., first vs. second half of the video). From left to right: major-first, minor-first, no break-first, major-second, minor-second, and no break-second. Reaction times (RT) are given in milliseconds (ms). No statistically significant differences between conditions were observed (all p-values > 0.05).

Table 2

Control experiment: Descriptive statistics.

	First			Second
	Major	Minor	None	Major	Minor	None
Mean	692.15	693.05	696.19	687.26	673.55	664.21
Standard Deviation	143.02	139.72	125.49	134.34	140.68	130.76

• Note. Descriptive statistics (means and standard deviations) for the control experiment with hearing non-signers per experimental condition.

4. Discussion

In the present study, we set out to probe whether the processing load during comprehension, as measured by RTs in detecting an extraneous perceptual signal (Holmes & Forster, 1970, 1972), was affected by hierarchical constituent structure during sign language processing. We therefore adapted the classical psycholinguistic click paradigm (Fodor & Bever, 1965; Garrett et al., 1966; Holmes & Forster, 1970, 1972), in its online version (Holmes & Forster, 1970), from the auditory-oral modality of spoken languages to the visuo-spatial modality of sign languages. Accordingly, we employed short flashes inserted into videos of DGS sentences as visual analogues to auditory clicks (see Geers, 1978, for a similar methodological approach). Contrary to our hypothesis, we did not observe any effect of our syntactic manipulation in the main experiment with deaf signers. An additional exploratory reanalysis of the data in a Bayesian framework even revealed strong evidence against an effect of Structure in the experiment with deaf signers. In what follows, we will discuss two different explanations that can account for this null finding. First, we continue to maintain the assumption that the click paradigm captures processing load during sentence comprehension (e.g., Holmes & Forster, 1970, 1972) and we highlight how sign languages might provide a linguistic signal which makes comprehension resilient in face of external sensory stimuli. Second, we drop such an assumption and review some conflicting findings in the literature concerning the click paradigm. We then turn to the results of our exploratory analyses, which raise questions about the processing of non-manual components in sign languages by signers and non-signers alike, and about whether the click paradigm might represent a valuable tool for sign-language studies.

Assuming that RTs in the click paradigm, also in its implementation in the visual modality (Geers, 1978), reflect processing load (Holmes & Forster, 1970, 1972), our results suggest that the hierarchical structure of DSG sentences does not modulate this cognitive aspect during comprehension. This raises the question of why, compared to the spoken modality, an extraneous signal sharing the same perceptual modality as linguistic information does not affect processing in sign, as opposed to spoken, languages. Compared to the spoken modality, the complex simultaneous and time-shifted (i.e., different onset and offset points) presence of syntax-relevant cues (i.e., hands, mouthings, and non-manual markers) may make the cognitive processing of the visual signal of sign language more robust against disruption by extraneous signals than the mostly strictly linear speech stream (Figure 4). Thus, multiple articulators might jointly convey information that can be used to optimize ongoing cognitive processing, hence “freeing” resources that, in the current task, could be exploited to detect an extraneous visual signal. To boost the importance of the sequential syntactic information provided by perceived word, Holmes and Forster (1970) controlled their experiment for possible prosodic confounds. Our study attempted to do something similar by using automated motion-tracking on our stimulus clips (see 2.2 above).

Following the logic of Holmes and Forster (1970), the parser is assumed to build up syntactic structure by sequentially integrating incoming words or signs into the constituent structure during sentence comprehension, based on expectations about the continuation of the sentence. Whereas we still concur that this, in principle, is a reasonable assumption, closer inspection of what syntax-relevant information is present at what time in the sign stream, as opposed to the speech stream, makes it clear that the signed signal is much richer (Meier, 2002) and, consequently, provides several affordances that allow the comprehender to maintain optimal performance despite the additional demands imposed by extraneous signals. In other words, the potential disruptive effect on ongoing processing of the extraneous signal (i.e., the flash) at exactly these points in time appears to have been readily compensated for by the syntax-relevant information provided by the other articulators. Along these lines, it has also been argued that the nature of the major articulators of sign languages requires longer transition phases between signs, whereas these transitions are likely already informative with regard to the next sign (Hosemann et al., 2013; Wienholz et al., 2023; also see Grosvald & Corina, 2012). Hence, the different demands on temporal chunking imposed by the visual and auditory modality may lead signers to rely less on serial maintenance of linguistic units (Giezen, 2021).

Interestingly, previous work inspired by the click paradigm showed that visual flashes disrupt lip reading of spoken sentences in deaf signers (Geers, 1978). This work is of both methodological and theoretical relevance for the present findings. On the methodological level, it validates the presentation of flashes to non-hearing signers to disrupt ongoing processing. On the theoretical level, it shows that when extraneous visual signals disrupt communicative information that is serial (i.e., lip movements) and, therefore, not as redundant as signed sentences (i.e., simultaneous effectors), the cognitive load is increased in signers. Therefore, the findings of Geers (1978) are, in principle, compatible with the hypothesis we put forward above. At the same time, we acknowledge that lip reading of spoken sentences might, in itself, pose an increased cognitive load for signers (i.e., the use of a second language and only a proxy of the actual signal being available) as opposed to the processing of signed sentences (i.e., the use of their native language). Accordingly, we believe that our account of the combined results of Geers (1978) and our study might be worth exploring in future studies, comparing the effect of visual flashes on signed sentences and other visual serial communicative information.

Figure 4: Syntax-relevant information present in the linguistic signal across modalities.

Note. A schematic depiction of the different properties of the auditory speech stream tested in classic click studies, opposed to the visual sign stream tested in the present work. The top panel shows the physical properties of the speech stream on top and depicts the different types of syntax-relevant information that can be imposed onto the representation of this signal at the bottom. Notice that the speech stream is, for the most part, strictly sequential, with suprasegmental information being the only exception. The bottom panel shows a still from a video of a DGS sentence to depict some of the relevant physical properties of the sign stream. Notice that the simultaneous and time-shifted (i.e., different onset and offset times) presence of syntax-relevant information is the norm. That is, a mouthing (or mouth gesture) that may accompany a sign (colour-coded in yellow) does not necessarily extend for the same duration as the manual component (hand form, hand position, contact area, starting point, movement, and end point; colour-coded in blue). Notice that the boundary between a sign’s start and end points and transitory movements is blurry (Hosemann et al., 2013; Jantunen, 2015; Wienholz et al., 2023). Similarly, syntax-relevant non-manual cues, such as raised eyebrows indicating topicalization or question components or position and movement of the head and upper body (colour-coded in green), can extend over several signs (Pendzich, 2020; Steinbach, 2023).

In addition to the many-to-one mapping required during sign language comprehension specifically for syntax-relevant cues, the canonical word order of DGS and other languages (sign and spoken) may impact the degree to which predictions of sequential elements constitute a cognitively useful strategy. That is, a central tenet of Holmes and Forster’s (1970) paradigm and, by extension, also of our adaption of this paradigm to the visuo-spatial modality, is that participants’ responses to an extraneous signal will be influenced by what further elements the parser anticipates based on previously encountered elements (i.e., the parser projects and maintains an open node which is reflected by increased processing load). While this is a reasonable assumption on the level of minor and major constituents, such predictive processing may simply not occur on the phrasal level (see Maran, Friederici, et al., 2022; Maran, Numssen, et al., 2022; for discussion), especially in languages where elements such as adjectives or adpositions usually follow the element that they modify (e.g., unlike in English or German, the adjective NEW follows the noun BOOK in DGS, as shown in the example sentence in (4); see Section 1). Accordingly, some trials of the within-phrase manipulation (i.e., the ‘no break’ conditions) in our main experiment may have been affected by this (i.e., the parser does not project an open node after BOOK, instead the structure [BOOK NEW] is only built up if and when a postpositional element is encountered). While the click paradigm has been employed in studies testing spoken languages in which adjectives or modifiers follow, rather than precede, the element they modify (e.g., Italian: van Ooijen et al., 1993; Spanish; Lobina et al., 2018; Mota & Igoa, 2017), they focused on linguistic dimensions other than the ones tested in the present paradigm.

The different points raised in the discussion above rest on the assumption that the click paradigm captures ongoing processing load; however, alternative explanations are possible once this assumption is dropped. That is, while a large number of studies have observed an effect on performance for the positioning of clicks with regard to the syntactic structure of sentences, other studies have questioned the general validity of the paradigm (see also Lobina et al., 2018 for related discussion). For example, Reber (1973) showed that even participants who are instructed to locate a “subliminal click” in a sentence have a tendency to locate it at a constituent boundary, despite the fact no click was actually presented. While this finding is compatible with the idea that constituents are perceptual units in language processing, they are problematic as they suggest that click paradigms might not capture the effect of an extraneous visual signal, but rather an implicit syntactic bias. Furthermore, Cutler and colleagues (1993) showed that the click paradigm might be strongly influenced by low-level properties of the stimulus materials, as similar results can be obtained also in participants without competence in the tested language. While these results (Cutler et al., 1993) were obtained focusing on a different linguistic feature (i.e., the open vs. closed class status of words) than the one tested here (i.e., hierarchical structure), they raise important questions about previous work that adapts this paradigm without ensuring that the effect is present only in participants who know the tested language.

Despite our null finding in the present study, we believe that the linguistic evidence for hierarchical constituent structure in the sign languages of the world is compelling (Cecchetto, 2017; Kocab et al., 2023; Mathur & Rathmann, 2014; Sandler & Lillo-Martin, 2001, 2008; Tang & Lau, 2012). Accordingly, we caution readers not to interpret our failure to observe syntax-specific effects in this study, which uses a comprehension paradigm, as evidence for the lack or irrelevance of hierarchical constituent structure in the comprehension or production of DGS in general. Instead, against the background of the general state of research on constituent structure and hierarchy in language and, specifically, sign language, we believe that either the greater robustness of the sign stream or limitations of the click paradigm provide a reasonable explanation for the observed null result. At the same time, we believe that our findings, and especially the methodological approach we developed for creating stimulus materials, which we make openly available in the dedicated repository, warrant further studies using different variations of the click paradigm (e.g., a version that does not rely on RTs during comprehension, but instead increases the memory demands imposed on participants by requiring them to first encode, and then decode, the perceived sentences; Holmes & Forster, 1972; Seitz & Weber, 1974).

To further explore the data collected, we performed two additional exploratory analyses, which suggested that the presence of non-manual markers for topicalization in some items has a significant effect on the general responses to the “flashes” in both groups of participants, demonstrating that hearing non-signers attend to some non-manual cues of DGS (i.e., raised eyebrows marking topicalized structures), despite their lack of knowledge of the language. Interestingly, the main effect of Topicalization was not just present in the main experiment as well as the control experiment, but also showed the same directionality (i.e., signers and non-signers both were generally slower to respond to topicalized items, regardless of whether the flash occurred in the first or second half of the sentence). Moreover, hearing non-signers’ responses were especially inhibited when the non-manual marker was present at the beginning of stimuli, presumably due to the overlap of this syntactic marker of DGS with a non-manual gesture (Dohen & Lœvenbruck, 2009; Wierzbicka, 2000). Building on Cutler et al. (1993), the fact that the effect is present in both groups rules out an explanation on the grounds of language knowledge. However, the exploratory nature of this analysis does not allow us to determine whether the observed effects are driven by different causes in the two experiments. Despite grammatical and lexical functions of non-manuals in the visuo-spatial modality, both deaf signers and hearing non-signers also use non-manuals to express emotions, reactions, and attitudes (Pendzich, 2020; Wilbur, 2021). Specifically, Pendzich (2020) showed that raised eyebrows presented in isolation, without a signed context, are interpreted as an interrogative marker by deaf signers as well as hearing non-signers, pointing towards a possible gestural origin of this grammatical marker (Domaneschi et al., 2017; Janzen, 1999; Van Loon et al., 2014). Furthermore, this facial cue also showed a strong relation to scepticism for hearing non-signers (Pendzich, 2020).

The finding that signers and non-signers show similar effects for topicalized items, despite the difference in their knowledge of DGS, in combination with the observation that in both experiments, participants were actually slower to respond when non-manual components in the form of raised eyebrows (i.e., topicalization markers) were present in the stimulus, raises questions regarding the cognitive status of non-manuals. In the literature, two different views have been put forward: Some have considered non-manuals as the direct surface realisation of syntax (Emmorey, 2002; Wilbur & Patschke, 1999), whereas others have argued that some aspects of the linguistic analysis of non-manuals may be achieved by other (not necessarily linguistic) cognitive systems (Atkinson et al., 2004), presumably in tandem with the language system. That is, signers are assumed to at least initially recruit additional cognitive resources (e.g., non-linguistic systems for processing facial gestures with a lexicalised meaning, such as ‘top’), but this information is then forwarded to the language system. Our data cannot resolve this question, but they seem to indicate that the use of non-manual components, indeed, imposes additional processing demands, regardless of participants’ knowledge of DGS: Both experiments showed that participants were slower to respond when items included topicalization markers, though only the group of deaf signers could have processed them as such grammaticalised components.

Lastly, the Bayesian reanalysis of the data from both experiments confirmed the null result for our syntactic manipulation, but, at the same time, also revealed evidence for an effect of position, thus indicating that the original effect of position observed by Holmes and Forster (1970) most likely is not linguistic in nature, but instead reflects a general effect of attention. In the original article that served as an inspiration for the present study, Holmes and Forster (1970) concluded that participants’ slower responses to clicks occurring in the first half of the sentence, as opposed to the second half, reflected the decreased processing demands during the second half of the sentence, because more linguistic information has already been encountered, while the kind of new linguistic information that needs to be integrated becomes more predictable. Our adaptation of the paradigm to the visuo-spatial modality of sign language, in combination with the data from the control experiment, allow us reconsider this claim: If the response pattern described by Holmes and Forster was, indeed, only caused by increased cognitive demands during processing complex sentences, we should not have observed an effect of position in our control experiment with hearing non-signers, who did not process the stimuli in a linguistic manner. In contrast, our Bayesian reanalysis showed an effect of Position in both experiments, though more pronounced for the main experiment with deaf signers. Hence, we argue that any observed effect of Position in this paradigm actually does not necessarily reflect decreased linguistic processing demands, but indexes decreasing uncertainty as a trial progresses during the experiment.

5. Conclusion

While linguistic research has demonstrated that sentences exhibit a complex internal structure independent of modality (i.e., speech or sign), our set of pre-registered experiments presented here, which adapted the classical psycholinguistic click paradigm to the visuo-spatial modality, failed to observe any influence of hierarchical constituent structure on cognitive processing load during sign language comprehension. Interestingly, additional exploratory analyses suggested that the behaviour of non-signers was systematically influenced by non-manual cues, despite their lack of knowledge of DGS. The Bayesian reanalysis of the data from both experiments then revealed a general effect of uncertainty, depending on whether the flash occurred in the first or second half of the video, independent of participants’ knowledge of DGS.

Against this background, we conclude that the simultaneous and time-shifted presence of different syntax-relevant cues (i.e., hands, mouthings, and non-manuals) makes the sign stream robust against disruption by extraneous visual signals. At the same time, our data indicate that non-signers without any knowledge of DGS nevertheless attend to the non-manual cue for topicalization (i.e., raised eyebrows), despite not comprehending its syntactic relevance, possibly because its form resembles a communicative gesture for interrogatives which has been grammaticalized in DGS (Pendzich, 2020). Future work should probe the constituent structure of sign languages using memory-based variations of the click paradigms similar to Holmes and Forster (1972).

Appendix Table 1

Main Experiment: Generalised Linear Mixed Model.

Appendix Table 2

Main Experiment: Generalised Linear Mixed Model.

Data accessibility statement

All stimulus clips, task instructions for participants, data, and analysis scripts are publicly available from the Open Science Framework: https://doi.org/10.17605/OSF.IO/9BR3P.

Ethics and consent

All procedures were approved by the local ethics committee at the University of Leipzig (Ethik-Kommission an der Medizinischen Fakultät der Universität Leipzig) as part of application 301/21-ek. Prior to both experiments, participants received instructions in their respective preferred language (DGS or German) and gave informed consent via button press.

Acknowledgements

We would like to thank all deaf and hearing participants, Jens-Michael Cramer for consultation during stimulus construction and for signing the instruction videos used for this study, as well as Joëlle A. M. Schroën for statistical advice and comments on a first draft of this manuscript. This work was funded by the Max Planck Society and the University of Göttingen. During preparation of this manuscript Patrick C. Trettenbrein was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 501984557 which is part of the DFG SPP 2392 “Visual Communication” (ViCom).

Competing interests

The authors have no competing interests to declare.

Authors’ contributions

Patrick C. Trettenbrein: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Validation, Visualization, Writing – original draft, Writing – review & editing; Matteo Maran: Conceptualization, Formal analysis, Methodology, Software, Validation, Visualization, Writing – review & editing; Jan Pohl: Data curation, Investigation, Methodology, Project administration, Software; Thomas A. Finkbeiner: Investigation, Methodology, Resources; Emiliano Zaccarella: Conceptualization, Funding acquisition, Methodology, Supervision; Angela D. Friederici: Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Writing – review & editing; Markus Steinbach: Conceptualization, Funding acquisition, Methodology, Resources, Supervision, Visualization, Writing – review & editing; Nina-Kristin Meister: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Writing – review & editing.

Patrick C. Trettenbrein and Matteo Maran contributed equally to this work and should jointly be considered first authors.

ORCiD IDs

Patrick C. Trettenbrein: https://orcid.org/0000-0003-2233-6720

Matteo Maran: https://orcid.org/0000-0002-5867-3203

Jan Pohl: https://orcid.org/0000-0003-2757-9561

Emiliano Zaccarella: https://orcid.org/0000-0002-5703-1778

Angela D. Friederici: https://orcid.org/0000-0002-6328-865X

Markus Steinbach: https://orcid.org/0000-0002-0870-1915

Nina-Kristin Meister: https://orcid.org/0000-0001-8560-9496