Glossa Psycholinguistics

2.1 Linguistic and dialectal variation

It is by now well-known that speakers of the same language can pattern very differently in their speech production, and display different phonological systems and rules, based on the geographic region in which they live (Baranowski, 2008; Blake & Josey, 2003; Chambers, 2006; Clopper & Bradlow, 2009; Clopper & Pisoni, 2006; Clopper, Pisoni, & de Jong, 2005; Foster et al., 2017; Fridland, 1999; Hubbell, 1950; Labov, 1962, 1966, 1998; Labov et al., 2006; McCullough et al., 2019; Preston, 1993; Sankoff & Blondeau, 2007; Trudgill & Hannah, 2002). The phonemic and subphonemic acoustic cues that signal diverse dialects are perceptible to speakers of a target language, even without significant exposure to dialectal variation (Clopper, 2008; Clopper & Pisoni, 2007). Pronunciation differences can also signal level of proficiency and when a language may have been acquired. These differences may not only be perceptible to a listener and signal the second language (L2) status of the speaker, but may also allow them to be identified as a fluent or native speaker¹ of one specific language rather than another, even when they are speaking English (see, e.g., Hartshorne et al., 2018; Johnson & Newport, 1989; Lenneberg, 1967). Late learners of English who are influenced by their own native language display patterns that reflect morphosyntactic features (see the discussion and references in, e.g., Hopp, 2013), or phonological and/or phonetic features (see, e.g., Baese-Berk et al., 2020; Bent et al., 2008; Davidson, 2006).

While L2 speaker status or regional dialect may be signaled at multiple levels of the speech signal, there is evidence that information encoded at the segmental level is weighted more heavily than information at the suprasegmental or intonational level (Alcorn et al., 2020; Carrie & McKenzie, 2018; Clopper & Pisoni, 2004a, 2007; Clopper, Pisoni, & de Jong, 2005; Flege, 1984; Leemann et al., 2018; Park, 2013; Ruch, 2018; Sereno et al., 2016; Van Bezooijen & Gooskens, 1999). At the same time, suprasegmental information may still play a role (Barkat et al., 1999; Munro, 1995; Munro et al., 2010). Research also indicates that listeners are highly sensitive to the presence of an L2 accent – even when they themselves are not native speakers of the target language (Major, 2007; see also Bradlow & Pisoni, 1999; Munro, 1998; Rogers et al., 2001, 2004; van Wijngaarden, 2001). Previous research has also demonstrated that L2 accents and accents of “non-standard” and/or unfamiliar dialects incur a processing penalty relative to those for native languages and familiar dialects (Adank et al., 2009; Clarke & Garrett, 2004; Floccia et al., 2006; Munro & Derwing, 1995; Wade et al., 2007).

Individual experience with languages and the environments or cultures in which these languages are spoken can improve a listener’s performance (see, e.g., Clarke & Garrett, 2004; Xie et al., 2018). In fact, intensive exposure to a language is shown to result in more accurate production and perception of that language at the segmental level (Best & Strange, 1992; Flege et al., 1997). A further distinction may exist between highly proficient speakers and experienced L2 learners involved in regular contact with a specific language environment and culture, who experience pressure to acquire lexical content and “re-phonologize” the perception of phonological contrasts (Best & Tyler, 2007). While there is evidence that self-reported familiarity with a language may aid in identification of the language in accented speech, this still does not prevent it from being confused with another language in L2 accented speech (Atagi & Bent, 2015; Bent et al., 2016; Derwing & Munro, 1997; McKenzie, 2015).

It thus appears that perceived shared geographic or regional similarity plays a role in classification, in addition to perceived phonological similarity – a point that we leverage and expand upon in the current work. At the same time, while having some familiarity with particular languages may facilitate recognition of those languages via accented speech, lack of familiarity does not necessarily amount to inaccurate accent identification. Perceptible patterns of production by speakers of certain languages may be sufficient to allow listeners to identify a speaker’s native language (see McCullough, 2015; Vieru et al., 2011).

Individual exposure to specific dialects can also improve listeners’ performance. In their survey of attitudinal evaluations of New Zealand English, Australian English, and American English varieties, Bayard et al. (2001) reported that when asked to identify the nationality of a speaker, all native listeners were better at identifying their respective target dialects, as one would anticipate. However, Australian and New Zealand listeners were better at correctly categorizing American English than American English listeners were at categorizing New Zealand and Australian dialects, and New Zealand listeners were more likely to mis-categorize speakers of their dialect as Australian speakers than vice versa. Likewise, Adank et al.’s (2009) participants who were recruited from Glasgow showed degraded performance with Spanish-accented English relative to both standard Southern English and Glaswegian dialects, outperforming their standard English dialect counterparts with the latter dialect. In addition, individual speakers who have more experience with dialectal variety (e.g., through regional mobility) may show superior performance relative to those who have a more insular, region-specific experience (Clopper & Pisoni, 2004a, 2004b, 2006, 2007; Williams et al., 1999; although see Alcorn et al., 2020, and the possibility of effects due to age of exposure). These findings strongly suggest that increased exposure to, and familiarity with, dialects and languages outside of one’s own generally results in enhanced perception and categorization of speakers of those languages and dialects, based on their accents. An open question that the current research seeks to answer is whether similar benefits of broader language exposure extend to listeners consistently exposed to multiple languages, and how specialized this additional linguistic experience must be to be beneficial.

2.2 Free classification task

The current study employs an auditory free classification task, in which participants are presented with audio samples, and must attend to acoustic phonetic attributes of the sound files, or sublexical and subphonemic cues in the speakers’ productions, to perform comparisons across speakers and create their own categories without pre-specified labels or category guides, based on the features they perceive (or not). See Imai (1966), Clopper (2008), and Clopper and Bradlow (2009). Listeners, therefore, choose to create a larger or smaller number of categories, and larger or smaller category sizes, based on their decisions about category membership, which are based on their detection of relevant features. Previous researchers have successfully used free classification, sometimes paired with other tasks, to investigate how listeners from different language backgrounds categorize speakers of different dialects and native speaker backgrounds. A summary of recent auditory free classification tasks is presented in Table 1.² We include the present research in the last row of the table.

With the exception of Atagi and Bent (2016), most free classification tasks to date have not manipulated native and L2 listeners and native and L2 speakers simultaneously within a task. In the current research, we do so, in order to investigate the role of language exposure and linguistic background in perception and classification. Three previous studies cited in Table 1 set the stage for the current work.

Clopper and Bradlow (2009) focused on the categorization of American English dialects. They targeted native and L2 listeners recruited from Northwestern University, a prestigious private Midwestern university in Evanston, Illinois. Across two experiments, their participants included native speakers of American English from diverse backgrounds and L2 speakers. Among the L2 group, the majority (38) were native speakers of Mandarin, and the remaining 30 represented 10–12 diverse languages and language families, thus constituting a very heterogeneous group. While most had spent little time in the US at the time of the study, they had been admitted to Northwestern and, therefore, demonstrated English proficiency in their high TOEFL scores. Clopper and Bradlow’s participants listened to 20 speakers from the TIMIT Acoustic-Phonetic Continuous Speech Corpus (Garofolo et al., 1993), representing four dialect regions in the United States (New England, North, Midland, and South), with no international dialects or L2 speakers. The two experiments differed mainly in the sentence prompt from the TIMIT speakers that listeners heard to perform the classification.

Participants in Clopper and Bradlow’s (2009) free classification task were asked to put all of the talkers from the same part of the country in a group together. Listeners across all three groups (native, L2 Mandarin, and L2 heterogeneous) generated the same number of groups: approximately 6 groups, on average, with three to four speakers per group. Perhaps unsurprisingly, native listeners were more accurate than the two L2 groups in their classifications, placing speakers from the same dialect together in a group and not placing speakers from different dialects together as often. Overall, native and L2 listeners patterned similarly within and across the two experiments. They were likely to identify three to four groups of speakers based on English dialects. Thus, both native and L2 listeners were able to classify native speakers of American English by regional dialect, but native listeners still outperformed their L2 counterparts.

Atagi and Bent (2016) focused on the categorization of a range of accents across diverse languages. They compared the performance of monolingual native listeners of US English, and native speakers of Spanish and Korean as L2 listeners, in a free classification task. Listeners were presented with two English sentences from the Hoosier Database of Native and Nonnative Speech for Children (Atagi & Bent, 2013; Bent, 2014) produced by speakers of various language backgrounds, including English. Native listeners of English were more accurate in their classification than L2 listeners, while listeners with a Spanish or Korean language background were more accurate in their categorizations of speakers of those languages than the native English listeners. What’s more, for all three groups of listeners, the native US English speakers were more closely clustered together compared to speakers from the other language backgrounds, and were treated as more similar to speakers of German and French than to speakers of the Asian languages. The authors concluded that “linguistic experience plays a significant role in shaping listeners’ classification of native and nonnative varieties of English” (p. 257). The authors noted that the largest population of international students at Indiana University (the home university of the native listeners) is Korean, and conjectured that “The high accuracy demonstrated by these native listeners, therefore, may be due to native listeners’ exposure to Korean-accented English on campus” (p. 257). This pattern led them to suggest that future researchers investigate native listeners with different experiences with L2 accents. The current research explicitly does so.

Finally, Bent et al. (2016) has had the most diverse sample of speakers and listeners to date. They employed a range of speaker productions of the script from the Speech Accent Archive (a segment of the exact script used in the present research), representing six U.S. regional dialects, six international English dialects, and twelve L2 accents within the same tasks. The U.S. regional dialects included Mid-Atlantic, Midland, New England, North, South, and West. The international English dialects included dialects spoken in Australia, England, Ireland, New Zealand, Scotland, and South Africa. The L2 accents were produced by speakers of Arabic, French, German, Gujarati, Japanese, Korean, Mandarin, Russian, Spanish, Somali, Swahili, and Thai. Thus, the geographic regions and language families were extremely wide-ranging. All told, there were 72 speakers representing these regions or language backgrounds. All of the listeners in their task were from a monolingual English background, and all were recruited from Indiana University in Bloomington, Indiana, a Midwestern state university with little ethnic and linguistic diversity. Forty-five of the 50 listeners grew up in the Midwest, and 30 grew up in Indiana. Most of the students had not studied abroad, and most had little experience with L2 speakers, some referring to their “foreign professors.”

Bent and colleagues paired together a free classification task (where listeners group speakers together) and a ladder task (where listeners rank talkers relative to their proximity to “standard American English”). Given the range of 24 possible groups, the listeners in Bent et al. (2016)’s study created, on average, 11 groups (range 5–19), with 7 speakers in a group, on average (range 4–14). Thus, despite their own lack of multilingual, multidialectal knowledge, listeners categorized beyond the native/L2 distinction, and created approximately five main groups: (a) International English, (b) a mix of American and International English, and within L2, three clusters including (c) French and German, (d) the Asian languages, and (e) a mixed bag, including Swahili, Russian, and Gujarati. It is these last two categories that led us to ask in the current research if exposure to diverse Asian languages and accents, in particular, and specialized knowledge of these languages within a geographic region, could result in more fine-grained groupings. Thus, we seek to probe further how a listener’s native background and regular exposure to other languages (whether at all, or relative to targeted speech samples) facilitates the categorization of accents.

2.3 Interim conclusions

The following picture emerges from the previous research. First, listeners of various backgrounds perceive a native/L2 contrast among speakers, sometimes even based on minimal segmental linguistic information. They are also able to distinguish among speakers of different dialects within a language. Second, experience with, and exposure to, speakers, languages, and dialects can positively impact – but does not necessarily ensure – accuracy in perception or classification. Third, both L2 and native listeners – even those from a homogeneous monolingual background – can successfully categorize native speakers of English according to regional dialects, and L2 speakers by general language groupings. However, native listeners are more accurate in their identification and classification than L2 listeners.

2.4 Current research

To date, no previous study has explicitly compared native listeners and L2 listeners with a linguistic background in specific target languages in a task involving both variation among native/regional English dialects and L2-accented English. In addition, no previous study has compared monolingual listeners with exposure to multiple languages, monolingual listeners with limited linguistic exposure, and multilingual speakers with or without knowledge of the target languages, to determine whether and how familiarity with linguistic diversity compares with specialized or multilingual competence. The current study not only aims to fill those gaps, it also builds on, and extends, previous research using an auditory free classification task methodology to shed light on the perceptual benefits of multilinguistic exposure and competence.

Our aims are twofold. First, we seek to investigate how accents capturing dialectal variation in English are perceived and classified, both by native English listeners with and without exposure to multiple languages and regional dialects, and by listeners with native proficiency in other languages. Second, we wish to determine how four specific listener groups perceive and classify a selection of Asian-language-accented speech: native speakers of these specific Asian languages, monolingual English speakers with exposure to multiple dialects and languages, including these Asian languages, monolingual English speakers with limited exposure to linguistic variation, and speakers who are multilingual, yet have no proficiency in these Asian languages. Three main questions guided our research and our choice of speakers and listeners.

First, does exposure to linguistic diversity, including consistent exposure to specific languages and accented English spoken by native speakers of those languages, allow for fine-grained categorization of speakers by their accent, based on their dialect, and accurate categorization of speakers by accent, based on their native language-accented English? To answer this question, we chose listeners who are native English speakers attending college in an extremely linguistically diverse context with a significant Asian population of speakers (where these English speakers can hear the Asian languages spoken, and hear Asian-language-accented English), and compared them to a group of similar-aged college students from a more homogeneous and non-/minimally linguistically diverse context who are not regularly exposed to these languages or related L2-accented English. We presented both groups with speech samples from speakers of multiple English dialects (regional and international), and from speakers of nine Asian languages, most, if not all, of which are regularly spoken in the first listener group’s context.

Second, does multilingual status (rather than exposure to multiple accents or specific languages or accents) afford listeners an enhanced perceptual ability to discriminate among accents representing different dialects and language backgrounds, even those which the listeners have no proficiency in? To answer this question, we chose listeners with knowledge of multiple European languages and some English proficiency, asking them to categorize the same speakers. We compare their performance to listeners who are native speakers of these languages, and the monolingual English speakers from a linguistically diverse setting.

Third, does native speaker status result in listeners’ increased ability to perceive and classify speakers not only from their own native language, but from languages spoken in the same general geographic region? To answer this, we constructed our speaker sample of nine Asian languages out of three subgroups of three languages each, representing three geographic regions within Asia. We then asked how listeners representing these three subgroups of Asian languages compared with each other, and with the other listeners in our sample, in categorizing these speakers. We also sought to determine if this fine-grained knowledge of acoustic distinctions within Asian languages might generalize to a refined ability to categorize other accents, notably those representing dialects within English.

3.1 Listeners

293 participants were recruited as listeners. Of these 293, 125 were recruited from a subject pool of introductory Linguistics and Cognitive Science students at Rutgers, The State University of New Jersey – New Brunswick, and were compensated with extra credit in their course. Represented in this population was our monolingual college-age population attending college in a highly ethnically and linguistically diverse setting (both in higher education and in the surrounding geographic location), in which they are exposed to multiple languages, accents, and dialogues on a daily basis, including the Asian languages and accented speech under investigation. The vast majority of these students grew up in the Mid Atlantic or Northeast region of the US. A second pool of 21 monolingual participants were recruited via targeted efforts from small liberal arts colleges (e.g., Swarthmore, Haverford, Bryn Mawr, Smith) or universities in the South (Texas Tech University) in which the participant population is highly homogeneous and majority monolingual, and were compensated with a $5 Amazon gift card for their participation. 147 participants were recruited from the Prolific platform online, and were compensated with a $5 Amazon gift card. No participant reported a history of hearing loss or a communication or speech disorder. Each participant provided informed consent as part of an IRB-approved protocol, conforming to research regulatory guidelines.³

The classification task involved dragging emojis from the left side of the screen into a grid on the right, forming categories of at least three members each. Of the 292 participants, 52 participants were excluded across samples for the following reasons: they did not move any emojis from the left side of the screen into groups on the right side (n = 10); they omitted 8 (>10%) emojis from the task altogether (n = 1); they moved emojis from the left side of the screen into the grid on the right with no distinct grouping whatsoever (n = 14); they took less than 10 minutes to complete the task (n = 21); they did not complete the task (n = 3); they attempted to participate and submit answers twice (n = 2); or they misreported their native language (n = 1). After these exclusions, we were left with 241 participants (female: 189, male: 48, transgender: 1, non-binary: 1, no response: 2). 115 of these participants were recruited from the Rutgers University Linguistics and Cognitive Science subject pool; 21, from outside recruitment; and 105, via Prolific.

A separate group of monolingual English-speaking participants (n = 26) formed a baseline group that performed the free classification task based solely on sorting the emojis, with no sound files linked to images. The performance of this emoji-sorting control group allowed us to determine if there was anything in the images themselves that might have contributed to the formation of consistent categories that could have resembled accent-based groupings, and if there was any correlation between visual properties of the emojis and languages.⁴ The rest of the participants (listeners) were divided into six groups.

The first group was English monolingual listeners from a diverse linguistic context,⁵ recruited directly from Rutgers University – New Brunswick, which is a large state university in the US on the East Coast in central New Jersey (Middlesex County) and near New York, in which they were exposed to diverse languages and dialects) (n = 61). The second group was English monolingual listeners recruited from small liberal arts colleges in the US where the population is largely homogenous and heavily monolingual (n = 21).⁶ The third group was fluent L2 speakers of English and a non-target non-Asian language spoken in Western Europe (i.e., Spanish, French, German, Greek) (n = 58). Measures of English proficiency, such as TOEFL scores, were not available for the L2 participants; they were recruited online on the Prolific platform for the sole purpose of the current study. All self-reported fluency and high proficiency in English.

The fourth, fifth, and sixth groups included fluent L2 speakers of English who are native speakers of at least one of the target Asian languages (n = 75). Within this set of Asian language groups, three subgroups were represented, which are characterized here by geographic region: (a) South Asian (e.g., Bengali, Gujarati, Hindi, Malayalam, Punjabi, Tamil, Telugu, Urdu) (n = 23); (b) Southeast Asian (e.g., Indonesian, Malay, Tagalog-Filipino, Thai, Vietnamese) (n = 27); and (c) East Asian (e.g., Cantonese, Korean, Mandarin, Japanese) (n = 25). Participant recruitment continued and was targeted towards a particular language and speaker samples until the groups and subgroups were comparable in size and robust enough for analysis.

3.2 Stimuli

3.3 Procedure

Individuals were invited to participate in a task called “Call Stella.” They were directed to an online form where they completed an IRB-approved consent form and indicated their language background and proficiency, self-identified gender (male, female, transgender, binary, other), and the languages they speak with native or heritage fluency (and for each of these languages, the age at which they began learning it and where they learned it). For English speakers recruited from the subject pool, we also asked how long they had lived in the US, if they had lived in any other countries, and if so, for how long.

Participants then viewed a 2-minute instructional video to acclimate them to the task before proceeding on to the task proper. In the video, the narrator (a researcher and the second author) welcomed the participant and explained that the experiment would be accessed via Google slides available via a link after the video. The narrator then proceeded to explain the setup of the experiment, using an example with five flower emojis on the left side of the screen instead of the actual emojis used in the experiment. See Figure 1.

Figure 1: Sequence of images from the instructional training video preceding the experimental task demonstrating free classification.

45 emojis were located on the left half of the screen, each arbitrarily representing a different speaker from one of the target languages/dialects. Care was taken not to have any visual features which were associated with the languages or dialects, or which invited the possibility that these superficial perceptual features could influence the classification in any way. The script read by all 45 speakers (see (1) above) was printed below the entire set of emojis. To the right of the emojis was a blank grid to be used as a background for classification (see Figure 2).

Figure 2: Initial pre-classification scene with 45 emojis and script to the left of a blank grid.

Participants were instructed to click on each emoji to listen to the audio file linked to it. They were told that while the passage may seem strange, it was chosen because it highlights differences in the languages and dialects of different speakers. They were also explicitly told that the facial expressions of the emojis were irrelevant, and that they should listen to the audio files, and drag the emojis to the grid, arranging them into clusters based on how similar or different they sound, with similar ones close together. (The instructions for the baseline condition without audio were different, in that participants did not need to listen to the audio files.) The narrator demonstrated each of these steps in the video.

Participants were not required to situate the emojis relative to the grid or next to each other in any particular way. They were told that they could listen to the audio files as many times as they wanted, create as many clusters as they wanted, and put as many emojis in a cluster as they wanted, but that each cluster must have at least three speakers in it. They did not have to listen to the files in any particular order, and they were allowed to move the emojis around on the grid as many times as they wanted or needed to. Participants were not asked to provide labels for their groupings or hypothesize about why speakers/emojis belonged together. They simply had to group them into clusters based on how they sounded.

When the participants were satisfied that they had successfully grouped the emojis into clusters based on how they sounded, the task was complete, and they exited the task. Their work was autosaved on Google slides. Participants were not granted credit or given compensation if they spent less than 10 minutes completing the task, and they were informed of this requirement in advance. Typically, participants took between 15 to 25 minutes to complete the classification clustering. (However, the baseline no-audio condition took approximately 8 to 10 minutes, so the restrictions on timing were loosened.) Once the participants’ classification was completed, a researcher then accessed the slides and took a screenshot of the final slide with the clusters, saving it with the participant number. A representative sample of clustering from a subgroup of participants is presented in Figure 3.

Figure 3: Examples of eight actual participant classifications/clusters of emojis/speakers.

Because our study was run online, not in a laboratory setting, we limited the timing of the experimental session to 30 minutes maximum.¹⁰ Participants were instructed to complete the study in a quiet setting without distractions, in one sitting, and on a laptop, desktop, or tablet (not a smartphone). We recorded the time it took each participant to complete the task, and excluded those who took less than 10 minutes, which we determined via piloting would be the absolute minimum time necessary. (In other studies, we have also excluded participants whose time stamp exceeded a certain amount of time, indicating that they did not take the study in one sitting or were otherwise distracted. This step was unnecessary for this study.).

3.5 Preparation for analysis

The clusters created by each participant were reviewed manually by a researcher (the first or second author), with reliability coding conducted by at least one other researcher for each participant file. The researcher tracked categories by assigning the same number to each member of a category, with the numbers being chosen arbitrarily. To make this concrete, in the classification scheme in the top right corner of Figure 3, there would have been seven categories, and all ten emojis in the group in the top right corner of that grid would have been assigned the same number (e.g., any number between 1 and 7). This strategy provided a count of how many categories each participant created by identifying the maximum number used to indicate grouping. For example, the participant whose output is in the bottom right corner of Figure 3 created nine categories, while the one in the upper left created five. When participants did not sort an emoji into a group, and it was either missing from the classification or remained to the left at the end of the task, this cell was indicated with n/a.

While we instructed participants (listeners) to create categories of at least three speakers, they did not always do so, and occasionally created categories with only one or two members. These 1- or 2-speaker groups were counted as errors and included in all analyses. The n/a classifications were replaced during the clustering analysis with a new category label that varied by participant. So, for example, if an individual made 14 groups of 3 and did not categorize the final 3 (or put them into individual groups), we counted this as three errors. If they grouped two speakers from the same category together and excluded one, this would be one error.

3.6 Results

We carried out multiple statistical analyses using R (R Core Team, 2022) to determine whether language background has an impact on the classification of accented English. We first present the description and rationale for each analysis, then present the results. Figures were generated using ggplot (Wickham, 2016).

3.6.2 Results of the analyses

3.6.2.1 Analysis 1: Number of categories created and error rate analysis

We begin by presenting the number of categories created by the listener groups. Figure 4 shows the total number of categories created by each of our six listener groups and the emoji-sorting control group. Recall that the categories were calculated by the unique groups created by each participant, which could, in principle, range from 1 (if the participant grouped all 45 speakers together into one large category, which we did not expect), to 45 (if the participant decided that none of the 45 speakers should be grouped together, which we also did not expect). In reality, by design, the speakers represented 15 groups of three speakers each (as outlined in 3.2.1). Overall, the average number of groups created by the listeners ranged from 7.73 (SD: 2.68) to 10 (SD: 2.62). The English monolingual group from a linguistically diverse context created the most categories, on average, while the control group created the lowest number of categories, on average. This difference amounts to an effect size of Cohen’s D = .87 (95% CI .39 – 1.34), which is typically considered to be a large effect (Cohen, 2013; Plonsky & Oswald, 2014).

Figure 4: The average number of categories created by each of the six listener groups and the emoji-sorting control group.

3.6.3 Error rates

We then calculated the error rates for these groups. (More details on the process are included in the supplementary data site.) See Table 2 below for the conditional effects for each group, linked to the Poisson regression model. Figure 5 shows the error rates by each group (including the non-listener emoji-sorting control group) in the 2-, 5-, and 15-category classifications, where the vertical lines represent the standard deviation of the error percentage in each group. Single categories were counted as errors and included. As expected, the 2-category error rate was low for all groups (less than 5%), indicating that listeners committed few errors separating English native speakers from L2 speakers. By contrast, in all cases, the 15-category error rate was the highest; each group had a total error rate of over 50%. This result indicated that no listener group created 15 categories of three speakers each, corresponding to the pre-selected speaker profiles from the Speech Accent Archive. Finally, the 5-category error rate fell in between the 2- and 15-category error rate and ranged from 18% to 30%. This pattern reveals that the listener groups committed errors in determining the variety of Asian languages (East, South or Southeast) or in separating American and international English varieties, around 1 in 4 times. Note that the emoji-sorting control group deviated sharply in their error rate from the groups that listened to the sound files to classify them.

Figure 5: Percentage of errors by each group in 2-, 5-, and 15-category classifications.

Table 2: The conditional effects of the number of errors by each group and error classification, based on the Poisson regression model.

Error Type	Group	Conditional Predicted Error Rate	Standard Error	Lower	Upper
2	Control	15.17	0.98	13.33	17.16
2	Non-Asian multilingual	2.74	0.24	2.32	3.23
2	South Asian	1.14	0.22	0.75	1.64
2	Southeast Asian	1.61	0.24	1.19	2.14
2	East Asian	1.60	0.25	1.15	2.17
2	English monolingual homogenous	1.54	0.28	1.06	2.16
2	English monolingual diverse	1.40	0.16	1.11	1.73
5	Control	26.11	1.43	23.42	29.08
5	Non-Asian multilingual	13.29	0.59	12.20	14.48
5	South Asian	8.08	0.68	6.87	9.51
5	Southeast Asian	10.04	0.71	8.73	11.46
5	East Asian	9.91	0.75	8.58	11.45
5	English monolingual homogenous	10.37	0.81	8.85	12.07
5	English monolingual diverse	9.03	0.44	8.19	9.92
15	Control	33.60	1.76	30.36	37.22
15	Non-Asian multilingual	27.10	0.99	25.23	29.17
15	South Asian	25.16	1.44	22.51	28.22
15	Southeast Asian	25.03	1.37	22.47	27.82
15	East Asian	24.69	1.35	22.12	27.32
15	English monolingual homogenous	26.03	1.55	22.97	29.28
15	English monolingual diverse	23.45	0.85	21.81	25.14

We then took a closer look at the error rate for the listener groups, comparing the English monolinguals from a linguistically diverse context, the English monolinguals from a linguistically homogeneous context, the non-Asian multilinguals, and the three groups of Asian language speakers. Figure 6 shows the posterior distributions of the predicted number of errors by each of these groups in each error classification. The distributions are made up of 4000 plausible estimates generated by the model (that is, those that are possible or able to occur, versus likely to occur; the posterior distribution is a distribution of outcomes that are able to occur, with some more likely than others). Each point represents the most plausible estimate (the mean of the posterior distribution), and the point interval represents the 66% and 95% of highest density intervals of the posterior distribution.

Figure 6: The predicted error rate per group in each error classification, including 2-category, 5-category, and 15-category error rates.

Overall, the emoji-sorting control group was predicted to produce the highest number of errors in all three error classification types. Their 2-category classification error number was predicted to be 15.16 [95% HDI 13.3–17.1], while their 5-category classification error number was 26.1 [95% HDI 23.4–29] and their 15-category classification error number was predicted to be 33.6% [95% HDI 30.4–37.2]. Accordingly, in Table 2, they also have the largest conditional predicted error rate. Of all of the listener groups, the non-Asian multilingual group had the second highest number of predicted errors: the model predicted an error total of 2.7 [95% HDI 2.3–3.2] in the 2-category error classification, 13.3 [95% HDI 12.2–14.4] in 5-category error classification, and 27.01 in 15-category error classification [95% HDI 25.2–29.1]. They also had a higher conditional predicted error rate than the other listener groups, who all seemed to have comparable error rates. Finally, the English monolingual diverse group had the lowest predicted number of 15 category errors (23.4 [95% HDI 21.8–25.1]) – a finding that may reflect the fact that listeners with native English proficiency who are exposed to multiple accents are best at discriminating among accents representing English dialects. In fact, the English monolingual diverse group consistently had lower predicted error rates than their English monolingual homogeneous counterparts, which comes across clearly in the 5- and 15-category error rates in Figure 6 and Table 2.

3.6.4 Analysis 2: Cluster analyses and additive trees

We now turn to our cluster analysis and additive trees, which provide an estimation of both the quantity of clusters created by each group and the group membership within each cluster. Neighbor-Joining trees (Saitou & Nei, 1987) were created in R, using the nj function. The x-axis of each of these plots represents perceptual similarity, such that the rightmost nodes are the most supported in clustering, while the leftmost nodes are the least supported. The distance of the languages on the vertical axis is irrelevant. The colors of the text represent the five pre-selected speaker groups mapping onto individual emoji images (American English dialects: purple; International English variants: blue;¹¹ East Asian: yellow; South Asian: red; Southeast Asian: orange). Below, we present an additive tree for each listener group, within which all five speaker groups are depicted. These trees continue to pull back the layers of the onion, revealing how language-specific exposure and knowledge benefits the perception of accents and the categorization of speakers.

Before turning to the results for the six listener groups, we want to be sure that participants in our free classification task were not influenced by the visual attributes of the emojis themselves when creating categories. While we had explicitly told participants verbally and in written instructions that the emojis were irrelevant, and they should sort by what they heard in the sound files, we could not be sure that the images did not influence their categorization. As we mentioned, we therefore ran a version of the task in which an independent group of monolingual English speakers was recruited from the same linguistically diverse context as the first monolingual group of participants. This group was simply instructed to “arrange the emojis into clusters any way you think they should be grouped. However you group them is up to you, but find some systematic way to group them!” Like the other groups, they were also told that each group should have at least three members.¹² The results of this control group are presented in Figure 7. As the figure shows, no consistent classification pattern surfaced, and similar dialects and languages were generally not grouped together.

Complementing the lack of any consistent pattern displayed in the additive tree, and the lack of groupings that could be perceived as based on the accents in the corresponding sound files, are the comments from a post-classification query that we included in this specific version of the task. We explicitly asked this group, unlike the other participant groups, to report on their classification strategy afterwards, thereby allowing us to determine if they were guided by a principled pattern (or, at least, were aware of one). While most participants were unsurprisingly influenced by the emotion or affect depicted, and by salient perceptual features, a wide variety of self-reported strategies of grouping and a wide variety in the resulting number of categories surfaced, as captured in (2–5). Thus, we can be confident when we turn to the listener groups that any groupings reflecting English dialects or languages are driven by perceived accent.

1. (2)

1. I have 5 groups of emojis. I grouped the top left with all emojis I thought showed happiness. Middle left is a bit more than happy. I would describe these emojis as silly. Bottom left is miscellaneous, but have personality. Top right shows negative emotions that are not sad. Middle right is all the emojis I perceived to show sadness.

1. (3)

1. I grouped the emojis based on how I use them. The top left section is the shocked/annoyed/frustrated group. The middle left section is the shocked/confused group. The bottom left section is the sad/sick/burnt out group. The top middle section is the happy group. The middle section is the funny group. The top right section is the flirty/lovey group. The middle right section is the goofy/silly group. The bottom right section is the annoyed/mad/ frustrated group.

1. (4)

1. Group 1: water/tear drops, Group 2: rosy cheeks, Group 3: distinct eyes, Group 4: showing teeth, Group 5: dot eyes, Group 6: line for mouth/eyes, Group 7: tongues, Group 8: expressions w/ eyebrows, Group 9: emojis with extra features/accessories

1. (5)

1. From left to right: Happy, Sad, Distressed, Miscellaneous.

Figure 7: Non-listener emoji-sorting control group additive tree.

We now turn to our six listener groups. Figure 8 shows the additive tree for the English monolingual listeners from a diverse linguistic context. The tree reveals two distinct clusters for the American English and International English categories (in the middle), which we predicted would happen. The listeners’ ability to categorize the New England speakers together also reflects their familiarity with this dialect. They also grouped together the South Asian languages (at the top) and the East Asian languages (at the bottom), but struggled to group the Southeast Asian languages together, or consistently within an Asian group. These distinctions are entirely consistent with these listeners’ exposure to specific Asian-language accents.

Figure 8: English monolingual group from diverse linguistic context additive tree.

Figure 9 shows the additive tree for the English monolingual listeners from a homogeneous context. The tree reveals a clear division between English and Asian-language-accented English, and within English, distinctions between American and International English dialects. However, these listeners failed to distinguish between Asian accents, and show no consistent subgroupings (though they do appear to diverge from the control group in Figure 7). In fact, even within American English dialects, these listeners struggled to make distinctions. Thus, the English monolingual group from a diverse linguistic context received an advantage in their ability to classify American English dialects and Asian-language-accented English through their consistent exposure to these accents.

Figure 9: English monolingual group from homogeneous linguistic context additive tree.

We turn next to the tree for the non-Asian multilingual listeners in Figure 10. Like the first English monolingual group (from a diverse linguistic context), this group also successfully clustered American English, International English and South Asian language accents. However, they struggled to successfully categorize both the East and Southeast Asian language accents, and they also did not group together any American English dialects. Thus, in this direct comparison between Figures 8, 9, 10 and their corresponding listener groups, we see clear and definite benefits of specific types of linguistic exposure.

Figure 10: Non-Asian multilingual group additive tree.

We turn now to the listeners with an Asian language background. Figure 11 presents the tree for the South Asian group. This group had a clear split between English language varieties (top) and Asian language accent varieties (bottom). In addition, both English groups belonged to distinct clusters (with the exception of one speaker of a New England dialect), although there was no grouping of dialects or varieties within either English group. Like the first English monolingual group, they grouped all of the South Asian speakers together, although they did not allow other languages to intrude in the cluster, and they grouped all of the Bengali speakers together, but unlike the English group, they mixed together the East and Southeast Asian speakers. Thus, their linguistic background afforded them the advantage of more accurately classifying South Asian language accents, but not classifying subgroups of Asian language accents in general, or perceiving distinctions within English dialects.

Figure 11: South Asian group additive tree.

Figure 12 presents the tree for the East Asian language group. They, too, successfully distinguished American English from International English. Additionally, the East Asian group showed some evidence of successful subgrouping clusters within the 15-category level. Perhaps unsurprisingly, they clustered all three Mandarin speakers together, and were somewhat able to distinguish between Japanese and Korean speakers. They also clustered six of the nine South Asian speakers together without distinguishing between the subgroups of speakers, and also successfully grouped together the Thai speakers. Rather surprisingly, they grouped all Southern American English speakers together, with one New England speaker and all Midland speakers together.

Figure 12: East Asian group additive tree.

Finally, we turn to the Southeast Asian group in Figure 13. This group separated the English native speakers from the Asian L2 speakers, but unlike the other groups, mixed together the American and International English speakers. This group showed evidence of some success with smaller clusters of all three Asian groups, and was perhaps the most successful of all the listener groups in grouping together speakers of Southeast Asian languages (although not at a comparable level to how the other Asian listeners did with speakers of their language groups), and also did so with many of the South Asian speakers.

Figure 13: Southeast Asian group additive tree.

The results and visualizations of a two-dimensional multidimensional scaling (MDS) analysis complementing these additive trees are included in the supplementary data site.