Non-linguistic factors leave a distinct thumbprint on our speech production that is perceptible to listeners. A steadily growing line of research demonstrates that listeners can perceive a contrast between native and non-native (L2) speakers based on accents and further classify these speakers according to dialectal variation, even when they are not native speakers of a language. Most of these studies have focused on dialectal variation within US English speakers, a combination of US and International English dialects, or L2 speakers representing a wide range of languages. Most have also featured listeners who are monolingual native speakers of the target language coming from a homogenous background, or a contrast between these and a targeted set of L2 speakers. We therefore lack knowledge of how exposure to, or familiarity with, diverse accents and languages, or specific native language competence of the native language of L2 speakers, can guide listeners’ accent perception and categorization. In this research, we employed a free classification task, presenting listeners with speech samples of native speakers with accents representing multiple English dialects, and L2 speakers of nine Asian languages across three geographic regions speaking Asian-accented English. There were six groups of listeners: monolingual US English listeners in a diverse linguistic context, monolingual US English listeners in a homogeneous linguistic context, native speakers of a non-Asian language and English (bilinguals), and native speakers of each of the three target Asian language groups who are L2 speakers of English. The results reveal that nearly all listeners are sensitive to accents capturing native/L2 contrasts and dialectal variation in English. While regular exposure to a diversity of accents results in increased classification accuracy, classification of Asian L2-accented English speakers is best performed when there is alignment of similar language family and geographic area, as demonstrated by South Asian listeners.