Test-Takers’ Perceptions of the Inclusion of Different English Accent Varieties in the IELTS Listening Tests

Abstract

Although scholars have argued for the inclusion of both standard and nonstandard English accents in second language English listening tests to improve the test's construct- and ecological validity, little is known about other stakeholders’ perspectives. The present study explored test-takers’ perspectives of the inclusion of different spoken varieties of English in the IELTS (International English Language Testing System) listening module in relation to geographic locations. Some 241 potential and recent IELTS test takers from three geographic locations were sampled (US, South Korea, and China). An exploratory factor analysis of items probing test-taker attitudes toward different accent varieties in the listening test resulted in a three-factor solution (acceptance of standard varieties, acceptance of nonstandard varieties, and overall acceptance of both standard and nonstandard spoken varieties). Overall, regardless of their own geographic background, test takers strongly preferred standard English accents. These findings, grounded in test-takers’ perspectives, have implications for language learning and teaching and further, for test development and administration.

Keywords

Global Englishes listening assessment test-taker perspectives IELTS World Englishes

Introduction

Scholars argue that second language (L2) English listening tests should include a variety of accents other than the “standard” American and British (Hamp-Lyons and Davies, 2008). This is because English nowadays is actively used by speakers with diverse accents (Eberhard et al., 2022), and the inclusion of such varieties would improve the representativeness of the listening materials in the test as well as the ecological validity of the test scores (Harding, 2012). Nonetheless, the field has not yet witnessed any material change to the existing language tests for at least two reasons. First, it is not clear to what extent the inclusion of different accent varieties might threaten test fairness. Specifically, when test takers hear familiar accents, they may have an advantage over other test takers (Taylor and Geranpayeh, 2011). Second, it is not clear to what extent stakeholders might appreciate the inclusion of different spoken varieties. This relates to the face validity of language assessment (i.e., whether stakeholders such as test takers believe that the test is valid (Loewen and Plonsky, 2016; Nevo, 1985); this aspect has the potential to influence test-takers’ performance (Ahn and Moore, 2011).

While several studies have explored how to maintain test fairness when introducing different spoken varieties into L2 listening modules (e.g., Kang et al., 2020), very few have sought to explore stakeholders’ perspectives of the inclusion of different accents in high-stakes English proficiency tests (Hamid, 2014). This study reports on potential and recent International English Language Testing System (IELTS) test-takers’ perceptions of the inclusion of different spoken varieties of English in the listening module. IELTS is one of the world's most popular English language proficiency tests for higher education and global migration (IELTS, 2022). The findings of the study address under-researched stakeholder perspectives, which could inform test development in both current and future language assessments.

IELTS “is designed to help you achieve your ambition of working, studying or moving to a country where English is the native language” (IELTS, 2023a). Furthermore, IELTS reports that its listening section includes accents representative of the following geographic locations: the North of England, the South of England, Scotland, Wales, Northern USA, Southern USA, Canada, Australia, New Zealand, and South Africa (IELTS, 2023b), although the Academic Student's Book with Answers with Audio with Resource Bank: Authentic Practice Tests series includes primarily accents from England, Australia, New Zealand, and Northern USA, with very few featuring other accents such as Japanese and Indian (Cambridge University Press & Assessment, 2023). With four sections, the IELTS listening test encompasses monologs (Sections 2 and 4) and dialogs (Sections 1 and 3) ranging from day-to-day topics (Sections 1 and 2) to more academic ones (Sections 3 and 4) (see IELTS, 2023c).

Literature Review

Rationale for the Inclusion of Different Spoken Varieties in Language Tests

At least two reasons motivated applied linguists and assessment specialists to include different spoken varieties of accent in L2 English listening tests: a) construct validity and b) ecological validity. Construct validity refers to how well a language test actually assesses the ability of test takers in a specific target language use (TLU) domain that it claims to assess (Bachman and Palmer, 2010; Messick, 1996). The definition of this concept has witnessed changes over the years, meaning construct validity is not a static but rather a fluid idea, subject to interpretation, depending on the context of the language assessment and on the different TLU domains (Chapelle and Lee, 2022; Messick, 1989).

Applying this idea in the context of L2 listening assessment, scholars have argued that, given the sheer percentage of L2 English users worldwide, perhaps language testers should reconsider the construct that traditional listening tests were designed to measure (Elder and Davies, 2006; Harding and McNamara, 2018). That is, scholars hope to reconceptualize the measured construct to include different spoken varieties in light of the diverse English users globally. One way to achieve this is by modifying listening materials to reflect the new construct, that is, by including different accent varieties in listening tests.

Related to construct validity is the notion of ecological validity: the extent to which performance on a test might transfer to real-world performance (Meiselman, 2019). Because there are more L2 than L1 English speakers, scholars argue that perhaps test-takers’ performance when listening to standard accent varieties may not fully transfer to their real-life performance. In other words, their ability to understand standard accents does not mean that they will understand other accents equally well (Major et al., 2002). One way to ensure the ecological validity of contemporary listening assessments is to make sure that the listening materials used in the test are grounded in real-life English, encompassing diverse accents (Harding, 2012).

Challenges for the Inclusion of Different Spoken Varieties in Language Tests

The call for the inclusion of different spoken varieties of accent is theoretically sound: it expresses language testers’ interests in further improving current language tests in light of globalization. However, there are several practical considerations that could make it difficult to accomplish this theoretical proposal.

The first challenge is related to fairness. Were scholars to arbitrarily introduce diverse spoken varieties in English tests, the accents could differentially benefit or disadvantage the listeners’ comprehension, depending on their linguistic background (Taylor and Geranpayeh, 2011). For example, the “shared-L1 effect,” documented by Bent and Bradlow (2003), found that when listening to different accent varieties, listeners with the same L1 were able to better transcribe the accented speech orthographically. Similarly, Major et al. (2002) found a shared-L1 effect in L1 Spanish test-takers’ performance in the Test of English as a Foreign Language exam, but not among L1 Mandarin test-takers. Authors1 (Year) also found support for the shared-L1 effect in Indian and South African test takers, but not in test takers from China and Mexico. These studies provide preliminary yet concerning findings that arbitrary inclusion of different accent varieties in the test may indeed differentially influence test-takers’ performance, posing a threat to test fairness.

Seeking to address this concern, scholars have explored how to minimize the shared-L1 effect by strategically selecting accented speakers. Authors1 (Year), for instance, found the presence of a shared-L1 effect from speakers of certain World Englishes. However, this effect was no longer significant when the accented speakers were highly intelligible. That is, above a certain intelligibility threshold, accented speakers can be used to produce test materials without jeopardizing fairness in high-stakes language tests. This result found support in a study by Ockey and French (2016) exploring test-takers’ performance in listening to American, British, and Australian varieties. They concluded that the degree of accent was correlated with test-taker performance, but when the degree of accent was very low, the use of non-familiar varieties no longer impeded test-taker performance.

In addition to research into fairness, the second challenge for including different accent varieties in listening tests relates to face validity, that is, different stakeholders’ beliefs and perceptions of the validity of the test. Research has shown that test-takers’ perceptions of a test have the potential to influence test-taker performance. Chan et al. (1997) found that test-takers’ perceptions of their tests influenced their test-taking motivation, which in turn influenced their test performance. Similarly, Bai (2020) found that test-takers’ attitudes toward the test’s use were correlated with their actual test performance.

Perceptions of Different English Spoken Varieties Across Geographic Locations

Including different spoken varieties in high-stakes listening tests implies substantial changes to any existing test materials. However, very few studies have sought to explore different stakeholders’ perceptions of such a change, particularly across diverse geographic locations. Of the limited studies about people's perceptions of different spoken varieties of English, most were not conducted in the context of language assessment, meaning that the findings do not have direct implications for language assessment. For example, studies have found that English learners generally prefer the standard varieties of English, including American and British English, especially when they may be unaware of relevant research on World Englishes (Boonsuk et al., 2021). However, learners from different geographic locations seem to diverge in their opinions, especially regarding attitudes toward their own accents (Shibata, 2021). This means that geographic locations may influence people's perceptions. Potentially, Asian test takers in East Asia might have different perspectives toward such accent inclusion issues from those in North America. Therefore, investigating stakeholders’ attitudes toward this change across different countries may provide additional insights useful for test design and development with location-specific implications.

For this reason, the purpose of this study was to explore test-takers’ perceptions of the inclusion of different accent varieties in the IELTS test across three different countries. The study was guided by the following research questions:

What are test-takers’ perceptions of the inclusion of different spoken varieties in IELTS?

How do test-takers’ perceptions differ across the three selected geographic locations (the US, China, and South Korea)?

Methodology

Participants

A total of 241 participants took part in the study. The inclusion criteria were that the participants a) planned to take the IELTS exam in the next 12 months, b) were preparing for the IELTS exam, or c) had completed the IELTS exam in the 12 months prior to the study. We believe that this inclusion criteria reflected test takers’ current and recent views. The participants were based in three geographic locations: South Korea (n = 81), China (n = 80), and the US (n = 80). These countries were selected as they represent the demographic and geographic popularity of IELTS test-takers (IELTS, 2022). The participants were recruited with the help of designated coordinators in South Korea and China. In the US, participants were recruited through personal contacts at the different universities of the research team and with IELTS USA's support in distributing advertisements to IELTS test-takers at different learning and testing centers. For a more detailed description of the sample and the recruitment approach, see the study by Kang et al. (2022).

Questionnaire

An online questionnaire was used to collect perception data from the test takers. The questionnaire elicited participants’ responses to (a) the background survey and (b) the survey of test-takers’ attitudes about the inclusion of different spoken varieties of English in the IELTS listening test. First, we probed test takers’ overall attitudes toward native and nonnative accents in the IELTS listening test. Specifically, using a 100-point Likert scale (0 = strongly disagree, 100 = strongly agree), we asked the test takers to indicate to what extent the IELTS listening test should include a) native English accents only and b) both native and nonnative English accents. Second, we took an analytic approach and asked the test takers to evaluate the acceptability of nine English varieties in the IELTS listening test, including UK, US, Australian, Indian, Singaporean, Egyptian, Chinese, Japanese, and Korean. Specifically, they were asked to what extent they thought different English accents would be acceptable in the IELTS listening test. This question was designed on the basis on Kachru's (1983) categorization of English users into the inner- (i.e., where English is used as a first langauge among people such as the United States), the outer- (i.e., where there is a historical colonial relationship between the United Kindom and where English is commonly used, such as India), and the expanding circle (i.e., where English is used as a foreign language primarily learned in school such as South Korea). Each circle was realized by two to four different items (i.e., countries). Finally, we asked the test takers to indicate (on a 100-point numerical scale) to what extent they would be happy to hear accents similar to their own L1 accent in the IELTS listening test.

In addition to collating quantitative data, we asked participants to provide reasons why they thought IELTS should include native English accents or both native and nonnative accents. Similarly, we asked the test takers to explain the scores they had provided for the inclusion of their own accents. For details about the data collection instrument, see Authors2 (Year).

Data Analysis

Our first research question probed test-takers’ perceptions of the inclusion of different spoken varieties in the IELTS listening test. Because of the potential conceptual overlap between the questions eliciting quantitative data, they were subject to an exploratory factor analysis (EFA) to investigate their underlying factor structure. Internal consistency across items was computed using omega coefficients (see DeVellis, 2017). Descriptive statistics were then provided for each factor to give a comprehensive view of test-takers’ attitudes toward the inclusion of different spoken varieties. Between-factor comparisons were conducted using the Kruskal–Wallis H test, because the data deviated significantly from normality. Our second research question used Kruskal–Wallis H tests to explore the extent to which test-takers’ perceptions differed according to their geographic locations. Relevant effect sizes and 95% confidence intervals were also reported to assist in data interpretation. Test-takers’ qualitative comments are presented in the next section, to explain and supplement the numerical findings.

Findings

Test-Takers’ Perceptions of the Inclusion of Different Accent Varieties

As stated, all 12 questions probing test-takers’ perceptions of the inclusion of different accents were subject to an EFA. Prior to that, statistical assumptions pertaining to the EFA was assessed. First, an inspection of the correlation matrix suggested the presence of numerous coefficients of .3 and above. Second, the Kaiser–Meyer–Olkin measure of sampling adequacy was .84, confirming sampling adequacy. Third, Bartlett's test of sphericity supported the factorability of the correlation matrix, χ² (66) = 1475.90, p < .001.

Parallel analyses suggested three factors underlying the data. This was supported by the inspection of the scree plot, which revealed a clear break after the first factor. Therefore, a three-factored model was identified, explaining 58.80% of the variance. Factor loadings are presented in Table 1. It is clear from the data that the three factors were highly interpretable. Factor 1 included all nonstandard varieties of English, explaining 31.10% of the variance. Factor 2 included all standard varieties of English, explaining an additional 12.9% of the variance. Factor 3 seemed to be a proxy for the listeners’ overall acceptance of the inclusion of different spoken accents, explaining an additional 13.80% of the variance.

Table 1.

Factor loadings for 12 questionnaire items.

	Factor 1	Factor 2	Factor 3
Egyptian	0.92
Japanese	0.86
Korean	0.73
Singaporean	0.68
Chinese	0.66
Indian	0.63
British		0.77
American		0.66
Australian		0.55
Both NS and NNS accents			0.94
NS accents only			–0.71
Test takers’ own accents

Note. NS: native speaker; NNS: nonnative speaker.

Following the EFA, three composite variables were created to represent the three factors underlying the 12 items used: Factor 1 (acceptance of nonstandard spoken varieties), Factor 2 (acceptance of standard variety), and Factor 3 (overall acceptance of both standard and nonstandard spoken varieties). The variable with a negative factor loading was inverted before the composite variables were created (i.e., preference for nonstandard accents only; see Table 1). Moreover, because one item (i.e., perceptions of one's own accent) did not seem to fall into any of the three factors, this question was treated as one item, and we reported any descriptive statistics of it as well (Table 2).

Table 2.

Descriptive statistics.

		Mean (SD)	Median
Factor 1: Acceptance of nonstandard varieties	Overall	38.33 (25.43)	33.83
	South Korea	37.88 (24.61)	37.67
	China	39.53 (22.33)	36.08
	US	37.59 (29.18)	31.42
Factor 2: Acceptance of standard varieties	Overall	78.45 (19.17)	82.00
	South Korea	75.31 (21.55)	79.00
	China	80.64 (14.16)	81.67
	US	79.52 (20.66)	85.33
Factor 3: Overall acceptance of both standard and nonstandard spoken varieties	Overall	52.66 (31.69)	51.00
	South Korea	53.67 (29.34)	51.00
	China	51.19 (32.06)	50.50
	US	53.07 (33.96)	52.25
Item: Preference for test takers’ own accents	Overall	69.82 (25.64)	72.00
	South Korea	68.05 (25.73)	72.00
	China	70.42 (24.13)	72.00
	US	71.05 (27.19)	74.50

Note. The scales ranged from 0.00 to 100.00. Higher values meant more positive ratings.

After all the factors were identified, composite variables were created by averaging the values across the items. Internal consistency across items was computed using omega coefficients. Table 3 presents a summary. Specifically, all factors demonstrated reasonably acceptable internal consistency, except for Factor 2 (ω = .69). Revisiting the data, we found that one item (i.e., acceptance of Australian accents) could be grouped into Factor 2 (standard varieties; factor loading = .55) and also tentatively into Factor 1 (nonstandard varieties; factor loading = .39). The fact that this item could possibly explain both factors might have resulted in the dip in internal consistency within the factor. This means that for some test takers, Australian accents were categorized as nonstandard:

I do not support the inclusion of local accent, because many accents differ so much from standard accents. For example, it's very difficult me to understand Indian and Australian English (#34, China).

I think in many times the accent actually affects my understanding about the speaker's meaning. For instance, once I traveled in Australia … it was really hard for me to understand the Australian English. But In Canada or America it is ok (#14, China).

Table 3.

Internal consistency across items.

	ω (SE)	95% CI
Factor 1: Acceptance of nonstandard varieties	.90 (0.01)	[0.87, 0.92]
Factor 2: Acceptance of standard varieties	.69 (0.05)	[0.58, 0.77]
Factor 3: Overall acceptance of both standard and nonstandard spoken varieties	.83 (0.03)	[0.76, 0.87]

Next, items within each factor were averaged to create three composite variables. Descriptive statistics for all factors and the item (that did not belong to these factors) are presented in Table 2. A Kruskal–Wallis H test showed that there was a statistically significant difference in test-takers’ acceptance of accent varieties across the four factors (i.e., three factors and one item), χ²(3) = 246, p < .001, η² = .26, 95% CI [.21, .31]. Specifically, test-takers’ acceptance showed the following patterns: standard varieties (M = 78.45, Mdn = 82.00) > test takers’ own accents (M = 69.82, Mdn = 72.00) > both standard and nonstandard spoken varieties of accent (M = 52.66, Mdn = 51.00) > nonstandard varieties (M = 38.33, Mdn = 33.83). Each pairwise comparison was statistically significant at the p < .001 level, except for the first pair (i.e., standard varieties vs. test takers’ own accents) at p = .0013 (all p-values were adjusted for multiple comparisons). That is, our findings showed that IELTS test-takers strongly preferred the standard varieties of English, and their acceptance rates for nonstandard varieties dropped significantly.

Reasons for Test-Takers’ Attitudes

While descriptive statistics suggested that test-takers’ acceptance of nonstandard varieties was significantly lower than that of standard varieties, their responses were widely dispersed within each factor. Thus, we will present the test-takers’ perspective from both sides. A total of 174 valid comments were obtained from the participants, including those with positive (55.74%), negative (31.61%), and neutral attitudes (12.64%) in which they mentioned both the advantages and disadvantages of including different accent varieties in the IELTS listening test. The comments that suggested positive attitudes toward the inclusion of different spoken varieties mainly argued for the increased ecological validity such inclusions afforded (Harding, 2012):

English has been widely used by people around the globe and people with different first language would normally speak English with accents … but communication is not always going on between a native English speaker and a foreigner … we are likely to communicate with people from any other countries other than English speaking countries. Under the circumstances, English is a intermediate language for both speakers. Hence … IELTS listening that includes native and non-native English accents is necessary (#49, China).

I think it is important to understand English in all different accents since English is the international language that everyone use (#93, Korea).

In real life, you will actually meet both native speakers and non-native speakers (#200, US).

However, many test takers argued against the inclusion of different spoken varieties, giving four main reasons. First, many believed that different accents were associated with decreased intelligibility or comprehensibility (mentioned by 24.59% respondents with negative attitudes). This perception supported the research findings observed in the literature (see e.g., Major et al., 2002):

Because native English accents are more easy to understand and clearer. Also, it is highly likely that your accent also becomes somewhere strange and you may pronounce poorly after listening to non-native English accents frequently (#25, China).

It is sometimes difficult to understand what they’re trying to say (#87, Korea).

Different accents are sometimes difficult to follow (#201, US).

From a practical perspective, test takers further believed that the inclusion of different spoken varieties would increase the difficulty of the test preparation process, the actual tests themselves, and threaten test fairness, due to their near-exclusive exposure to and familiarity with the standard varieties (mentioned by 47.54% respondents with negative attitudes). This argument, again, supported previous research (Taylor and Geranpayeh, 2011).

We have been exposed to standard British or American English in class and after class. So other accents … are very difficult for us (#19, China).

If including non-native accents, It will be difficult to get a good score (#36, China).

Using standard English accents can make the test more fair. Also, we can be more specific in preparing for the test (#68, China).

Native things are difficult for me but non-native is much more difficult (#88, Korea).

Pronunciation of words from nonnative speakers might make the test and preparation harder since you do not know what general accent would be presented during the test (#176, US).

Next, many test takers believed that the target population of English communication only included L1 English speakers. That is, they mistakenly thought that the TLU domain of the IELTS test or English learning in general only included communicative situations with L1 English speakers. Moreover, they believed that the purpose of IELTS was to ensure successful communication with only L1 English speakers. This was mentioned by 11.45% respondents with negative attitudes.

No matter we study or travel abroad, we will meet mostly English native speakers. Therefore, the ability to understand standard accents should be tested (#40, China).

The country we are preparing to go use native English so I think the test speaking should be based on the native accents. If the test contains non-native English accents, it will blur the test taker's ideas about which is the proper accent (#84, Korea).

I feel no reason such various accents to be tested and learned by non-English native people. It doesn’t seem fit for the original purpose of the test (#91, Korea).

[T]he point of taking the test is to make sure that the test taker can easily cope with the accents of the native speakers (#233, US).

Lastly, many test takers still held the standard language ideology (Lippi-Green, 2012), supporting the inclusion of only standard accents—from a circular logic standpoint—because they were standard (mentioned by 16.39% respondents with negative attitudes).

[T]he main purpose of English learning is to communicate, so everyone must learn the most standard English, that is, English spoken by native speakers (#75, China).

Native English accent is authentic hence I prefer that (#212, US).

[I]t's better to understand what locals do (#122, Korea).

Test-Takers’ Perceptions by Geographic Location

The second research question asked whether test-takers’ perceptions differed across the three selected geographic locations (the US, China, and South Korea). Four Kruskal–Wallis H tests were performed to explore to what extent test-takers’ perceptions of Factor 1 (acceptance of nonstandard varieties), Factor 2 (acceptance of standard varieties), Factor 3 (overall acceptance of both standard and nonstandard spoken varieties), and the single item (preference for test takers’ own accents) differed according to geographic background. Table 2 gives the descriptive statistics of the participants’ perceptual ratings for each of the four factors.

First, the Kruskal–Wallis H test showed that there was no statistically significant difference in test-takers’ acceptance of nonstandard varieties (Factor 1) across geographic locations, χ²(2) = 0.96, p = .618, η² = .00, 95% CI [–.00, .01]. Second, there was no statistically significant difference in test-takers’ acceptance of standard varieties (Factor 2) across geographic locations, χ²(2) = 2.18, p = .336, η² = .00, 95% CI [–.00, .01]. Third, there was also no statistically significant difference in test takers’ overall acceptance of both standard and nonstandard spoken varieties (Factor 3) across geographic locations, χ²(2) = 0.19, p = .908, η² = .00, 95% CI [–.00, .01]. Finally, there was no statistically significant difference in test-takers’ preference of their own accents (stand-alone item) across geographic locations, χ²(2) = 0.96, p = .618, η² = .00, 95% CI [–.00, .01]. This means that geographic locations did not seem to influence test-takers’ perceptions of the inclusion of standard and nonstandard varieties in the IELTS listening test. In other words, IELTS test-takers’ perceptions of the inclusion of different English accent varieties in the ILETS listening test were somewhat indistinguishable.

Discussion

First, an EFA was run to assess the underlying factor structures of the items in the questionnaire. The results yielded a highly interpretable three-factor solution, each pertaining to test-takers’ acceptance of nonstandard-, standard-, and overall acceptance of different spoken varieties of English, respectively. That the factors were relatively clear-cut with high interpretability has certain implications for theory-building in relation to Kachru's model (1983). However, the model was criticized for not considering the effect of globalization and increased human mobility (Rose and Galloway, 2019). The test-takers’ data suggested that their perceptions of different accents did not conform to the three circles, but rather two: standard and nonstandard English accents. Moreover, in the case of Australian accents, test-takers’ perceptions could potentially blur the boundaries between standard and nonstandard varieties. Overall, this suggests that while the Kachru's model pertains to the description and categorization of English speakers across the world, the perceptions of the test takers in our sample did not agree with the categorizations of World Englishes scholars. However, it is possible that most of the test takers sampled were not aware of the concept of World Englishes, and were still very much embraced by the paradigm of standard English. The current IELTS test-takers certainly seemed to dwell in the binominal view of English: standard and nonstandard. This phenomenon might not be surprising, but might reflect the reality of the test takers’ current educational curricula, that is, solely focusing on standard English for testing purposes.

However, the fields of TESOL (Teachers of English to Speakers of Other Languages) and language assessment are well-informed that English is used as a global language for various international purposes (Canagarajah, 2006). English is the predominant L2 spoken across the world (Eberhard et al., 2022). Even though it may be difficult to stay away from the label of “standard language” completely (Isaacs and Rose, 2022) we believe that it is the responsibility of educators and language-testing practitioners to make efforts in shifting the focus from the inner circle standard of language assessment (Harding and McNamara, 2018) in this fast-changing era of globalization. Knowing that leaners’ perceptions are often influenced by teachers and language educators (Kang, 2009), such stakeholders may need to be trained or retrained for the current movement of World Englishes given their consequential educational outcomes.

Second, there was no geographic difference in IELTS test-takers’ perceptions of accent varieties. From the quantitative data presented above it is clear that, regardless of their geographic locations, test takers preferred the standard varieties of English used in high-stakes listening tests like IELTS. Although some of them commented on the potential positive aspects of the inclusion of diverse spoken varieties, many expressed concerns from four different angles. These concerns have implications for language testers regarding test development, language teaching, and learning in general.

The current findings about test-takers’ perceptions of the inclusion of different accent varieties in IELTS across different countries have several pedagogical and research implications. First, many test takers believed that different foreign accents could cause listeners’ difficulties and hinder comprehension, echoing the findings of numerous studies in the literature (e.g., Major et al., 2002). However, their comments did not seem to reflect an awareness that comprehensibility and intelligibility are co-constructed by both speakers and listeners (Rajadurai, 2007) and, as such, can be influenced by listener qualities such as familiarity with a particular accent variety (Kennedy and Trofimovich, 2008). To address this concern, perhaps it would be useful for language-testing policymakers and language teachers to formally promote the idea that comprehension is symbiotic between speakers and listeners. Given exposure, the effect of foreign accents on comprehensibility can be minimized.

Second, test takers were worried that the inclusion of different spoken varieties would threaten test fairness. This also reflects several theoretical arguments in the literature where different accent varieties may differentially benefit or disadvantage test takers depending on their linguistic backgrounds (see Authors1, Year). However, taking our participants’ comments into consideration, it was not clear to what extent they were aware of recent findings indicating that test-takers’ performance in listening tests incorporating accented English was not biased by certain linguistic backgrounds if the speakers were intelligible, despite the foreign accents (Authors1, Year). While these research findings would benefit from replication studies across different assessment contexts, test developers, policymakers, and language practitioners could perhaps inform potential test takers about such findings for awareness-raising purposes.

Third, many test takers seem to be confused about the TLU domain of the test. It would perhaps be useful if test developers make more explicit that the test is to assess a candidate's English proficiency interacting with English users including L1 English speakers and speakers of other varieties. In fact, many test-takers’ comments still reflected standard language ideology, that is, they had preference for the standard varieties of English over nonstandard varieties. Overall, the test-takers’ beliefs seemed somewhat constrained: they did not reflect the globalized English use in reality. It would, of course, be wrong to blame the test takers. Rather, it is possible that students’ typical exposure of the language through teaching, learning, and assessment is highly normative and institutionalized, reflecting the dominant powers of English users within certain countries (Rosa and Flores, 2021). For example, most, if not all, English textbooks include only prestigious English (Syrbe and Rose, 2018 for a survey in Germany). Many teachers are also reluctant to include nonstandard varieties in the classroom and associate linguistic diversity with being problematic (see Dooly, 2009; Siegel, 1999). Such views may have a direct or indirect influence on students’ or test-takers’ perspectives. Similarly, most English tests do not include nonstandard varieties, thus assisting in the establishment of the standard language ideology in the views of test takers.

These seem to be questions for the field of applied linguistics in general, not something that can be directly addressed by this paper. Future studies could explore the student- and teacher-related effects of awareness-raising activities or instructions on students’ or test-takers’ perspectives (see Galloway and Rose, 2014), the effect of a global-Englishes-informed syllabus on student learning, and perhaps an exploration of the “washback effect” (the impact of testing on curriculum design, teaching practices, and learning behaviors) of smaller-scale listening tests that incorporate different spoken varieties.

Limitations and Future Directions

This section discusses the methodological limitations of the current research and suggests future directions. First, the sample was limited to three geographic locations, meaning that the findings should be interpreted with caution before being generalized to other contexts. Future studies could explore test-takers’ perspectives across other geographic locations, including those in the outer circle.

Second, in the present study, we did not include audio-based stimuli to gauge test-takers’ preferences for specific accents, meaning that they did not have the chance to listen to the accent varieties before stating the extent to which they preferred them. This measure was limited, because perceptual measures with or without speech stimuli are two underlying factors of speech perception (Miao and Kang, 2023); the inclusion of questions without audio stimuli only probed one underlying factor. Therefore, future studies could include both measures (i.e., with and without audio stimuli) to investigate test-takers’ attitudes toward different spoken varieties in a listening test.

Conclusion

The present study sought to explore test-takers’ perspectives regarding the inclusion of different spoken varieties of English in IELTS. An EFA of the test-takers’ survey responses resulted in three factors: a) acceptance of nonstandard spoken varieties, b) acceptance of the standard variety, and c) overall acceptance of both standard and nonstandard spoken varieties. Overall, the test takers’ geographic backgrounds did not influence their perceptions. Moreover, test takers showed a strong preference for standard, prestigious varieties over nonstandard ones. Qualitative comments from the test takers revealed four main concerns over the inclusion of diverse accents, relating to a) the comprehensibility and intelligibility associated with foreign accents, b) test difficulty and fairness, c) conceptual understanding of the TLU domain of English, and d) holding a standard language ideology. Prior research has sought to address the first two test-takers’ concerns. Relevant policymakers, researchers, and teachers could help disseminate those research findings in the hope of alleviating such concerns. The last two concerns reflected the test-takers’ perceptions at a more macro level, beyond the scope of language testing. They pose interesting questions for the field of applied linguistics as to how to address issues related to language learning, teaching, and testing that are associated with power.

Footnotes

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the IELTS Partners: British Council, Cambridge Assessment English and IDP: IELTS Australia (the IELTS Joint-Funded Research Program).

ORCID iD

Yongzhi Miao

Correction (September 2024):

Article updated to correct the author byline to include author ‘Hyunkee Ahn’. Please see <> for more details.

References

Ahn

Moore

(2011) The relationship between students’ accent perception and accented voice instructions and its effect on students’ achievement in an interactive multimedia environment. Journal of Educational Multimedia and Hypermedia 20(4): 319–335.

Bachman

Palmer

(2010) Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford: Oxford University Press.

Bai

(2020) The relationship of test takers’ learning motivation, attitudes towards the actual test use and test performance of the College English Test in China. Language Testing in Asia 10(1): 1–18.

Bent

Bradlow

(2003) The interlanguage speech intelligibility benefit. The Journal of the Acoustical Society of America 114(3): 1600–1610.

Boonsuk

Ambele

McKinley

(2021) Developing awareness of Global Englishes: moving away from “native standards” for Thai university ELT. System 99: 102511.

Cambridge University Press & Assessment (2023) IELTS 18 Academic Student's Book with Answers with Audio with Resource Bank: Authentic Practice Tests. Cambridge: Cambridge University Press & Assessment.

Canagarajah

(2006) The place of World Englishes in composition: pluralization continued. College English Composition and Communication 57(4): 586–619.

Chan

Schmitt

DeShon

, et al. (1997) Reactions to cognitive ability tests: the relationships between race, test performance, face validity perceptions, and test-taking motivation. Journal of Applied Psychology 82(2): 300–310.

Chapelle

Lee

(2022) Conceptions of validity. In: Fulcher

Harding

(eds) The Routledge Handbook of Language Testing. Oxford: Routledge, pp. 59–75.

10.

DeVellis

(2017) Scale Development: Theory and Applications (3rd ed.). Thousand Oaks: Sage Publications.

11.

Dooly

(2009) How aware are they? Research into teachers’ attitudes about linguistic diversity. Language Awareness 14(2-3): 97–111.

12.

Eberhard

Simons

Fennig

(2022) Ethnologue: Languages of the World. Dallas, TX: SIL International.

13.

Elder

Davies

(2006) Assessing English as a Lingua Franca. Annual Review of Applied Linguistics 26: 282–304.

14.

Galloway

Rose

(2014) Using listening journals to raise awareness of Global Englishes in ELT. ELT Journal 68(4): 386–396.

15.

Hamid

(2014) World Englishes in international proficiency tests. World Englishes 33(2): 263–277.

16.

Hamp-Lyons

Davies

(2008) The English of English tests: bias revisited. World Englishes 27(1): 26–39.

17.

Harding

(2012) Accent, listening assessment and the potential for a shared-L1 advantage: a DIF perspective. Language Testing 29(2): 163–180.

18.

Harding

McNamara

(2018) Language assessment: the challenge of ELF. In: Jenkins

Baker

Dewey

(eds) Routledge Handbook of English as a Lingua Franca. Oxford: Routledge, pp. 510–523.

19.

IELTS (2022) IELTS test taker performance. Available at: https://www.ielts.org/en-us/for-researchers/test-statistics/test-taker-performance.

20.

IELTS (2023a) Available at: https://ielts.org/.

21.

IELTS (2023b) Available at: https://ielts-academic.com/2015/10/31/ielts-listening-english-accents/.

22.

IELTS (2023c) Available at: https://ielts.org/take-a-test/test-types/ielts-academic-test#Listening.

23.

Isaacs

Rose

(2022) Redressing the balance in the native speaker debate: assessment standards, standard language, and exposing double standards. TESOL Quarterly 56: 401–412.

24.

Kachru

(1983) The Indianization of English: The English Language in India. Delhi: Oxford University Press.

25.

Kang

(2009) ESL learners' attitudes toward pronunciation instruction and varieties of English. In Levis

LeVelle

(eds) Proceedings of the 1st Pronunciation in Second Language Learning and Teaching Conference, Iowa State University, Sept. 2009. Ames, IA: Iowa State University, pp. 105–118.

26.

Kang

Hirschi

Miao

, et al. (2022) Test-takers’ IELTS preparations and their attitudes towards IELTS practices and the use of technologies in the global pandemic. IELTS Research Reports Online Series, No. 2/22. British Council, Cambridge Assessment English and IDP: IELTS Australia. Available at https://www.ielts.org/teaching-and-research/research-reports

27.

Kang

Thomson

Moran

(2020) Which Features of Accent affect Understanding? Exploring the Intelligibility Threshold of Diverse Accent Varieties. Applied Linguistics 41(4): 453–480. DOI: https://doi.org/10.1093/applin/amy053.

28.

Kennedy

Trofimovich

(2008) Intelligibility, comprehensibility, and accentedness of L2 speech: the role of listener experience and semantic context. The Canadian Modern Language Review 64(3): 459–489.

29.

Lippi-Green

(2012) English with an Accent: Language, Ideology, and Discrimination in the United States (2nd ed.). Oxford: Routledge.

30.

Loewen

Plonsky

(2016) An A-Z of Applied Linguistics Research Methods. London: Palgrave.

31.

Major

Fitzmaurice

Bunta

, et al. (2002) The effects of nonnative accents on listening comprehension: implications for ESL assessment. TESOL Quarterly 36(2): 173–190.

32.

Meiselman

(2019) The language of context research. In: Meiselman

(ed) Context: The Effects of Environment on Product Design and Evaluation. Cambridge: Woodhead Publishing, pp.3–18.

33.

Messick

(1989) Validity. In: Linn

(ed) Educational Measurement. New York: American Council on Education/Macmillan, 13–103.

34.

Messick

(1996) Validity and washback in language testing. Language Testing 13(3): 241–256.

35.

Miao

Kang

(2023) An empirical approach to measuring accent familiarity: Phonological and correlational analyses. System 116: 103089. DOI: https://doi.org/10.1016/j.system.2023.103089.

36.

Nevo

(1985) Face validity revisited. Journal of Educational Measurement 22(4): 287–293.

37.

Ockey

French

(2016) From one to multiple accents on a test of L2 listening comprehension. Applied Linguistics 37(5): 693–715.

38.

Rajadurai

(2007) Intelligibility studies: a consideration of empirical and ideological issues. World Englishes 26(1): 87–98.

39.

Rosa

Flores

(2021) Decolonization, language, and race in applied linguistics and social justice. Applied Linguistics 42(6): 1162–1167.

40.

Rose

Galloway

(2019) Global Englishes for Language Teaching. Cambridge: Cambridge University Press.

41.

Shibata

(2021) L2 English speakers’ perception of their English accent: an investigation of European and Asian attitudes. English Language Teaching 14(12): 126–143.

42.

Siegel

(1999) Stigmatized and standardized varieties in the classroom: interference or separation? TESOL Quarterly 33(4): 701–728.

43.

Syrbe

Rose

(2018) An evaluation of the global orientation of English textbooks in Germany. Innovation in Language Learning and Teaching 12(2): 152–163.

44.

Taylor

Geranpayeh

(2011) Assessing listening for academic purposes: defining and operationalising the test construct. Journal of English for Academic Purposes 10(2): 89–101.