This study investigated factors that could affect inter-examiner reliability in the pronunciation assessment component of speaking tests. We hypothesized that the rating of pronunciation is susceptible to variation in assessment due to the amount of exposure examiners have to nonnative English accents. An inter-rater variability analysis was conducted on the English pronunciation ratings of three test candidate interlanguages: Chinese, Korean, and Indian English. Pronunciation was rated by 99 International English Language Testing System (IELTS) examiners across five geographically dispersed test centres. The examiners had either prolonged exposure, or no, or little exposure to the interlanguage of the candidates. A significant proportion of examiners rated pronunciation higher when they had prolonged exposure, and lower when they had no, or little, exposure to the candidates’ interlanguage. The location of the test centre also had a significant effect on the pronunciation rating, independent of the familiarity variable, with a significant proportion of NNS raters scoring candidates from their home country higher than those who were not. It is recommended that interlanguage phonology familiarity should be considered in the design of speaking tests and rater training and that research is required into test centre bias and the phonological judgements and awareness of OPI raters.
Anderson-Hsieh, J., & Koehler, K. ( 1988). The effect of foreign accent and speaking rate on native speaker comprehension. Language Learning, 38, 561-613.
2.
Barnwell, D. ( 1989). ‘Naïve’ native speakers and judgments of oral proficiency in Spanish. Language Testing, 6, 152-163.
3.
Best, C.T. ( 1995). A direct realist view of cross-language perception. In W. Strange (Ed.), Speech perception and linguistic experience: Theoretical and methodological issues in cross-language speech research. Timonium, MD: York Press.
4.
Best, C.T. ( 2001). Discrimination of non-native consonant contrasts varying in perceptual assimilation to the listener’s native phonological system . Journal of the Acoustical Society of America, 109, 775-794.
5.
Bilbow, G.T. ( 1989). Towards an understanding of overseas students’ difficulties in lectures: A phenomenographic approach. Journal of Further and Higher Education, 13, 85-89.
6.
Bohn, O-S. ( 1995). Cross-language speech perception in adults: First language transfer doesn’t tell it all. In W. Strange (Ed.), Speech perception and linguistic experience: Theoretical and methodological issues in cross-language speech research. Timonium, MD: York Press.
7.
Brennan, E., & Brennan, J.S. ( 1981). Measurements of accent and attitude toward Mexican-American speech. Journal of Psycholinguistic Research, 10, 487-501.
8.
Brown, A. ( 1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing , 12, 1-15.
9.
Brown, A. ( 2006). An examination of the rating process in the revised IELTS speaking test, IELTS Research Reports, 6, 41-70.
10.
Brown, K. ( 1968). Intelligibility. In A. Davies (Ed.), Language testing symposium (pp. 180-191). Oxford: Oxford University Press.
11.
Cargile, A.C. ( 1997). Attitudes toward Chinese-accented speech: An investigation in two contexts. Journal of Language and Social Psychology , 16, 434-443.
12.
Chalhoub-Deville, M. ( 1995). Deriving oral assessment scales across different tests and rater groups. Language Testing, 12, 16-33.
13.
Chalhoub-Deville, M., & Wigglesworth, G. (2005). Rater judgment and English language speaking proficiency. World Englishes, 24, 383-391.
14.
Eisentein, M.R., & Berkowitz, D. ( 1981). The effect of phonological variation on adult learner comprehension. Studies in Second Language Acquisition, 4, 75-80.
15.
Ekong, P. ( 1982). On the use of an indigenous model for teaching English in Nigeria. World Language English, 1, 87-92.
16.
Fayer, J.M., & Krasinski, E. ( 1987). Native and nonnative judgments of intelligibility and irritation. Language Learning, 37, 313-326.
17.
Flege, J.E. ( 1987). The production of ‘new’ versus ‘similar’ phones in a foreign language: Evidence for the effect of equivalence classification , Journal of Phonetics, 15, 47-65.
18.
Flege, J.E. ( 1995). Second language learning: Theory, findings and problems . In W. Strange (Ed.), Speech perception and linguistic experience: Theoretical and methodological issues in cross-language speech research. Timonium, MD : York Press.
19.
Flege, J.E., Schirru, C., & MacKay, I.R.A. (2003). Interaction between the native and second language phonetic systems. Speech Communication , 40, 467-491.
20.
Flowerdew, J. ( 1994). Research of relevance to second language lecture comprehension - an overview. In J. Flowerdew (Ed.), Academic listening (pp. 7-29). New York: Cambridge University Press.
21.
Fowler, C.A. ( 1986). An event approach to the study of speech perception from a direct-realist perspective. Journal of Phonetics, 14, 3-28.
22.
Galloway, V.B. ( 1980). Perceptions of the communicative efforts of American students of Spanish. Modern Language Journal, 64, 428-433.
23.
Grieser, D., & Kuhl, P.K. ( 1989). Categorization of speech by infants: Support for speech-sound prototypes. Developmental Psychology, 25, 577-588.
24.
Guenther, F.H., & Bohland, J.W. ( 2002). Learning sound categories: A neural model and supporting experiments, Acoustical Science and Technology, 23(4), 213-221.
25.
Guenther, F.H., Nieto-Castanon, A., Ghosh, S.S., & Tourville, J.A. (2004). Representation of sound categories in auditory cortical maps, Journal of Speech, Language, and Hearing Research, 47(1), 46-57.
26.
Hadden, B.L. ( 1991). Teacher and nonteacher perceptions of second-language communication. Language Learning, 41, 1-24.
27.
Johnson, K. ( 1996). Resonance in an exemplar-based lexicon: The emergence of social identity and phonology. Journal of Phonetics, 34, 485-499.
28.
Johnson, K. ( 2005). Speaker normalization in speech perception. In D. B. Pisoni & R. Remez (Eds.), The handbook of speech perception. Oxford: Blackwell.
29.
Kim, Y-H. ( 2009). An investigation into native and non-native teachers’ judgments of oral English performance: A mixed methods approach, Language Testing, 26(2), 187-217.
30.
Kuhl, P.K. ( 1991). Human adults and human infants show a ‘perceptual magnet effect’ for the prototypes of speech categories, monkeys do not . Perception & Psychophysics, 50, 93-107.
31.
Kuhl, P.K., & Iverson, P. ( 1995). Age-related changes in cross-language speech perception: Standing at the crossroads. In W. Strange (Ed.), Speech perception and linguistic experience: Theoretical and methodological issues in cross-language speech research. Timonium, MD: York Press.
32.
Liberman, A.M., Harris, K.S., Hoffman, H.S., & Griffith, B.C. (1957). The discrimination of speech sounds within and across phoneme boundaries. Journal of Experimental Psychology54(5), 358-368.
33.
Linacre, S. ( 2007). Australian social trends 2007: International students in Australia, Australian Bureau of Statistics, Catalogue no. 4102.0. Retrieved August 30, 2008, from http://www.ausstats.abs.gov.au.
34.
Lindemann, S. ( 2005). Who speaks ‘broken English’? US undergraduates’ perceptions of nonnative English. International Journal of Applied Linguistics, 15(2), 187-212.
35.
Mackey, L.S., & Finn, P. ( 1997). Effect of speech accent on speech naturalness ratings: A systematic replication of Martin, Haroldson, and Triden (1984). Journal of Speech, Language & Hearing Research, 40, 349-361.
36.
Major, R.C., Fitzmaurice, S.F.Bunta, F., & Balasubramania, C. (2002). The effects of nonnative accents on listening comprehension: Implications for ESL assessment. TESOL Quarterly, 36, 173-190.
37.
Massaro, D.W. ( 1989). Testing between the TRACE Model and the Fuzzy Logical Model of Speech perception. Cognitive Psychology, 21(3), 398-421.
38.
Miller, J.L. ( 1994). On the internal structure of phonetic categories: A progress report. Cognition, 50, 271-285.
39.
Nesdale, D., & Rooney, R. ( 1996). Evaluations and stereotyping of accented speakers by pre-adolescent children. Journal of Language & Social Psychology, 15, 133-155.
40.
New Zealand Ministry of Education (2008). International student enrolments in New Zealand 2001- 2008. Retrieved September 20, 2008, from http://www.educationcounts.govt.nz/__data/assets/pdf_file/0015/24711/International_Student_Enrolments_in_New_Zealand_2001_-_2007.pdf .
41.
Nittrouer, S., Manning, C., & Meyer, G. ( 1993). The perceptual weighting of acoustic cues changes with linguistic experience. Journal of theAcoustical Society of America, 94, 1865.
42.
Papajohn, D. ( 2002). Concept mapping for rater training. TESOL Quarterly, 36(2).
43.
Pierrehumbert, J.B. ( 2001). Exemplar dynamics: Word frequency, lenition, and contrast . In J. Bybee & P. Hopper (Eds.), Frequency effects and the emergence of lexical structure (pp. 137-157). Amsterdam: John Benjamins.
44.
Ramanathan, H. ( 2008). Testing of English in India: A developing concept, Language Testing, 25(1), 111-126.
Rochet, B.L. ( 1995). Perception and production of second-language speech sounds by adults. In W. Strange (Ed.), Speech perception and linguistic experience: Theoretical and methodological issues in cross-language speech research. Timonium, MD : York Press.
47.
Rubin, D.L. ( 1992). Non-language factors affecting undergraduates’ judgements of non-native English-speaking teaching assistants. Research in Higher Education, 33(4).
48.
Rubin, D.L., & Smith, K.A. ( 1990). Effects of accent, ethnicity, and lecture topic on undergraduates’ perceptions of nonnative English-speaking teaching assistants. International Journal of Intercultural Relations, 14, 337-353.
49.
Stevens, K.N. ( 2002). Toward a model of lexical access based on acoustic landmarks and distinctive features. Journal of the Acoustical Society of America, 111(4), 1872-1891.
50.
Wilcox, G.K. ( 1978). The effect of accent on listening comprehension: A Singapore study. English Language Teaching Journal, 32, 118-127.
51.
Xi, X., & Mollaun, P. ( 2009). How do raters from India perform in scoring the TOEFL iBT speaking section and what kind of training helps? TOEFL iBT Research Report . Princeton, NJ: English Testing Service.
52.
Zhang, Y., & Elder, C. ( 2011). Judgments of oral proficiency by non-native and native English speaking teacher raters: Competing or complementary constructs?Language Testing, 28(1).
53.
Zhang, Y., Kuhl, P.K., Imada, T., Kotani, M., & Tohkurad, Y. ( 2005). Effects of language experience: Neural commitment to language-specific auditory patterns. NeuroImage, 26, 703-720.
