Modelling relationships among some test-taker characteristics and performance on EFL tests: an approach to construct validation

Abstract

Construct validation has seen two proposals recently: the use of construct representation and nomothetic span variables through structural modelling and the concept of population generalizability. This study investigated the influence of two major test-taker characteristics (TTCs), social milieu or cultural background and exposure or previous instruction, on test perform ance (TP) in tests of English as a foreign language (EFL) for two native language groups, the non-Indo-European (NIE) and the Indo-European (IE). Data from the Cambridge-TOEFL comparability study (Bachman et al., 1991; N = 1448) from eight sites in eight countries was used. The instruments were 1) a 45-item Likert scale background questionnaire which captured the test-taker characteristics; and 2) the First Certificate in English, administered by the University of Cambridge Local Examinations Syndicate, the TOEFL and the SPEAK, administered by the Educational Testing Service, Princeton, and the Test of English Writing. Modelling of the TTCs and the TP factors generally supported an equal influence factors model (where the factors have equal status) and an intervening factors type model (where the factors are not equal in status and one factor is an intervening factor) for both the NIE and IE groups.

References

Au, S.Y. 1988: A critical appraisal of Gardner's social-psychological theory of second-language (L2) learning. Language Learning 38, 75-100.

Bachman, L.F. 1988: Language testing-SLA research interfaces. Annual Review of applied Linguistics 9, 193-209.

— 1990: Fundamental considerations in language testing. Oxford: Oxford University Press.

Bachman, L.F. , Davidson, F. , Ryan, K. and Choi, I.-C. 1991: An investigation into the comparability of two tests of English as a foreign language: the Cambridge-TOEFL comparability study. Final report. Author.

Bachman, L.F. and Palmer, A.S. 1981: The construct validation of the FSI oral interview. Language Learning 31, 67-86.

— 1982: The construct validation of some components of communicative proficiency. TESOL Quarterly 16, 449-65.

Bechtold, H.P. 1959: Construct validity: a critique. American Psychologist 14, 619-29.

Bentler, P.M. 1978: The interdependence of theory, methodology, and empirical data: causal modeling as an approach to construct validation. In Kandel, D.B. , editor, Longitudinal research on drug use. Washington, DC: Hemisphere Publishing, 267-302.

— 1986: Structural modelling and Psychometrika: an historical perspective on growth and achievements. Psychometrika 51, 35-51.

10.

— 1989: EQS: structural equations program manual. Los Angeles, CA: BMDP Statistical Software.

11.

— 1990: Comparative fit indexes in structural models. Psychological Bulletin 107, 238-46.

12.

Berk, R. , editor, 1982: Handbook of methods for detecting test bias. Baltimore, MD: Johns Hopkins University Press.

13.

Boldt, R.F. 1988: Latent structure analysis of the TOEFL. TOEFL Research Report 28. Princeton, NJ: Educational Testing Service.

14.

Boomsma, A. 1987: The robustness of maximum likelihood estimation in structural equation models. In Cuttance, P. and Ecob, R. , editors, Structural modelling by example. Cambridge: Cambridge University Press, 160-88.

15.

Briere, E.J. 1968: Testing ESL among Navajo children. In Upshur, J. and Fata, J., editors, Problems in foreign language testing. Language Learning 3, 11-21.

16.

— 1973: Cross cultural bias in language testing. In Oller, J. and Richards, J. editors, Focus on the learner: pragmatic perspectives for the language teacher. Rowley, MA: Newbury House, 214-27.

17.

Briere, E.J. and Brown, R.H. 1971: Norming tests of ESL among Amerindian children. TESOL Quarterly 5, 327-4.

18.

Campbell, D.T. and Fiske, D.W. 1959: Convergent and discriminant validity in the multitrait-multimethod matrix. Psychological Bulletin 56, 81-105.

19.

Carroll, J.B. 1983: Psychometric theory and language testing. In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House , 80-107.

20.

Chapelle, C. 1988: Field independence: a source of language test variance? Language Testing 5, 62-68.

21.

Chen, Z. and Henning, G. 1985: Linguistic and cultural bias in language proficiency tests . Language Testing 2, 155-63.

22.

Chou, C.-P. , Bentler, P.M. and Satorra, A. 1989: Scaled test statistics and robust standard errors for non-normal data in covariance structure analysis: a Monte Carlo study. Paper presented at the AERA, San Francisco, CA.

23.

Churchman, C.W. 1971: The design of inquiring systems: basic concepts of systems and organization. New York: Basic Books.

24.

Clement, R. and Krudenier, B.G. 1985: Aptitude, attitude and motivation in second language proficiency: a test of Clement's model. Journal of Language and Social Psychology 4, 21-37.

25.

Cooper, L.A. and Regan, D.T. 1982: Intelligence, attention, and perception. In Sternberg, R.J. , editor, Handbook of human intelligence. Cambridge: Cambridge University Press, 123-69.

26.

Cronbach, LJ 1989: Construct validation after thirty years. In Linn, R.L. , editor, Intelligence: measurement, theory, and public policy. Urbana, IL: University of Illinois Press, 147-71.

27.

Cronbach, L.J. and Meehl, P.E. 1955: Construct validity in psychological tests. Psychological Bulletin 52, 281-302.

28.

Cuttance, P. 1987: Issues and problems in the applications of structural equation models. In Cuttance, P. and Ecob, R. , editors, Structural modelling by example. Cambridge: Cambridge University Press, 241-79.

29.

Ecob, R. 1987: Applications of structural equation modelling to longitudinal educational data. In Cuttance, P. and Ecob, R. , editors, Structural modelling by example. Cambridge: Cambridge University Press, 138-59.

30.

Ecob, R. and Cuttance, P. 1987: An overview of structural equation modelling. In Cuttance P. and Ecob, R. , editors, Structural modelling by example. Cambridge: Cambridge University Press, 9-23.

31.

Embretson, S.E. 1983: Construct validity: construct representation versus nomothetic span. Psychological Bulletin 93, 179-97.

32.

— 1985: Multicomponent latent trait models for test design. In Embretson, S. E. , editor, Test design: developments in psychology and psychometrics. Orlando, FL: Academic Press, 195-218.

33.

Gardner, R. 1985: Social psychology and second language learning: the role of attitudes and motivation. London: Edward Arnold.

34.

— 1988: The socio-educational model of second language learning: assumptions, findings and issues. Language Learning 38, 101-26.

35.

Gardner, R. , Lalonde, R.N. , Moorcraft, R. and Evers, F.T. 1987: Second language attrition: the role of motivation and use. Journal of Language and Social Psychology 6, 1-47.

36.

Gardner, R. , Lalonde, R.N. and Pierson, R. 1983: The socioeducational model of second language acquisition: an investigation using LISREL causal modelling. Journal of Language and Social Psychology, 2, 1-15.

37.

Guiora, A.Z. , Paluszny, M. , Beit-Hallahmi, B. , Catford, J.C. and Cooley, R.E. 1975: Language and person: studies in language behaviour. Language Learning 25, 43-61.

38.

Hale, G.A. , Rock, D.A. and Jirele, T. 1989: Confirmatory factor analysis of the Test of English as a Foreign Language. TOEFL Research Report 32. Princeton, NJ: Educational Testing Service.

39.

Hamayan, E.V. and Tucker, G.R. 1980: Language input in the bilingual classroom and its relationship to second language achievement. TESOL Quarterly 14, 453-68.

40.

Hansen, L. and Stansfield, C. 1984: Field dependence-independence and language testing: evidence from six Pacific-Island cultures. TESOL Quarterly 18, 311-24.

41.

Hill, P.W. 1987: Modelling the hierarchical structure of learning. In Cuttance, P. and Ecob, R. , editors, Structural modelling by example. Cambridge: Cambridge University Press, 65-85.

42.

Hinofotis, F. 1983: The structure of oral communication in an educational environment: a comparison of factor analytical rotation procedures. In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House, 170-87.

43.

Holland, P.W. and Wainer, H. , editors, 1993: Differential item functioning . Hillsdale, NJ: Lawrence Erlbaum Associates.

44.

Joreskog, K.G. 1978: Structural analysis of covariance and correlational matrices . Psychometrika 43, 443-77.

45.

Krashen, S.D. 1982: Principles and practice in second language acquisition . Oxford: Pergamon Press.

46.

Kunnan, A.J. 1990: Differential item functioning and native language and gender groups: the case of an ESL placement examination. TESOL Quarterly 24, 741-46.

47.

Kyllonen, P.C. , Lohman, D.F. and Woltz, D. 1984: Componential modelling of alternative strategies for performing spatial tasks. Journal of Educational Psychology 76, 1325-45.

48.

Laosa, L.M. 1991: The cultural context of construct validity and the ethics of generalizability. Research Report 91-51. Princeton, NJ: Educational Testing Service.

49.

Loevinger, J. 1957: Objective tests as instruments of psychological theory . Psychological Reports 3, 635-94.

50.

Messick, S. 1989: Validity. In Linn, R.L. , editor, Educational measurement (3rd edn. New York: American Council on Education, 13-103.

51.

Muthen, B. 1988: Some uses of structural equation modeling in validity studies: extending IRT to external variables. In Wainer, H. and Braun, H. , editors, Test validity. Hillside, NJ: Lawrence Erlbaum Associates, 213-38.

52.

— 1989: Latent variable modelling in heterogeneous populations . Psychometrika 54, 557-85.

53.

Nelson, F.H. , Lomax, R.G. and Perlman, R. 1984: A structural equation model of second language acquisition of adult learners. Journal of Experimental Education 53, 29-39.

54.

Oller, J.W. 1982: Gardner on affect: a reply to Gardner. Language Learning 32,183-89.

55.

— 1983: Evidence for a general language proficiency factor: an expectancy grammar. In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House, 3-10.

56.

Oller, J.W. and Hinofotis, F. 1980: Two mutually exclusive hypotheses about second language ability: indivisible or partially divisible competence. In Oller, J. and Perkins, K. , editors, Research in language testing. Rowley, MA: Newbury House, 13-23.

57.

Oltman, P.K. , Stricker, L.J. and Barrows, T. 1988: Native language, English proficiency, and the structure of the TOEFL. TOEFL Research Report 27 Princeton, NJ: Educational Testing Service.

58.

Pena-Taveras, M.S. and Cambel, A.B. 1989: Nonlinear, stochastic model for energy investment in manufacturing. Energy - The International Journal 14, 421-33.

59.

Purcell, E.T. 1983: Models of pronunciation accuracy. In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House , 133-53.

60.

Ryan, K. and Bachman, L.F. 1992: Differential item functioning on two tests of EFL proficiency . Language Testing 9, 12-29.

61.

Sang, F. , Schmitz, B. , Vollmer, H.J. , Baumert, J. and Roeder, P.M. 1986: Models of second language competence: a structural equation approach. Language Testing 3, 54-79.

62.

Sasaki, M. 1991: Relationships among second language proficiency, foreign language aptitude, and intelligence: a structural equation modeling approach. Unpublished PhD dissertation, University of California , Los Angeles.

63.

Satorra, A. and Bentler, P.M. 1988a: Scaling corrections for statistics in covariance structure analysis. UCLA Statistics Series 2. Los Angeles, CA: University of California.

64.

— 1988b: Scaling corrections for chi-square statistics in covariance structure analysis. Proceedings of the American Statistical Association 308-13.

65.

Skehan, P. 1989: Individual differences in second language learning. London: Edward Arnold.

66.

Snow, C.E. and Hoefnagle-Hohle, M. 1978: Age differences in second language acquisition. In Hatch, E. , editor, Second language acquisition: a book of readings. Rowley, MA: Newbury House, 333-44.

67.

Spolsky, B. 1989: Conditions for second language learning. Oxford: Oxford University Press.

68.

Stansfield, C. and Hansen, J. 1983: Field dependence-independence as a variable in second language cloze test performance. TESOL Quarterly 17, 29-38.

69.

Swinton, S.S. and Powers, D.E. 1980: Factor analysis of the TOEFL. TOEFL Research Report 6 Princeton, NJ: Educational Testing Service.

70.

Turner, C.E. 1989: The underlying factor structure of L2 cloze test performance in Francophone, university-level students: causal modelling as an approach to construct validation. Language Testing 6, 172-97.

71.

Upshur, J. 1983: Measurement of individual differences and explanation in the language sciences. Language Learning 33, 99-140.

72.

Upshur, J. and Homburg, T.J. 1983: Some relations among tests at successive ability levels . In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House, 188-202.

73.

Vollmer, H.J. 1983: The structure of foreign language proficiency. In Hughes A. , and Porter, D. , editors, Current developments in language testing . London: Academic Press, 3-29.

74.

Vollmer, H.J. and Sang, F. 1983: Competing hypotheses about second language ability: a plea for caution. In Oller, J. , editor, Issues in language testing research. Rowley, MA: Newbury House.

75.

West, B.J. and Salk, J. 1987: Complexity, organizations, and uncertainty. European Journal of Operational Research 30, 117-28.

76.

Wheaton, B. , Muthen, B. , Alwin, D.F. and Summers, G.F. 1977: Assessing reliability and stability in panel models. In Heise, D.R. , editor, Sociological methodology 1977. San Francisco, CA: Jossey Bass, 84-136.

77.

Zeidner, M. 1986: Are English language aptitude tests biased towards culturally different minority groups? Some Israeli findings. Language Testing 3, 80-98.