This study investigated the convergent validity of expert-system scores for four mathematical constructed-response item formats. A five-factor model comprised of four constructed-response for mat factors and a Graduate Record Examination (GRE) General Test quantitative factor was posed. Confirmatory factor analysis was used to test the fit of this model and to compare it with several alter natives. The five-factor model fit well, although a solution comprised of two highly correlated dimensions_GRE-quantitative and constructed- response—represented the data almost as well. These results extend the meaning of the expert system's constructed-response scores by relating them to a well-established quantitative measure and by indicating that they signify the same underlying proficiency across item formats.
Bennett, R.E. (in press). Intelligent assessment: Toward an integration of constructed-response testing, artificial intelligence, and model-based measurement In N. Frederiksen, R. J. Mislevy, and I. Bejar (Eds.), Test theory for a new generation of tests. Hillsdale NJ : Erlbaum.
2.
Bennett, R.E., Gong, B., Kershaw, R.C., Rock, D.A., Soloway, E., & Macalalad, A. (1990). Assessment of an expert system's ability to grade and diagnose automatically student's constructed responses to computer science problems. In R. O. Freedle (Ed.), Artificial intelligence and the future of testing (pp. 293-320). Hillsdale NJ: Erlbaum .
3.
Bennett, R.E., Rock, D.A., Braun, H.I., Frye, D., Spohrer, J.C., & Soloway, E. (1990). The relationship of expert-system scored constrained free-response items to multiple-choice and open-ended items. Applied Psychological Measurement, 14, 151-162.
4.
Bennett, R.E., Rock, D.A., & Wang, M. (1991). Equivalence of free-response and multiple-choice itemsJournal of Educational Measurement , 28, 77-92.
5.
Bennett, R.E., Sebrechts, M.M., & Rock, D.A. (1991). The convergent validity of expert system scores for complex constructed-response quantitative items (Research Rep. No. RR-91-12)Princeton NJ: Educational Testing Service.
6.
Bennett, R.E., Ward, W.C., Rock, D.A., & LaHart, C. (1990). Toward a framework for constructed-response items (Research Rep. No. RR-90-7)Princeton NJ: Educational Testing Service.
7.
Bentler, P.M. (1989). EQS structural equations program manualLos Angeles CA: BMDP Statistical Software, Inc.
8.
Braun, H.I., Bennett, R.E., Frye, D., & Soloway, E. (1990). Scoring constructed responses using expert systemsJournal of Educational Measurement , 27, 93-108.
9.
Bridgeman, B. (in press). A comparison of open-ended and multiple-choice question formats for the quantitative section of the Graduate Record ExaminationPrinceton NJ: Educational Testing Service .
10.
Cudeck, R., & Browne, M.W. (1983). Cross validation of covariance structuresMultivariate Behavioral Research, 18, 147-167.
11.
Educational Testing Service. (1989a). GRE information bulletin. Princeton NJ : Author.
12.
Educational Testing Service. (1989b). 1989-90 GRE guide to the use of the Graduate Record Examinations Program . Princeton NJ: Author.
13.
Educational Testing Service. (1989c). ETS research plan designed to create a new generation of Graduate Record Examinations. ETS Examiner, 18 (26; February 23), 1.
14.
Fiske, E.B. (1990, January 31). But is the child learning ? Schools trying new testsThe New York Times, pp. A1, B6.
15.
Frederiksen, J.R., & Collins, A. (1989). A systems approach to educational testing. Educational Researcher, 18 (9), 27-32.
Hinsley, D.A., Hayes, J.R., & Simon, H.A. (1977). From words to equations: Meaning and representation in algebra word problems In P. Carpenter & M. A. Just, (Eds.), Cognitive processes in comprehension (pp. 89-106). Hillsdale NJ : Erlbaum.
18.
Jöreskog, K., & Sörbom, D. (1986). PRELIS: A program for multivariate data screening and data summarizationMooresville IN: Scientific Software, Inc.
19.
Jöreskog, K., & Sörbom, D. (1988). LISREL 7: A guide to the program and applicationsChicago IL: SPSS, Inc.
20.
Little, R.J.A., & Rubin, D.B. (1987). Statistical analysis with missing data . New York: Wiley.
Marsh, H.W., Balla, J.R., & McDonald, R.P. (1988). Goodness-of-fit indexes in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 103 , 391-410.
23.
Marsh, H.W., & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept : First and higher order factor models and their invariance across groups. Psychological Bulletin , 97, 562-582.
24.
Mayer, R.E. (1981). Frequency norms and structural analysis of algebra story problems into families, categories, and templates. Instructional Science, 10 , 135-175.
25.
Newell, A., & Simon, H.A. (1972). Human problem solving. Englewood Cliffs NJ: Prentice-Hall.
26.
Nickerson, R.S. (1989). New directions in educational assessment. Educational Researcher, 18 (9), 3-7.
27.
Rock, D.A., Bennett, R.E., & Jirele, T. (1988). The factor structure of the Graduate Record Examinations General Test in handicapped and nonhandicapped groupsJournal of Applied Psychology, 73, 383-392.
28.
Rock, D.A., Werts, C., & Grandy, J. (1982). Construct validity of the GRE Aptitude Test across populations: An empirical confirmatory study (Research Rep. No. RR-81-57)Princeton NJ: Educational Testing Service.
29.
Sebrechts, M.M., Bennett, R.E., & Rock, D.A. (1991). Machine-scorable complex constructed-response quantitative items: Agreement between expert-system and human raters' scores (Research Rep. No. RR-91-11)Princeton NJ: Educational Testing Service.
30.
Sebrechts, M.M., LaClaire, L., Schooler, L.J., & Soloway, E. (1986). Toward generalized intention-based diagnosis: GIDE In R. C. Ryan (Ed.), Proceedings of the 7th National Educational Computing Conference (pp. 237-242). Eugene OR: International Council on Computers in Education.
31.
Sebrechts, M.M. , & Schooler, L.J. (1987). Diagnosing errors in statistical problem solving: Associative problem recognition and plan-based error detection In E. Hunt (Ed.), Proceedings of the Ninth Annual Cognitive Science Meeting (pp. 691-703). Hillsdale NJ: Erlbaum.
32.
Stricker, L.J., & Rock, D.A. (1987). Factor structure of the GRE General Test in young and middle adulthoodDevelopmental Psychology , 23, 526-536.
33.
Swinton, S.S., & Powers, D.E. (1980). A factor anulytic study of the restructured GRE Aptitude Test (GREB Report No. 77-6P)Princeton NJ: Educational Testing Service.
34.
Traub, R.E. (in press). On the equivalence of the traits assessed by multiple-choice and constructed-response tests In R. E. Bennett & W. C. Ward, (Eds.), Construction versus choice in cognitive measurement . Hillsdale NJ: Erlbaum.
35.
Traub, R.E., & Fisher, C.W. (1977). On the equivalence of constructed-response and multiple-choice tests. Applied Psychological Measurement , 1, 355-369.
36.
Tucker, L.R., & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysisPsychometrika, 38, 1-10.
37.
van den Bergh, H. (1990). On the construct validity of multiple-choice items for reading comprehensionApplied Psychological Measurement , 14, 1-12.
38.
Ward, W.C. (1982). A comparison of free-response and multiple-choice forms of verbal aptitude testsApplied Psychological Measurement , 6, 1-11.
