Item scores that do not fit an assumed item response theory model may cause the latent trait value to be inaccurately estimated. For a computerized adaptive test (CAT) using dichotomous items, several person-fit statistics for detecting mis.tting item score patterns have been proposed. Both for paper-and-pencil (P&P) tests and CATs, detection ofperson mis.t with polytomous items is hardly explored. In this study, the nominal and empirical null distributions ofthe standardized log-likelihood statistic for polytomous items are compared both for P&P tests and CATs. Results showed that the empirical distribution of this statistic differed from the assumed standard normal distribution for both P&P tests and CATs. Second, a new person-fit statistic based on the cumulative sum (CUSUM) procedure from statistical process control was proposed. By means ofsimulated data, critical values were determined that can be used to classify a pattern as fitting or misfitting. The effectiveness of the CUSUM to detect simulees with item preknowledge was investigated. Detection rates using the CUSUM were high for realistic numbers ofdisclosed items.
References
1.
Akkermans, L.M.W. (1998). Studies on statistical methods for polytomously scored test items. Unpublished doctoral thesis, University of Twente, Enschede.
2.
Chang, H. H. , & Mazzeo, J. (1994). The unique correspondence ofthe item response function and the item category response functions of polytomously scored item response models. Psychometrika, 59, 391-404.
3.
Davey, T. , & Nering, M. (1998). Controlling item exposure and maintaining item security. Paper presented at Symposium on Computer Based Testing: Building for Future Assessments, Philadelphia, PA.
4.
Dodd, B. G. , De Ayala, R. J., & Koch, W. R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19, 5-22.
5.
Dodd, B. G. , Koch, W. R., & De Ayala, R. J. (1993). Computerized adaptive testing using the partial credit model: Effects of item pool characteristics and different stopping rules. Educational and Psychological Measurement, 53, 61-77.
6.
Drasgow, F. , Levine, M. V., & Williams, E. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
7.
Hambleton, R. K. , & Swaminathan, H. (1985). Item response theory: Principles and applications. Boston: Kluwer-Nijhoff.
8.
Hemker, B. T. (1996). Unidimensional IRT models for polytomous items, with results for Mokken scale analysis. Unpublished doctoral thesis, University of Utrecht, Utrecht.
9.
Hoijtink, H. (1986). Detecting aberrant response patterns in the unidimensional scaling model of Rasch (Heymans Bulletin HB-86-792-SW). University of Groningen, Groningen, The Netherlands.
10.
Jansen, P.G.W. , & Roskam, E. E. (1986). Latent trait models and the dichotomization ofgraded responses. Psychometrika, 51, 69-91.
11.
Koch, W. R. , & Dodd, B. G. (1989). An investigation ofprocedures for computerized adaptive testing using partial credit scoring. Applied Measurement in Education, 2, 335-357.
12.
Levine, M. V. , & Rubin, D. B. (1979). Measuring the appropriateness ofmultiple-choice test scores. Journal of Educational Statistics, 4, 269-290.
13.
Lindgren, B. W. (1993). Statistical theory (4th ed.). New York: Chapman and Hall.
14.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
15.
McLeod, L. D. (1998). Alternative methods for the detection of item preknowledge in computerized adaptive testing. Unpublished doctoral dissertation, University of North Carolina, Chapel Hill.
16.
Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18, 311-314.
17.
Meijer, R. R. , Molenaar, I. W., & Sijtsma, K. (1994). Influence oftest and person characteristics on non-parametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
18.
Meijer, R. R. , & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
19.
Meijer, R. R. , & van Krimpen-Stoop, E.M.L.A. (2001). Person fit across subgroups: An achievement testing example. In A. Boomsma, M.A.J. van Duijn, & T.A.B. Snijders (Eds.), Essays on item response theory. New York: Springer-Verlag.
20.
Montgomery, D. C. (1997). Introduction to statistical quality control (3rd ed.). New York: John Wiley.
21.
Nering, M. L. (1997). The distribution ofindexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21, 115-127.
22.
Page, E. S. (1954). Continuous inspection schemes. Biometrika, 41, 100-115.
23.
Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests. Copenhagen: Danish Institute for Educational Research. Expanded edition (1980), with foreword and afterword by B. D. Wright. Chicago: University of Chicago Press.
24.
Reise, S. P. , & Due, A. M. (1991). The influence of test characteristics on the detection ofaberrant response patterns. Applied Psychological Measurement, 15, 217-226.
25.
Rogers, H. J. , & Hattie, J. A. (1987). A Monte Carlo investigation ofseveral person and item fit statistics for item response models. Applied Psychological Measurement, 11, 47-57.
26.
Roskam, E. E. , & Jansen, P.G.W. (1989). Conditions for Rasch-dichotomization ability of the unidimensional polytomous Rasch model. Psychometrika, 54, 317-332.
27.
Samejima, F. (1969). Estimation oflatent ability using response models ofgraded scores. Psychometrika Monograph, Supplement No. 17.
28.
Snijders, T. (2001). Asymptotic null distribution of person-fit statistics with estimated person parameter. Psychometrika, 66, 331-342.
29.
Tatsuoka, K. K. (1984). Caution indices based on item response theory. Psychometrika, 49, 95-95.
30.
van den Brink, W. P. (1977). Het verken-effect [The scouting effect]. Tijdschrift voor On derwijs research, 2, 253-261.
31.
van der Linden, W. J. , & Hambleton, R. K. (1997). Handbook of modern item response theory. New York: Sprin.
32.
van Krimpen-Stoop, E.M.L.A. , & Meijer, R. R. (1999). The null distribution ofperson-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345.
33.
van Krimpen-Stoop, E.M.L.A. , & Meijer, R. R. (2000). Detecting person misfit in adaptive testing using statistical process control techniquesJ. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 201-219). Boston, MA: Kluwer-Nijhoff.
34.
van Krimpen-Stoop, E.M.L.A. , & Meijer, R. R. (2001). CUSUM-based person-fit statistics for adaptive testing, Journal of Educational and Behavioral Statistics, 26, 199-217.
35.
Wright, B. D. , & Masters, G. N. (1982). Rating scale analysis. Chicago: Mesa.
36.
Wright, B. D. , & Stone, M. H. (1979). Best test design. Chicago: Mesa.
37.
Zickar, M. J. , & Drasgow, F. (1996). Detecting faking on a personality instrument using appropriateness measurement. Applied Psychological Measurement, 20, 71-87.
