This study presents a procedure for detecting differential item functioning (DIF) for dichotomous and polytomous items in testlet-based tests, whereby DIF is taken into account by adding DIF parameters into the Rasch testlet model. Simulations were conducted to assess recovery of the DIF and other parameters. Two independent variables, test type and anchor item method, were manipulated. The proportion of DIF items in the tests was set at approximately 40%. All parameters could be accurately recovered whenever the model was the generating model or only one DIF-free item was set as the anchor and all other items were treated as presumed DIF items. The bootstrap method is recommended to approximate the standard errors of the estimators for the DIF parameters so that the Wald statistic can be computed to test the null hypothesis of no DIF. Alternatively, the likelihood ratio test can be used to test the same hypothesis by comparing a full model with the DIF parameter and a reduced model without it.
Adams, R. J., Wilson, M. R., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
2.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
3.
Angoff, W. H., & Sharon, A. T. (1974). The evaluation of differences in test performance of two or more groups. Educational and Psychological Measurement, 34, 807-816.
4.
Ankenmann, R. D., Witt, E. A., & Dunbar, S. B. (1999). An investigation of the power of the likelihood ratio goodness-of-fit statistic in detecting differential item functioning. Journal of Educational Measurement, 36, 277-300.
5.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick, Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
6.
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
7.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
8.
Bock, R. D., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.
9.
Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
10.
Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153-168.
11.
Budgell, G. R., Raju, N. S., & Quartetti, D. A. (1995). Analysis of differential item functioning in translated assessment instruments. Applied Psychological Measurement, 19, 309-321.
12.
Candell, G. L., & Drasgow, F. (1988). An iterative procedure for linking metrics and assessing item bias in item response theory. Applied Psychological Measurement, 12, 253-260.
13.
Cleary, T. A., & Hilton, T. J. (1968). An investigation of item bias. Educational and Psychological Measurement, 28, 61-75.
14.
Cohen, A. S., Kim, S.-H., & Wollack, J. A. (1996). An investigation of the likelihood ratio test for detection of differential item functioning. Applied Psychological Measurement, 20, 15-26.
15.
De Boeck, P., & Wilson, M. R. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York: Springer-Verlag.
16.
Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, 39B, 1-38.
17.
Dodeen, H., & Johanson, G. A. (2003). An analysis of sex-related differential item functioning in attitude assessment. Assessment and Evaluation in Higher Education, 28, 129-134.
18.
Dorans, N. J., & Kulick, E. (1986). Demonstrating the utility of the standardization approach to assessing unexpected differential item performance on the Scholastic Aptitude Test. Journal of Educational Measurement, 23, 355-368.
19.
Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7, 1-26.
20.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
21.
Fischer, G. H. (1973). The linear logistic test model as instrument in educational research. Acta Psychologica, 37, 359-374.
22.
Fischer, G. H., & Pononcy, I. (1994). An extension of the partial credit model with an application to the measurement of change. Psychometrika, 59, 177-192.
23.
Hidalgo-Montesinos, M. D., & Lopez-Pina, J. A. (2002). Two-stage equating differential item functioning detection under the graded response model with the Raju area measures and the Lord statistic. Educational and Psychological Measurement, 62, 32-44.
24.
Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. Braun (Eds.), Test validity (pp. 129-145). Hillsdale, NJ: Lawrence Erlbaum.
25.
Holland, P. W., & Wainer, H. (Eds.). (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.
26.
Huang, C. D., Church, A. T., & Katigbak, M. S. (1997). Identifying cultural differences in items and traits: Differential item functioning in the NEO personality inventory. Journal of Cross-Cultural Psychology, 28, 192-218.
27.
Ip, E. H.-S. (2000). Adjusting for information inflation due to local dependency in moderately large item clusters. Psychometrika, 65, 73-91.
28.
Jodoin, M. G., & Gierl, M. J. (2001). Evaluating Type I error and power rates using an effect size measure with the logistic regression procedure for DIF detection. Applied Measurement in Education, 14, 329-349.
29.
Kim, S.-H., & Cohen, A. S. (1995). A comparison of Lord’s chi-square, Raju’s area measures, and the likelihood ratio test on detection of differential item functioning. Applied Measurement in Education, 8, 291-312.
30.
Kim, S.-H., & Cohen, A. S. (1998). Detection of differential item functioning under the graded response model with the likelihood ratio test. Applied Psychological Measurement, 22, 345-355.
31.
Li, H., & Stout, W. (1996). A new procedure for detection of crossing DIF. Psychometrika, 61, 647-677.
32.
Lim, R. G., & Drasgow, F. (1990). Evaluation of two methods for estimating item response theory parameters when assessing differential item functioning. Journal of Applied Psychology, 75, 164-174.
33.
Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago: Measurement, Evaluation, Statistics, and Assessment Press.
34.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
35.
Maller, S. J. (2001). Differential item functioning in the WISC-III: Item parameters for boys and girls in the national standardization sample. Educational and Psychological Measurement, 61, 793-817.
36.
Mantel, N., & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22, 719-748.
37.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
38.
McCullagh, P., & Nelder, J. A. (1989). Generalized linear models (2nd ed.). London: Chapman & Hall.
39.
McCulloch, C. E., & Searle, S. R. (2001). Generalized, linear, and mixed models. New York: John Wiley.
40.
Miller, M. D., & Oshima, T. C. (1992). Effect of sample size, number of biased items and magnitude of bias on a two-stage item bias estimation method. Applied Psychological Measurement, 16, 381-388.
41.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
42.
Nelder, J. A., & Wedderburn, R.W.M. (1972). Generalized linear models. Journal of the Royal Statistical Society A, 135, 370-384.
43.
Neyman, J., & Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika, 20A, 174-240, 263-294.
44.
Raju, N. S. (1988). The area between two item characteristics curves. Psychometrika, 53, 495-502.
45.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Institute of Educational Research.
46.
Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185-205.
47.
Rosenbaum, P. R. (1988). Item bundles. Psychometrika, 53, 349-359.
48.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph Supplement, 17, 1-100.
49.
SAS Institute Inc. (1999). The NLMIXED procedure [Computer software]. Cary, NC: Author.
50.
Scheuneman, J. D. (1979). A method of assessing bias in test items. Journal of Educational Measurement, 16, 143-152.
51.
Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58, 159-194.
52.
Shepard, L., Camilli, G., & Averill, M. (1981). Comparison of procedures for detecting test-item bias with both internal and external ability criteria. Journal of Educational Statistics, 6, 317-375.
53.
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28, 237-247.
54.
Swaminathan, H., & Rogers, H. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361-370.
55.
Thissen, D., Steinberg, L., & Gerrard, M. (1986). Beyond group mean differences: The concept of item bias. Psychological Bulletin, 99, 118-128.
56.
Thissen, D., Steinberg, L., & Wainer, H. (1988). Use of item response theory in the study of group differences in trace lines. In H. Wainer & H. Braun (Eds.), Test validity (pp. 147-169). Hillsdale, NJ: Lawrence Erlbaum.
57.
van der Flier, H., Mellenbergh, G. J., Adér, H. J., & Wijn, M. (1984). An iterative item bias detection method. Journal of Educational Measurement, 21, 131-146.
58.
Volodin, N., & Adams, R. J. (1995, April). Identifying and estimating a D-dimensional Rasch model. Paper presented the International Objective Measurement Workshop, Berkeley, California.
59.
Wainer, H. (1995). Precision and differential item functioning on a testlet-based test: The 1991 Law School Admissions Test as an example. Applied Measurement in Education, 8, 157-186.
60.
Wainer, H., Bradlow, E. T., & Du, Z. (2000). Testlet response theory: An analog for the 3PL model using in testlet-based adaptive testing. In W. van der Linden & C.A.W. Glas (Eds.), Computerized adaptive testing: Theory and practice (pp. 245-269). London: Kluwer.
61.
Wainer, H., & Kiely, G. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24, 185-202.
62.
Wainer, H., & Lukhele, R. (1997). How reliable are TOEFL scores?Educational and Psychological Measurement, 57, 749-766.
63.
Wainer, H., Sireci, S., & Thissen, D. (1991). Differential testlet functioning: Definitions and detection. Journal of Educational Measurement, 28, 197-219.
64.
Wainer, H., & Thissen, D. (1996). How is reliability related to the quality of test scores? What is the effect of local dependence on reliability?Educational Measurement: Issues and Practice, 15(1), 22-29.
65.
Wainer, H., & Wang, X. (2000). Using a new statistical model for testlets to score TOEFL. Journal of Educational Measurement, 37, 203-220.
66.
Wald, A. (1943). Tests of statistical hypotheses concerning several parameters when the number of observations in large. Transactions of the American Mathematical Society, 54, 426-482.
67.
Wang, W.-C. (2000a). Modeling effects of differential item functioning in polytomous items. Journal of Applied Measurement, 1, 63-82.
68.
Wang, W.-C. (2000b). The simultaneous factorial analysis of differential item functioning. Methods of Psychological Research Online, 5(1), 51-76.
69.
Wang, W.-C. (2004). Effects of anchor item methods on the detection of differential item functioning within the family of Rasch models. Journal of Experimental Education, 72, 221-261.
70.
Wang, X., Bradlow, E. T., & Wainer, H. (2001). SCORIGHT (version 1.2) [Computer software]. Princeton, NJ: Educational Testing Service.
71.
Wang, X., Bradlow, E. T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26, 109-128.
72.
Wang, W.-C., & Wilson, M. R. (in press). The Rasch testlet model. Applied Psychological Measurement.
73.
Wang, W.-C., & Yeh, Y.-L. (2003). Effects of anchor item methods on differential item functioning detection with the likelihood ratio test. Applied Psychological Measurement, 27, 479-498.
74.
Wilson, M. R. (1992). The partial order model: An extension of the partial credit model. Applied Psychological Measurement, 16, 309-325.
75.
Wilson, M. R., & Adams, R. J. (1995). Rasch models for item bundles. Psychometrika, 60, 181-198.
76.
Wolfinger, R. D., & SAS Institute Inc. (n.d.). Fitting nonlinear mixed models with the new NLMIXED procedure. Retrieved August 17, 2003, from http://support.sas.com/rnd/app/papers/nlmixedsugi.pdf
77.
Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: Measurement, Evaluation, Statistics, and Assessment Press.
78.
Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ConQuest [Computer software and manual]. Camberwell: Australian Council for Educational Research.
79.
Yen, W. (1993). Scaling performance assessment: Strategies for managing local item dependence. Journal of Educational Measurement, Australia30, 187-213.
80.
Zimowski, M., Muraki, E., Mislevy, R., & Bock, D. (2003). BILOG-MG (version 3) [Computer software and manual]. Lincolnwood, IL: Scientific Software International.
