Adjusting the Adjusted χ 2 / df Ratio Statistic for Dichotomous Item Response Theory Analyses

Abstract

Two Monte Carlo simulation studies investigated the effectiveness of the mean adjusted χ²/df statistic proposed by Drasgow and colleagues and, because of problems with the method, a new approach for assessing the goodness of fit of an item response theory model was developed. It has been previously recommended that mean adjusted χ²/df values greater than 3 using a cross-validation data set indicate substantial misfit. The authors used simulations to examine this critical value across different test lengths (15, 30, 45) and sample sizes (500, 1,000, 1,500, 5,000). The one-, two- and three-parameter logistic models were fitted to data simulated from different logistic models, including unidimensional and multidimensional models. In general, a fixed cutoff value was insufficient to ascertain item response theory model–data fit. Consequently, the authors propose the use of the parametric bootstrap to investigate misfit and evaluated its performance. This new approach produced appropriate Type I error rates and had substantial power to detect misfit across simulated conditions. In a third study, the authors applied the parametric bootstrap approach to LSAT data to determine which dichomotous item response theory model produced the best fit. Future applications of the mean adjusted χ²/df statistic are discussed.

Keywords

bootstrap fit statistic item response theory

Item response theory (IRT; Lord, 1980) is widely used for the measurement of cognitive ability. A critical issue in the application of IRT is ensuring that the model fits the data. One statistic commonly used to evaluate IRT model–data fit is the adjusted (to a sample size of 3,000) χ²/df ratio with values greater than 3 in a cross-validation data set proposed as indicating model misfit (Drasgow, Levine, Tsien, Williams, & Mead, 1995). This statistic has been widely used for examining IRT model–data fit for the purposes of differential item functioning (DIF; Kulas & Finkelstein, 2007; Rivers, Meade, & Fuller, 2009; Robie, Zickar, & Schmit, 2001; Zickar & Robie, 1999), scale development and evaluation (Rauch, Schweizer, & Moosbrugger, 2008; Scherbaum, Cohen-Charash, & Kern, 2006; Schmit, Kihm, & Robie, 2000), and IRT model comparisons to determine which model to use (Chernyshenko, Stark, Chan, Drasgow, & Williams, 2001; Stark, Chernyshenko, Drasgow, & Williams, 2006; Tay, Drasgow, Rounds, & Williams, 2009).

Although the adjusted χ²/df ratio has been widely used, there has not been a systematic examination of its effectiveness for evaluating IRT model–data fit for dichotomously scored items. In contrast, IRT indices for a variety of other applications have been evaluated: for example, Raju, van der Linden, and Fleer’s (1995) DIF analysis (e.g., Bolt, 2002; Flowers, Oshima, & Raju, 1999; Meade, Lautenschlager, & Johnson, 2007), person–fit measures such as Drasgow, Levine, and Williams’s (1985) z₃ (e.g., Drasgow, Levine, & McLaughlin, 1987), and Tatsuoka’s (1984) extended caution indices (e.g., St-Onge, Valois, Abdous, & Germain, 2009), among others. Recently, an examination of the adjusted χ²/df ratio statistic for polytomously scored items from scales of different lengths (10 and 20 items) with various sample sizes (500, 1,000, and 2,000) showed that a cutoff value of 3 produced large Type I error rates for cross-validation data (LaHuis, Clark, & O’Brien, 2011). In this study, we simulated dichotomous items to ascertain whether the adjusted χ²/df ratio can be used for assessing model-fit in this context. We also examine a wide range of data conditions. As an alternative to the cutoff value of 3, which may not be generalizable across conditions, we propose and test the use of a parametric bootstrap approach to the adjusted χ²/df ratio for assessing absolute model–data fit.

The Adjusted χ²/df Ratio: Singles, Doubles, and Triples

Drasgow et al. (1995) proposed the use of an IRT model–data fit based on the ordinary χ² value,

χ_{i}^{2} = \sum_{k} \frac{{[O_{i k} - E_{i k}]}^{2}}{E_{i k}},

where O_ik is the observed frequency for response option k to item i, and E_ik is the expected frequency, which is computed from the item response function for option k integrated over a standard normal density f(θ) multiplied by the total sample N,

E_{i k} = N \int P (u_{i} = k | θ) f (θ) d θ .

This χ²-statistic was termed the singles χ². However, because the singles χ² is a marginal statistic, it is insensitive to certain types of misfit (Van den Wollenberg, 1982). Hence, another statistic, termed the doubles χ², was developed for a pair of items i and i′. Observed frequencies are compared with the expected frequency of endorsement given by

E_{i i^{'} k k^{'}} = N \int P (u_{i} = k | θ) P (u_{i^{'}} = k^{'} | θ) f (θ) d θ .

By extension, the triples χ² value is computed using a multiway contingency table for triples of items. We note that the doubles χ² value is analogous to the bivariate residual statistic implemented in Latent GOLD (Vermunt & Magidson, 2000, 2002, 2005). Both examine the correspondence between the model-based expected values and the observed values.

One problem with the χ² is its sensitivity to sample size because the expected value of a noncentral χ² is equal to its df plus N times its noncentrality parameter δ,

E (χ^{2}) = d f + N δ .

To resolve this problem, Drasgow and his colleagues recommended that the χ² indices be adjusted to a fixed sample size of 3,000, with the expectation that these indices would be comparable across different sample sizes. Thus, an estimate of the noncentrality parameter is

\hat{δ} = (χ^{2} - d f) / N .

An observed χ² can be adjusted to fixed sample size of, say, n = 3,000, by

χ_{3000}^{2} = [d f + 3000 (χ^{2} - d f) / N] .

After the adjustment, the values can be divided by the degrees of freedom to produce the adjusted χ²/df value,

Adjusted χ_{3000}^{2} = [d f + 3000 (χ^{2} - d f) / N] / d f .

As a heuristic, Drasgow and his colleagues recommended that a cutoff value of 3 be used as an index of misfit. In their calculations, they took the df as the number of cells in a contingency table minus one.

There are still several issues which are not resolved with the use of an adjusted χ²/df value. Foremost, there has not been a systematic examination of the proposed cutoff value across a variety of data conditions. Indeed, for this statistic and recommended critical value to be useful, it has to be generalizable across different sample sizes and test lengths. In addition, although cross-validation samples have been recommended as a way to ensure that models are not overfit to the data (Drasgow et al., 1995), it has been found that validation samples have little power to detect model misspecification when the sample sizes are small (<2,000; Tay, Ali, Drasgow, & Williams, 2011). Joe and Maydeu-Olivares (2006) have shown that the χ²-statistic does not follow a χ² distribution for cross-validation samples. However, are there conditions where the adjusted χ²/df statistic can be used if the sample size is large enough?

Furthermore, a critical question is whether the singles-, doubles-, or triples-adjusted χ²/df fares better in detecting misfit and whether the same critical value applies across the three variations of the same statistic. We expected that because the singles χ²/df is a marginal statistic, it would be less sensitive in detecting model misspecifications as compared with the doubles- or triples-adjusted χ²/df values.

Study 1

In this study, we used a simulation to examine (a) whether the rule of thumb of 3 holds across a range of sample sizes and test lengths, (b) the effects of using a calibration or cross-validation sample, and (c) the usefulness of the mean singles-, doubles-, and triples-adjusted χ²/df. We focused on only dichotomously scored items as these are most commonly used in cognitive ability testing. Specifically, we simulated data based on the one-parameter (1PLM), two-parameter (2PLM), or three-parameter logistic (3PLM) IRT model and subsequently fit all three models to the data. Fitting the correct IRT model to the data (null condition) or a more general model to the data (e.g., 3PLM to 2PLM data) allows us to ascertain the number of instances the cutoff value of 3 is breached, producing the Type I error rate. Alternatively, by fitting a more restricted IRT model to the data (e.g., 2PLM fit to 3PLM data), we were able to ascertain power based on a cutoff value of 3. In addition to examining the power to detecting whether a more restricted unidimensional IRT model fits the data (e.g., 2PLM fit to 3PLM), we also examined whether unidimensional IRT models fit multidimensional IRT data. We evaluated the effectiveness of the adjusted χ²/df index for detecting model misspecification by adjusted power, which equals power minus the actual Type I error rate.

In this study, we used sample sizes of 500, 1,000, 1,500, and 5,000 simulees crossed with test lengths of 15, 30, and 45. Across each of these 4 (sample size) × 3 (test length) = 12 conditions for three-dimensionality, 200 replications were undertaken in which data were simulated, IRT models were fit, and the adjusted χ²/df ratio was computed. The dimensionality conditions included unidimensionality simulations as well as simulations of low and moderate multidimensionality where two latent traits were correlated at .80 and .60, respectively.

Simulation

First, data were simulated based on the 1PLM, 2PLM, or 3PLM. The 3PLM is given by

P (u_{i} = 1 | θ_{j}) = c_{i} + \frac{1 - c_{i}}{1 + \exp [- 1.702 a_{i} (θ_{j} - b_{i})]},

where the probability of correctly answering an item (u_i = 1) is based on the simulated θ _j sampled from a standard normal distribution, a_i is the item discrimination parameter sampled from a log-normal (0, .5) distribution and divided by 1.702, b_i is the item difficulty parameter randomly sampled from a uniform (−2, 2) distribution, and c_i was sampled from a logit-normal (−1.1, 0.5) distribution resulting in a mean value of c_i = 0.25. The 2PLM data were simulated by fixing c_i = 0 and the 1PLM was simulated with c_i = 0 and a single a_i value for all the items.

Multidimensionality was simulated following procedures by Zhang and Stout (1999). The latent trait distribution was bivariate normal with zero means, unit variance, and a correlation of .80, or .60, reflecting low and moderate multidimensionalities, respectively. The odd-numbered items were set to measure the first dimension and the even-numbered items were set to measure the second dimension, resulting in a simple structure with two factors.

Next, all three IRT models were fit to the data using BILOG-MG (Zimowski, Muraki, Mislevy, & Bock, 1996) using the default settings, and the adjusted χ²/df ratios for each fitted IRT model was computed using FORSCORE (Williams & Levine, 1993) for the calibration samples as well as a cross-validation sample with the same number of simulated items and people. The measure used in this study was the mean adjusted χ²/df ratio for all singles, doubles, and triples for the test.

Type I Error and Adjusted Power

Both Type I error and adjusted power were determined by the proportions of replications that had mean adjusted χ²/df ratios greater than 3. These proportions were considered Type I errors when the correct model or a more general IRT model was fit to the data. In contrast, adjusted power was determined when a less restricted model was fit to data from a more general IRT model.

Results

For our results, we only present Type I error rates when the correct model was fit to the data because similar values were obtained when a more general IRT model was fit to the data. In view of space considerations, we present detailed results for moderate multidimensionality (latent traits correlated .60) in the tables. The trend was very similar when data were less multidimensional (latent traits correlated .80); in general, there was less power to detect low multidimensionality as compared with moderate multidimensionality.

1PLM data

Table 1 shows the results of fitting the 1PLM to unidimensional and multidimensional 1PLM data. For the calibration samples, the Type I error rates were uniformly close to zero and, therefore, we do not present them in Table 1. For the validation sample, high Type I error rates are evident for relatively smaller sample sizes of 500 but are close to the desired nominal rate when the sample size was much larger (i.e., 5,000). Indeed, we expected that item parameter estimation error would be larger for smaller sample sizes, leading to higher Type I error rates as compared with the other conditions.

Table 1.

Proportion of Mean Adjusted χ²/df Ratios Greater Than 3 for 1PLM Simulated Data

		Simulated 1PLM calibration samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Adjusted power
1PLM (.6)	Singles	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	Doubles	0.11	0.31	0.49	0.60	0.12	0.30	0.43	0.63	0.11	0.31	0.48	0.59
	Triples	0.40	0.52	0.64	0.70	0.40	0.53	0.65	0.71	0.36	0.59	0.66	0.69
		Simulated 1PLM Validation Samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Type I error rates
1PLM	Singles	0.77	0.59	0.39	0.04	0.85	0.64	0.46	0.03	0.85	0.58	0.52	0.04
	Doubles	0.74	0.48	0.27	0.01	0.82	0.50	0.31	0.01	0.82	0.43	0.30	0.02
	Triples	0.66	0.36	0.17	0.00	0.73	0.32	0.14	0.00	0.75	0.22	0.14	0.00
		Adjusted power
1PLM (.6)	Singles	0.00	0.02	0.05	0.00	0.00	0.00	0.00	0.01	0.06	0.15	0.00	0.00
	Doubles	0.19	0.39	0.60	0.70	0.16	0.44	0.58	0.75	0.17	0.53	0.58	0.68
	Triples	0.28	0.50	0.70	0.77	0.26	0.60	0.73	0.78	0.24	0.72	0.74	0.76

Note. 1PLM = One-parameter logistic model. Type I error rates for calibration samples were zero across all simulated conditions and are omitted.

When the data were multidimensional, the doubles- and triples-adjusted χ²/df ratios had higher adjusted power as sample size and test length increased. However, singles-adjusted χ²/df ratios were not useful in detecting multidimensionality as evidenced from low adjusted power. As expected, adjusted power increased as multidimensionality increased. Using the triples-adjusted χ²/df ratios, the average power across the conditions to detect moderate multidimensionality was .57 (latent traits correlated at .60) whereas the average power to detect low multidimensionality was .11 (latent traits correlated at .80).

2PLM data

As with the simulated 1PLM data, Type I error rates were zero for the calibration samples. Again, for validation samples, the Type I error rates were much higher for smaller sample sizes but were close to zero for sample sizes of 5,000 as shown in Table 2.

Table 2.

Proportion of Mean Adjusted χ²/df Ratios Greater Than 3 for 2PLM Simulated Data

		Simulated 2PLM calibration samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Adjusted power
1PLM	Singles	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	Doubles	0.11	0.49	0.68	0.86	0.12	0.58	0.78	0.98	0.08	0.72	0.92	1.00
	Triples	0.41	0.80	0.91	0.94	0.61	0.94	0.97	1.00	0.67	0.99	1.00	1.00
2PLM (.6)	Singles	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	Doubles	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	Triples	0.00	0.00	0.01	0.02	0.00	0.00	0.00	0.01	0.00	0.00	0.00	0.00
		Simulated 2PLM validation samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted Model		Type I error rates
2PLM	Singles	0.81	0.60	0.42	0.01	0.90	0.69	0.40	0.01	0.94	0.72	0.40	0.00
	Doubles	0.81	0.53	0.33	0.00	0.90	0.52	0.23	0.01	0.94	0.62	0.21	0.00
	Triples	0.76	0.35	0.15	0.00	0.86	0.30	0.05	0.00	0.87	0.31	0.03	0.00
		Adjusted power
1PLM	Singles	0.04	0.00	0.06	0.00	0.05	0.08	0.00	0.00	0.09	0.15	0.00	0.00
	Doubles	0.24	0.52	0.72	0.96	0.18	0.50	0.69	0.99	0.18	0.57	0.70	0.98
	Triples	0.33	0.64	0.82	0.98	0.27	0.68	0.86	1.00	0.25	0.78	0.86	1.00
2PLM (.6)	Singles	0.00	0.03	0.06	0.00	0.00	0.03	0.11	0.00	0.01	0.00	0.00	0.00
	Doubles	0.10	0.28	0.37	0.12	0.05	0.35	0.55	0.11	0.06	0.25	0.57	0.06
	Triples	0.18	0.51	0.55	0.22	0.09	0.60	0.78	0.24	0.13	0.58	0.77	0.23

Note. 1PLM = one-parameter logistic model; 2PLM = two-parameter logistic model. Type I error rates for calibration samples were zero across all simulated conditions and are omitted.

Using a cutoff value of 3, the adjusted power to detect misfit—when the 1PLM was fit to 2PLM data—for doubles- and triples-adjusted χ²/df ratios was moderate to excellent across all conditions, including calibration and validation data sets. Adjusted power increased as a function of a larger sample size and test length. However, unlike 1PLM data, the adjusted power to detect multidimensionality was poor. For calibration samples, adjusted power was uniformly close to zero for both low and moderate multidimensionality. Importantly, the mean adjusted χ²/df ratios for the multidimensional data were consistently elevated as compared with when 2PLM was correctly fit to unidimensional data, but values were lower than the cutoff value of 3. For example, the mean values of the triples-adjusted χ²/df ratio for 15 items ranged from −.71 to 2.02 when data were moderately multidimensional whereas they ranged from −2.24 to 0.68 when the data were unidimensional.

For validation samples of small to moderate sample sizes, Type I error rates were too high for power to be useful, whereas Type I error rates were low for sample sizes of 5,000. For this largest sample size, the adjusted power to detect moderate multidimensionality was low, ranging from .06 to .23 for doubles- and triples-adjusted χ²/df ratios. In addition, for sample sizes of 1,500 to 5,000, the mean adjusted χ²/df ratios for doubles and triples decreased from above the critical value of 3 to below 3. This resulted in low power to detect multidimensionality even when Type I error rates were low. These results clearly show that a simple cutoff value of 3 is not generalizable.

3PLM data

The Type I error rates were very low and uniformly close to zero for calibration data as with the simulated 2PLM and 1PLM data (Table 3). For validation data, Type I error rates also had the same trend as found before—Type I error rates were high for small sample sizes but were low for the largest sample size (i.e., 5,000).

Table 3.

Proportion of Mean Adjusted χ²/df Ratios Greater Than 3 for 3PLM Simulated Data

		Simulated 3PLM calibration samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Adjusted power
1PLM	Singles	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
	Doubles	0.00	0.02	0.05	0.16	0.00	0.01	0.03	0.17	0.00	0.00	0.01	0.14
	Triples	0.04	0.17	0.20	0.40	0.01	0.13	0.27	0.47	0.01	0.11	0.22	0.53
		Simulated 3PLM validation samples
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Type I error rates
3PLM	Singles	0.76	0.63	0.52	0.02	0.90	0.68	0.45	0.00	0.95	0.81	0.49	0.00
	Doubles	0.75	0.53	0.32	0.00	0.88	0.57	0.25	0.00	0.92	0.62	0.23	0.00
	Triples	0.68	0.34	0.11	0.00	0.82	0.29	0.04	0.00	0.83	0.28	0.04	0.00
		Adjusted power
2PLM	Singles	0.00	0.02	0.08	0.00	0.00	0.01	0.05	0.00	0.00	0.07	0.11	0.01
	Doubles	0.00	0.03	0.01	0.00	0.03	0.09	0.04	0.00	0.01	0.09	0.08	0.00
	Triples	0.00	0.07	0.01	0.00	0.03	0.11	0.05	0.00	0.05	0.14	0.06	0.00
1PLM	Singles	0.00	0.03	0.10	0.00	0.05	0.04	0.00	0.00	0.10	0.24	0.00	0.00
	Doubles	0.14	0.36	0.53	0.49	0.14	0.44	0.58	0.61	0.17	0.56	0.66	0.65
	Triples	0.26	0.52	0.65	0.63	0.24	0.63	0.77	0.72	0.25	0.77	0.83	0.82
3PLM (.6)	Singles	0.04	0.01	0.00	0.00	0.00	0.01	0.00	0.01	0.04	0.00	0.01	0.00
	Doubles	0.08	0.10	0.06	0.01	0.03	0.05	0.05	0.00	0.05	0.07	0.08	0.00
	Triples	0.11	0.13	0.05	0.01	0.05	0.10	0.07	0.00	0.07	0.12	0.05	0.00

Note. 1PLM = One-parameter logistic model; 2PLM = two-parameter logistic model; 3PLM = three-parameter logistic model. For calibration sample, Type I error rates were zero across all simulated conditions; adjusted power was zero for multidimensional 3PLM (latent traits correlated at .60) and for 2PLM. These are omitted from the table.

When a 1PLM was fit to 3PLM data, the adjusted power was different between validation and calibration samples. Therefore, although there was little difference in adjusted power between validation and calibration samples when the 1PLM was fitted to 2PLM data, validation samples had higher adjusted power than calibration samples in this instance. When fitting the 2PLM to 3PLM data using a cutoff value of 3, there was little or no adjusted power to detect misfit for both calibration and validation data. Hence, these two models were not distinguishable using the rule of thumb. Similarly, there was little power to detect moderate multidimensionality for both calibration and validation data. Although the mean adjusted χ²/df was higher when a misspecified model was fit to the data, many values were lower than the critical value of 3 resulting in low power.

Discussion

This initial study contributes to the current literature on IRT model–data fit using the adjusted χ²/df ratios in several important ways. Past studies have frequently used the cutoff value of 3 across all three different adjusted χ²/df ratios—singles, doubles, and triples—as a justification that the IRT model fits the data. To our knowledge, this study was the first to use Monte Carlo simulations to rigorously determine the effectiveness of this cutoff value for the different adjusted χ²/df ratios. We found that adjusted χ²/df ratios were generally effective for detecting model misfit when a 1PLM was fit to 2PLM or 3PLM data in doubles and triples but not in singles. Furthermore, with large sample sizes of 5,000 or more, analysis of validation data was somewhat effective. This confirms the recommendations by Drasgow et al. (1995) and their rule of thumb. Thus, in spite of past research showing that the χ² value does not follow a χ² distribution in validation samples (Joe & Maydeu-Olivares, 2006), this approach to assessing goodness of fit appears to be reasonable in large samples. However, when moderate sample sizes are used (i.e., 500 to 1,500), the use of validation samples is not recommended because of high Type I error rates.

Nevertheless, our simulations suggest that the 3PLM and 2PLM are not easily distinguishable on the basis of the adjusted χ²/df ratio for a cutoff value of 3; similarly, there was little power to detect moderate multidimensionality for the 3PLM and 2PLM. This occurred because even though the mean adjusted χ²/df ratios were elevated for misspecified models, they did not usually exceed the cutoff value of 3. To ascertain the relative fit of several models, we recommend another strategy: Fit different IRT models to the data and compare the relative magnitudes of the adjusted χ²/df ratios for the calibration data set. Indeed, this strategy has been undertaken in the past when attempting to compare the fits of different IRT models with empirical data (e.g., Chernyshenko et al., 2001; Stark et al., 2006; Tay et al., 2009). Furthermore, simulation studies have shown that this method is helpful to distinguish dominance and ideal point data (Tay et al., 2011). This is particularly important given that the adjusted χ²/df ratio appears to be affected by sample size and test length despite the sample size adjustment. However, this method does not give an indication of absolute fit. In Study 2, we propose a parametric bootstrap approach for assessing absolute IRT model-fit.

Finally, we also determined that not all types of adjusted χ²/df ratios are equally effective for detecting misfit. Specifically, the singles-adjusted χ²/df ratio had very low power and we do not recommend its use. Instead, we recommend that the doubles- and triples-adjusted χ²/df ratios be used in the future. For the types of model misspecification studied here, triples had consistently larger power than doubles.

Study 2

Method

Instead of using a fixed cutoff value of 3 for examining model-fit, which Study 1 found to be ineffective across various conditions, we propose the use of a parametric bootstrap approach instead (Langeheine, Pannekok, & Van de Pol, 1996; Vermunt, 2001; von Davier, 1997). The general procedure is as follows:

Following procedures from Study 1, we simulated data from one of the dichotomous IRT models. This data set is referred to as the observed data.

We estimated item parameters for a model (i.e., a 1 PLM, 2 PLM, or 3PLM) and calculated the mean adjusted χ²/df ratio; this is referred to as the observed mean adjusted χ²/df value.

The estimated item parameters from the observed data were then used to simulate data using the IRT model from Step (b). This model was then fit to the simulated data. The mean adjusted χ²/df ratio for the IRT model was calculated for this new simulated data set.

Step (c) was repeated multiple times (1, . . ., m) so that a sampling distribution for the mean adjusted χ²/df ratio for the IRT model from Step (b) was obtained; the observed adjusted χ²/df value was then compared with the sampling distribution to obtain an empirical p value; a relatively larger observed adjusted χ²/df value would have a relatively smaller p value.

Steps (a) to (d) were then replicated (1, . . ., r) to determine power and Type I error rates. Power was determined by the proportion of replications that had an empirical p value less than .05 when a more restricted model was fitted to the observed data (e.g., by fitting a 1PLM to 3PLM observed data). In contrast, the Type I error rate is the proportion of replications that had an empirical p value less than .05 when the correct model, or a less restricted model, was fitted to the observed data (e.g., a 3PLM was fit to 1PLM observed data).

We used the same sample sizes and test lengths as the first simulation study. Across each of these 12 conditions, m = 50 replications were taken and r = 50 bootstrap samples drawn per replication. As before, for each of the conditions, the simulated observed data conformed to the 1PLM, 2PLM, or 3PLM. We decided to use a smaller number of replications and bootstrap samples because the results from m = 100 replications and r = 200 bootstrap samples were very similar in the trial runs as using a large number of replications and bootstrap samples is very time consuming. For example, for the 1PLM 15-item condition, the Type I error rates (m = 50 and r = 50) for singles, doubles, and triples were .02, .06, and .05, respectively; whereas a larger number of replications and bootstrap samples (m = 100 and r = 200) yielded Type I error rates of .02, .03, and .03, respectively. Furthermore, each replication was very involved, requiring 100 bootstrap samples of each IRT model (i.e., 1PLM, 2PLM, and 3PLM) and calculation of the singles-, doubles-, and triples-adjusted χ²/df ratio; as a result, each replication required 2 hours, and a total of 2 hours × 100 replications × 12 conditions × 3 different dichotomous IRT models = 14,400 computing hours would be needed for 200 bootstrap samples of 100 replications per condition.

Results

1PLM data

Table 4 presents the Type I error rates and the adjusted power to detect misfit using the bootstrap approach. Foremost, the Type I error rates were controlled better than with the greater-than-3 rule of thumb. They ranged from .00 to .14, for singles-, doubles-, and triples-adjusted χ²/df ratio when fitting the 1PLM to the 1PLM data. Although not presented in the table, we found that the Type I error rates (M = .79) were much higher when fitting the 3PLM to the 1PLM data. This was a function of the default c-prior in BILOG; assuming four choices in an item, the c-prior is given a mean of .25. This resulted in item parameter estimates that were consistently biased away from the simulated 1PLM item parameters. When a low 3PLM mean prior (i.e., c = .01) was specified for the 1PLM observed data, the bootstrap approach yielded excellent Type I error rates (M = .02) for the 3PLM, equivalent to the 1PLM. These results show that the 3PLM can fit 1PLM data with an appropriate prior for c; however, the results also suggest that we can determine when the 1PLM is sufficient because it fits better than the 3PLM with BILOG’s default c-prior.

Table 4.

Type I Error Rates and Power to Detect Misfit Using a Replication Approach

		Simulated 1PLM
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
Fitted model		Type I error
1PLM	Singles	0.02	0.06	0.12	0.08	0.06	0.02	0.04	0.06	0.06	0.06	0.10	0.02
	Doubles	0.06	0.02	0.06	0.04	0.12	0.06	0.00	0.12	0.06	0.04	0.14	0.08
	Triples	0.04	0.02	0.04	0.08	0.12	0.04	0.00	0.12	0.06	0.04	0.10	0.10
		Adjusted power
1PLM (.6)	Singles	0.00	0.00	0.02	0.00	0.00	0.06	0.02	0.00	0.02	0.02	0.00	0.00
	Doubles	0.88	0.96	0.92	0.96	0.86	0.90	0.98	0.88	0.88	0.94	0.86	0.92
	Triples	0.90	0.96	0.94	0.92	0.88	0.92	0.98	0.88	0.90	0.94	0.90	0.90
Fitted model		Simulated 2PLM
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
		Type I error
2PLM	Singles	0.02	0.04	0.04	0.00	0.04	0.10	0.06	0.02	0.08	0.04	0.02	0.02
	Doubles	0.08	0.08	0.04	0.10	0.16	0.12	0.10	0.06	0.14	0.06	0.10	0.04
	Triples	0.12	0.04	0.04	0.10	0.14	0.14	0.08	0.04	0.12	0.06	0.12	0.08
		Adjusted power
1PLM	Singles	0.08	0.14	0.24	0.44	0.18	0.10	0.28	0.38	0.08	0.18	0.18	0.46
	Doubles	0.92	0.92	0.96	0.90	0.84	0.88	0.90	0.94	0.86	0.94	0.90	0.96
	Triples	0.88	0.96	0.96	0.90	0.86	0.86	0.92	0.96	0.88	0.94	0.88	0.92
2PLM (.6)	Singles	0.00	0.02	0.00	0.04	0.00	0.00	0.00	0.08	0.00	0.04	0.04	0.00
	Doubles	0.92	0.92	0.96	0.90	0.84	0.88	0.90	0.94	0.86	0.94	0.90	0.96
	Triples	0.84	0.96	0.96	0.90	0.86	0.86	0.92	0.96	0.88	0.94	0.88	0.92
Fitted model		Simulated 3PLM
		15 items				30 items				45 items
		500	1,000	1,500	5,000	500	1,000	1,500	5,000	500	1,000	1,500	5,000
		Type I error
3PLM	Singles	0.06	0.10	0.02	0.04	0.06	0.04	0.04	0.04	0.06	0.06	0.00	0.08
	Doubles	0.14	0.08	0.06	0.00	0.12	0.10	0.06	0.10	0.10	0.16	0.10	0.10
	Triples	0.14	0.06	0.04	0.00	0.14	0.12	0.04	0.06	0.08	0.20	0.10	0.08
		Adjusted power
2PLM	Singles	0.18	0.30	0.48	0.58	0.80	0.82	0.80	0.82	0.84	0.90	0.98	0.92
	Doubles	0.28	0.20	0.34	0.38	0.78	0.78	0.84	0.84	0.90	0.80	0.90	0.90
	Triples	0.26	0.28	0.46	0.84	0.76	0.82	0.90	0.92	0.90	0.78	0.90	0.92
1PLM	Singles	0.02	0.04	0.26	0.48	0.36	0.42	0.44	0.68	0.40	0.60	0.80	0.86
	Doubles	0.76	0.86	0.94	1.00	0.88	0.90	0.94	0.90	0.90	0.84	0.90	0.90
	Triples	0.76	0.88	0.96	1.00	0.86	0.88	0.96	0.94	0.92	0.80	0.90	0.92
3PLM (.6)	Singles	0.02	0.00	0.04	0.08	0.00	0.02	0.04	0.18	0.06	0.00	0.10	0.12
	Doubles	0.34	0.54	0.78	0.94	0.62	0.86	0.94	0.90	0.86	0.84	0.90	0.90
	Triples	0.26	0.54	0.78	1.00	0.66	0.84	0.96	0.94	0.88	0.80	0.90	0.92

Note. 1PLM = One-parameter logistic model; 2PLM = two-parameter logistic model; 3PLM = three-parameter logistic model.

Table 4 shows that there was high adjusted power (M = .92) to detect moderate multidimensionality using doubles- and triples-adjusted χ²/df ratio. When their low multidimensionality was simulated, the average power (M = .84) for both doubles- and triples-adjusted χ²/df was lower, as would be expected.

2PLM data

For Type I error rates, the same trends were observed as with the 1PLM data. Although not presented in the table, we found that fitting a 2PLM resulted in excellent Type I error rates (M = .07). However, fitting the 3PLM with the default prior for c produced high Type I errors (M = .72). When a low BILOG c-prior was used, the Type I error rates were substantially lower (M = .02).

Importantly, the adjusted power to detect misfit was consistently higher than .84 for doubles and triples across all the simulated conditions when a 1PLM was fit to the 2PLM data. In contrast, the singles-adjusted χ²/df ratio had little power. This again confirms the notion that the singles-adjusted χ²/df ratio is much less useful than the doubles and triples. When fitting the 2PLM to moderately multidimensional 2PLM data, the adjusted power for doubles- and triples-adjusted χ²/df ratio was also high (M = .91). For less multidimensional 2PLM data, the adjusted power was lower (M = .83) but increased with sample size and test length. This is quite different from the results shown in Table 2 and illustrates the advantage of the parametric bootstrap approach.

3PLM data

When the 3PLM was fit to simulated 3PLM data, the Type I error rates were excellent to slightly inflated, ranging from .00 to .20, respectively. Although the Type I error rates at times were higher than the nominal .05 level, this is a vast improvement over using a critical value of 3. Across all conditions, the mean Type I error rates for singles-, doubles-, and triples-adjusted χ²/df ratios were .05, .09, and .09, respectively. When a more restricted IRT model—the 2PLM—was fit to 3PLM data, the power to detect misfit was excellent for longer tests (i.e., more than or equal to 30 items). Excluding the 15-item test length condition, the average adjusted powers of doubles- and triples-adjusted χ²/df ratios were .84 and .86, respectively. Adjusted power was lower, however, for the simulated 15-item tests. This shows that a longer test length is needed to differentiate the 2PLM and 3PLM. The power to detect misfit was higher when a 1PLM was fit to 3PLM data for the doubles- and triples-adjusted χ²/df ratios, with an average of .90 for both doubles and triples. In contrast, the singles-adjusted χ²/df ratio was much less effective.

Empirical Example

We applied the parametric bootstrap approach to the LSAT 6 data from Bock and Lieberman (1980), which consists of the responses of 1,000 participants to five dichotomous items. Joe and Maydeu-Olivares (2006) obtained goodness-of-fit statistics for the 1PLM, 2PLM, and 3PLM fit to a split-half sample of the LSAT 6 data; their p values were .52, .33, and .13, respectively. They suggested that the 1PLM fit provided a satisfactory fit because it is more parsimonious. Using the full sample, the parametric bootstrap triples-adjusted χ²/df ratio p values were .80, .49, and .46 for the 1PLM, 2PLM, and 3PLM, respectively; the parametric bootstrap doubles-adjusted χ²/df ratio p values were .83, .47, and .50 for the 1PLM, 2PLM, and 3PLM, respectively. These statistics show that the 1PLM fit the LSAT 6 data quite well. Importantly, the parametric bootstrap procedure produced results that were similar to that of Joe and Maydeu-Olivares (2006).

General Discussion

Proposing a simple rule of thumb, such as a value of 3, for ascertaining misfit using the mean adjusted χ²/df is helpful but may be misleading in some circumstances. This is because the cutoff value of 3 does not apply across conditions with varying sample sizes and test lengths. This rule of thumb is especially ineffective for smaller sample sizes. In addition, there are important differences in whether this rule is applied to the calibration or validation sample. To overcome the problems with the original adjusted χ²/df ratio, we proposed and tested a bootstrap approach and found that it was useful for ascertaining model–data fit. Specifically, it provided reasonably good control of Type I errors with very high power to identify misspecified models. Currently, we are developing software that implements this approach so that it can be easily used by researchers and practitioners—specifically, users can input item parameters and the number of bootstrap samples. The shell program then implements the routine by calling the relevant IRT software to create an appropriate sampling distribution.

In addition to finding that the performance of the mean adjusted χ²/df can be improved by the bootstrap method, this study provides several other interesting results. First, the singles statistic provides little power and should not be used. Apparently, BILOG (and presumably other IRT estimation software) can fit the marginal numbers of right and numbers of wrong regardless of whether an IRT model fits. In contrast, fitting 2 × 2 and 2 × 2 × 2 contingency tables presents a more challenging task. We found that more restrictive models were unable to fit data from more general models in that the chi-squares that resulted from the two-way and the three-way tables were relatively large. Interestingly, the bootstrap method provided fairly good control of Type I error rates when the correct model was fit and when a more general model was fit with appropriate specifications for the priors. It is arguable whether a Type I error occurs when the 3PLM with an inappropriate prior for c is rejected when fit to 1PLM data: Clearly, there is misspecification.

Drasgow et al. (1995) developed the adjusted χ²/df measure in the context of fitting IRT models to large data sets. In retrospect, it is clear that this measure confounds goodness of fit of an IRT model with estimation error when it is applied in a cross-validation sample. Tables 1 and 2 show that this confound is very serious when sample sizes are 1,500 and less but not a problem for samples of 5,000. In essence, the bootstrap method ascertains the degree to which estimation error affects the χ²/df ratio and thereby provides a basis for judging what constitutes an unexpectedly large ratio.

How does the parametric bootstrap procedure compare with other newer IRT fit statistics? A comparison of the parametric bootstrap adjusted χ²/df ratio shows that it may have higher adjusted power than that of the S-χ² procedure (Orlando & Thissen, 2000). We calculated the adjusted power from Table 1 of Orlando and Thissen’s (2000) article. For a sample size of 1,000 simulees and 40 items, the adjusted power was .52 when fitting the 1PLM to 3PLM data; .47 when fitting the 1PLM to 2PLM data; and .07 when fitting the 2PLM to 3PLM data. In contrast, for a sample size of 1,000 simulees and 30 items, the parametric bootstrap adjusted χ²/df ratio has adjusted powers of .90, .94, and .94, respectively. Adjusted power was even higher for a test length of 45 items.

Future Research

In this article, we focused on the adjusted χ²/df ratio because it has been frequently used even though its properties were poorly understood. Our simulations clearly identified a shortcoming and consequently we proposed and examined an improvement to the adjusted χ²/df ratio. In future research it will be important to carefully examine how the proposed parametric bootstrap approach compares with other IRT fit indices.

We also found that a more general model such as the 3PLM may not fit simulated 1PLM data well. When the c-prior was adjusted lower, the model fit the data substantially better as indicated from the low Type I error rates. Practically, this suggests that overparameterization may be detected with the parametric bootstrap approach.

Conclusion

In sum, evaluating the fit of an IRT model is important for practical applications (e.g., DIF analyses) and theoretical considerations (e.g., determining whether a dominance or ideal point process underlies responses). In this article, we have proposed an improvement on the adjusted χ²/df ratio measure and shown it to function better than the original. Although the parametric bootstrap method requires extensive computations (about 2 hours on a modern personal computer to compare three different IRT models), its qualitatively superior performance appears to justify its use. We expect that in the future, increased computation power would reduce the time needed to obtain a sampling distribution and enable a more extensive bootstrapping process.

Footnotes

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bock

R. D.

Lieberman

(1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.

Bolt

D. M.

(2002). A Monte Carlo comparison of parametric and nonparametric polytomous DIF detection methods. Applied Measurement in Education, 15, 113-141.

Chernyshenko

O. S.

Stark

Chan

K. Y.

Drasgow

Williams

(2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523-562.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11, 59-79.

Drasgow

Levine

M. V.

Tsien

Williams

B. A.

Mead

A. D.

(1995). Fitting polytomous item response models to multiple-choice tests. Applied Psychological Measurement, 19, 145-165.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.

Flowers

C. P.

Oshima

T. C.

Raju

N. S.

(1999). A description and demonstration of the polytomous-DFIT framework. Applied Psychological Measurement, 23, 309-326.

Joe

Maydeu-Olivares

(2006). On the asymptotic distribution of Pearson’s χ² in cross-validation samples. Psychometrika, 71, 587-592.

Kulas

J. T.

Finkelstein

L. M.

(2007). Content and reliability of discrepancy-defined self-awareness in multisource feedback. Organizational Research Methods, 10, 502-522.

10.

LaHuis

D. M.

Clark

O’Brien

(2011). An examination of item response theory item fit indices for the graded response model. Organizational Research Methods, 14, 10-23. doi:10.1177/109442810935093010.1177/1094428109350930

11.

Langeheine

Pannekoek

Van de Pol

(1996). Bootstrapping goodness-of-fit measures in categorical data analysis. Sociological Methods & Research, 24, 492-516.

12.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

13.

Meade

A. W.

Lautenschlager

G. J.

Johnson

E. C.

(2007). A Monte Carlo examination of the sensitivity of the differential functioning of items and tests framework for tests of measurement invariance with Likert data. Applied Psychological Measurement, 31, 430-455.

14.

Orlando

Thissen

(2000). Likelihood-based item fit indices for dichotomous item response theory models. Applied Psychological Measurement, 24, 50-64.

15.

Raju

N. S.

van der Linden

W. J.

Fleer

P. F.

(1995). IRT-based internal measures of differential functioning of items and tests. Applied Psychological Measurement, 19, 353-368.

16.

Rauch

W. A.

Schweizer

Moosbrugger

(2008). An IRT analysis of the personal optimism scale. European Journal of Psychological Assessment, 24, 49-56.

17.

Rivers

D. C.

Meade

A. W.

Fuller

W. L.

(2009). Examining question and context effects in organization survey data using item response theory. Organizational Research Methods, 12, 529-553.

18.

Robie

Zickar

M. J.

Schmit

M. J.

(2001). Measurement equivalence between applicant and incumbent groups: An IRT analysis of personality scales. Human Performance, 14, 187-201.

19.

Scherbaum

C. A.

Cohen-Charash

Kern

M. J.

(2006). Measuring general self-efficacy: A comparison of three measures using item response theory. Educational and Psychological Measurement, 66, 1047-1063.

20.

Schmit

M. J.

Kihm

J. A.

Robie

(2000). Development of a global measure of personality. Personnel Psychology, 53, 153-193.

21.

Stark

Chernyshenko

O. S.

Drasgow

Williams

B. A.

(2006). Examining assumptions about item responding in personality assessment: Should ideal point models be considered for scale development and scoring? Journal of Applied Psychology, 91, 25-39.

22.

St-Onge

Valois

Abdous

Germain

(2009). A Monte Carlo study of the effect of item characteristic curve estimation on the accuracy of the three person-fit statistics. Applied Psychological Measurement, 33, 307-324.

23.

Tatsuoka

K. K.

(1984). Caution indices based on item response theory. Psychometrika, 49, 95-110.

24.

Tay

Ali

U. S.

Drasgow

Williams

B. A.

(2011). Fitting IRT models to dichotomous and polytomous data: Assessing the relative model-data fit of ideal point and dominance models. Applied Psychological Measurement. Advance online publication. doi:10.1177/014662161039067410.1177/0146621610390674

25.

Tay

Drasgow

Rounds

Williams

(2009). Fitting measurement models to vocational interest data: Are dominance models ideal? Journal of Applied Psychology, 94, 1287-1304.

26.

Van den Wollenberg

A. L.

(1982). Two new test statistics for the Rasch model. Psychometrika, 47, 123-140.

27.

Vermunt

J. K.

(2001). The use of restricted latent class models for defining and testing nonparametric and parametric item response theory models. Applied Psychological Measurement, 25, 283-294.

28.

Vermunt

J. K.

Magidson

(2000). Latent GOLD 4.0 user manual. Belmont, MA: Statistical Innovations.

29.

Vermunt

J. K.

Magidson

(2002). Latent class cluster analysis. In Hagenaars

J. A.

McCutcheon

A. L.

(Eds.), Applied latent class analysis (pp. 89-106). Cambridge, England: Cambridge University Press.

30.

Vermunt

J. K.

Magidson

(2005). Technical guide for latent GOLD 4.0: Basic and advanced. Belmont, MA: Statistical Innovations.

31.

von Davier

(1997). Bootstrapping goodness-of-fit statistics for sparse categorical data: Results on a Monte Carlo study. Methods of Psychological Research Online, 2, 29-48.

32.

Williams

B. A.

Levine

M. V.

(1993). FORSCORE: A computer program for nonparametric item response theory. Unpublished manuscript, University of Illinois at Urbana-Champaign, Champaign, IL.

33.

Zhang

Stout

(1999). The theoretical detect index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.

34.

Zickar

M. J.

Robie

(1999). Modeling faking good on personality items: An item-level analysis. Journal of Applied Psychology, 84, 551-563.

35.

Zimowski

M. F.

Muraki

Mislevy

R. J.

Bock

R. D.

(1996). BILOG-MG [Computer software]. Chicago, IL: Scientific Software International.

Adjusting the Adjusted χ 2 / df Ratio Statistic for Dichotomous Item Response Theory Analyses

Abstract

Keywords

The Adjusted χ2/df Ratio: Singles, Doubles, and Triples

Study 1

Simulation

Type I Error and Adjusted Power

Results

1PLM data

2PLM data

3PLM data

Discussion

Study 2

Method

Results

1PLM data

2PLM data

3PLM data

Empirical Example

General Discussion

Future Research

Conclusion

Footnotes

References

The Adjusted χ²/df Ratio: Singles, Doubles, and Triples