Strategies for testing statistical and practical significance in detecting DIF with logistic regression models

Abstract

This study examines three controversial aspects in differential item functioning (DIF) detection by logistic regression (LR) models: first, the relative effectiveness of different analytical strategies for detecting DIF; second, the suitability of the Wald statistic for determining the statistical significance of the parameters of interest; and third, the degree of equivalence between the main DIF classification systems. Different strategies to tests–LR models, and different DIF classification systems, were compared using data obtained from the University of Tehran English Proficiency Test (UTEPT). The data obtained from 400 test takers who hold a master’s degree in science and engineering or humanities were investigated for DIF. The data were also analyzed with the Mantel–Haenszel procedure in order to have an appropriate comparison for detecting uniform DIF. The article provides some guidelines for DIF detection using LR models that can be useful for practitioners in the field of language testing and assessment.

Keywords

DIF classification systems differential item functioning effect size measures logistic regression Mantel–Haenszel procedure small sample size Wald statistic

Many methods for detecting Differential Item Functioning (DIF) have been widely employed in the field of language assessment (see Ferne & Rupp, 2007, for a review). In general, the techniques derived under item response theory (IRT) are usually preferable to other types of analysis, as long as the sample size is large enough and the data fulfill the assumptions required for these models. On the contrary, observed-score methods, such as the Mantel–Haenszel (MH) procedure (Holland & Thayer, 1988) or logistic regression (LR), require fewer assumptions to be met, smaller sample sizes, and less complicated computing than IRT-based analyses. In research on language testing, use of MH (Harding, 2012; Koo, Becker, & Kim, 2013; Ockey, 2007; Pae, 2004, 2012; Roever, 2007; Uiterwijk & Vallen, 2005) is clearly predominant over RL (Alavi, Rezaee, & Amirian, 2011; Kim, 2001; Li & Suen, 2013; Rezaee & Shabani, 2010), even though MH was designed to detect uniform DIF and is not appropriate for detecting non-uniform DIF, especially when it arises in moderately difficult items (Narayanan & Swaminathan, 1996). In this respect, the advantages of LR over MH are that it is capable of simultaneously detecting both uniform and non-uniform DIF, and the ability is treated as a continuous variable without losing stratification power.

One of the goals of this study is to examine the application of LR to DIF detection when sample sizes are small, which is frequently the case in language assessment. Specifically, we used the responses of a random sample of 400 takers of the University of Tehran English Proficiency Test (UTEPT) to investigate three controversial aspects associated with the application of logistic regression (LR) models in DIF detection: (a) usefulness of different detection strategies with respect to DIF type; (b) feasibility of the Wald statistic for checking the statistical significance of DIF with small samples; and (c) extent to which different DIF classification systems are equivalent. These goals are described in detail below.

DIF detection strategies. Almost all simulation studies that compare LR to the MH chi-square ( $χ_{MH}^{2}$ ) have concluded that the $χ_{M H}^{2}$ statistic has a higher power for detecting uniform DIF, and LR for detecting non-uniform DIF (Hidalgo & López-Pina, 2004; Navas-Ara & Gómez-Benito, 2002; Rogers & Swaminathan, 1993; Swaminathan & Rogers, 1990). In the articles cited the differences between one detection method and the other in the case of uniform DIF was never over 10%. However, a recent simulation study has found differences in uniform DIF detection that varied from 59% to 82% more correct detections by the MH procedure (Herrera & Gómez, 2008). Is this possible? The authors attribute the difference between their results and the rest of the literature to the simulation conditions: “the discrepancy between the results may be due to the amount of the DIF, 0.6 and 0.8 in the study by Swaminathan and Rogers (1990), but only 0.4 in the present one” (Herrera & Gómez, 2008, p. 753). In our understanding, the explanation is quite different and has to do with the use of incorrect LR application strategy (described in the following section for reasons of length). The use of that strategy would also explain the results of some empirical studies that report a wide difference in number of items identified between the MH procedure and LR in favor of the MH procedure (Doğan & Öğretmen, 2012). We therefore believe that an analysis that can discern correct strategies from those that, although they may seem plausible, are clearly incorrect, is necessary.

With regard to this point, we hypothesize that, if there is uniform DIF, when the strategy followed by Herrera and Gómez (2008) is used, LR has almost no power compared to MH. On the contrary, when any of the three analytical strategies we consider appropriate is used (described in the following section), results similar to those of MH are expected.

Statistical significance. Some simulation studies have shown that there is no difference between applying the likelihood ratio statistic or the Wald test for testing the statistical significance of the parameters of interest in the LR models (Rogers, 1989). However, this equivalence may not hold true in some situations. The Wald and likelihood ratio tests are asymptotically equivalent; this means that in small samples there might be differences among them (see Paek, 2012). Moreover, the Wald statistic has two major disadvantages compared to the likelihood ratio test. First, it has a tendency to commit Type II errors when logistic regression coefficients are very large. In such situations, the standard error is usually inflated, increasing the probability of concluding that there is no statistical significance when in fact there is a very large effect (Menard, 2002). Second, it is less reliable in small samples (Agresti, 2002). It would, therefore, be important for there to be studies that compare how equivalent the two tests of significance in a realistic situation. We can consider the study on empirical data an instrument allowing the results of the simulation studies to be generalized to new situations. By using common terminology in experimental and quasi-experimental designs, empirical studies, to the extent that they corroborate the results of simulation studies, increase their external validity.

Practical significance. This study compares the application of two of the most popular DIF classification systems using the LR, the Educational Testing Service (ETS) classification system (Monahan, McHorney, Stump, & Perkins, 2007a) and the one proposed by Jodoin and Gierl (2001). Although very little used in the field of education and language testing and assessment, the Crane, Hart, Gibbons, & Cook (2006) criterion was also employed. Finding the degree of equivalence in the different classification systems is a very relevant question, because in case of discrepancy, choosing one or another could make a drastic difference in the results. It should be stressed that to date there are no studies, whether simulation or empirical, that compare the degree of equivalence of these classification systems when sample sizes are small.

The following section gives a brief review of the statistics employed in this study.

Logistic regression

Testing DIF

LR was first proposed for detecting DIF by Swaminathan and Rogers (1990), based on the inclusion of different terms in the regression model that would indicate different types of DIF. It assesses to what extent item scores (1 correct response, 0 incorrect response) can be predicted from total scores alone (model 1), from total scores and group membership (model 2), or from total scores, group membership, and interaction between total scores and group membership (model 3).

l n (\frac{π}{1 - π}) = β_{0} + β_{1} X (m o d e l 1)

(1)

l n (\frac{π}{1 - π}) = β_{0} + β_{1} X + β_{2} G (model 2)

(2)

l n (\frac{π}{1 - π}) = β_{0} + β_{1} X + β_{2} G + β_{3} X G (model 3)

(3)

Where, ln is the natural logarithm, π is the probability of correct response to the studied item $[π \in (0, 1)]$ , X is total test scores used as a measure of overall proficiency, G is a dummy variable representing group membership (typically coded as 1 = reference group, 0 = focal group), XG is the interaction term between proficiency and group membership, and βs are the parameters in the model.

The expression in front of the equal sign is known as a logit and is equal to the natural logarithm of the odds. The model parameters (β₀, …, β₃) are estimated via maximum likelihood method on the log odds or logit scale. The strategy for evaluating the DIF is based on the search for the most parsimonious model that best fits the data. The choice of model usually depends on the existence of statistically significant differences in the likelihood functions of the model that incorporate all of the parameters of interest (full model) and a model whose parameters are a subset of the parameters present in the full model (nested model). The likelihood ratio statistic that is used for this is equal to twice the natural logarithm of the ratio of the likelihood functions of the models compared at their maximum likelihood estimate of the parameters:

Likelihood ratio = - 2 l n (\frac{L (nested model)}{L (full model)})

(4)

This statistic follows a χ² distribution with degrees of freedom equal to the difference in the number of parameters of the models compared.

Several strategies have been proposed in the literature to determine whether the item shows DIF based on Models 1, 2, and 3. First, simultaneously test uniform and non-uniform DIF by comparing the model without DIF (Model 1) with the model that includes terms for both non-uniform and uniform DIF (Model 3). This strategy checks the hypothesis that there is no DIF (H₀ : β₃ = β₂= 0). The null hypothesis of No DIF is tested against any of three alternatives: β₃ ≠ 0 or β₂ ≠ 0 or β₃ ≠ 0 and β₂ ≠ 0. The likelihood ratio statistic would be distributed with 2 degrees of freedom (df). It should be pointed out that the same hypothesis could be checked by applying the Wald statistic, which would allow only one model to fit to the data, Model 3. Under the hypothesis that there is no DIF, the Wald statistic follows a χ² distribution with 2 degrees of freedom. (This is the statistic employed by Swaminathan & Rogers, 1990, in their seminal study.) Second, evaluate non-uniform DIF (H₀ : β₃ = 0) separately by comparing Models 2 and 3, and the uniform DIF ( $H_{0} : β_{2} = 0$ ) by comparing Models 1 and 2 (Jodoin & Gierl, 2001). In both cases, the likelihood ratio statistic would be distributed with 1 df. The above strategy uses model comparison to test the individual statistical significance of parameter $β_{3}$ (for non-uniform DIF) and $β_{2}$ (for uniform DIF). The same thing can be done by evaluating the statistical significance of parameter β₂ (in Model 2) and β₃ (in Model 3) directly by using the Wald test. In this case, the Wald statistic is distributed with 1 degree of freedom. Third, evaluate the non-uniform DIF as in the second strategy, and evaluate uniform DIF by comparing the regression coefficient ${\hat{β}}_{1}$ in Models 1 and 2, and taking a difference $(% Δ {\hat{β}}_{1} = | {\hat{β}}_{1 (model 2)} - {\hat{β}}_{1 (model 1)}) / {\hat{β}}_{1 (model 1)} | * 100)$ of 10% (Crane, van Belle, & Larson, 2004) to 5% (Crane et al., 2006) as an indicator of high DIF. This strategy for evaluating uniform DIF is based on a simulation study of confounder-selection strategies (Maldonado & Greenland, 1993) and has frequently been used in health-related quality of life instrument research (Scott et al., 2010; Teresi, 2006). Finally, a fourth strategy which may be found in the literature consists of fitting Model 3 and evaluating the individual statistical significance of parameter β₂ (uniform DIF) and β₃ (non-uniform DIF) using the Wald test (Herrera & Gómez, 2008) with 1 degree of freedom. Unfortunately, this strategy is completely unadvisable, since it involves a single model to detect both uniform and non-uniform DIF. If the items had uniform DIF, we would be committing a model specification error by including more parameters than necessary. This leads to an increase in the standard error of the parameters estimated, diminishing power (Menard, 2002). In our opinion this would be the correct explanation for the weak power of LR in detecting uniform DIF found by Herrera and Gomez (2008) in his simulations. The use of this analytical strategy with LR could be much more common that might be supposed. Both the Herrera and Gómez (2008) study and the one by Doğan and Öğretmen (2012) used the EZDIF program (Waller, 1998) for the DIF analysis. This program provides regression coefficients corresponding to Model 3 as the output along with its standard errors, test statistics with 1 df, and associated significance levels. That is, that according to our hypothesis, the EZDIF program output does not allow correct evaluation of uniform DIF by LR. If the items show uniform DIF, and the LR output is used to detect it, we find a very low power, which is exactly what the Herrera and Gómez (2008) and Doğan and Öğretmen (2012) studies report. As mentioned above, the problem would not be LR or the program, but the analytical strategy.

Effect size measures

Many researchers have emphasized that an examination of both the statistical significance and a measure of effect size is needed to identify DIF using logistic regression (Jodoin & Gierl, 2001; Monahan et al., 2007a; Zumbo & Thomas, 1997). So using the LR coefficient ${\hat{β}}_{2}$ , or some transformation of it has often been proposed as an estimation of the magnitude of uniform DIF (Clauser & Mazor, 1998; Monahan et al., 2007a; Swanson, Clauser, Case, Nungester, & Featherman, 2002). The ${\hat{β}}_{2}$ coefficient is the same as the ln of the ratio of the odds of an item being right in the reference group to the odds of the item being right in the focal group when they are comparable in ability. Thus positive values show DIF against the focal group and negative values show DIF against the reference group. Using ${\hat{β}}_{2}$ as an estimator of the magnitude of DIF, Monahan et al. (2007a) proposed applying the ETS DIF classification system to LR using the MH procedure. This DIF classification scheme is based on both the statistical significance of ${\hat{β}}_{2}$ , tested using the Wald statistic, and the magnitude of ${\hat{β}}_{2}$ used as an effect size measure. According to this classification,

DIF is negligible (category A) if $β_{2}$ is not significantly different from 0 (p $\geq$ .05) or $| {\hat{β}}_{2} | < 0.426$ .

DIF is moderate (category B) if $β_{2}$ is significantly different from 0 (p < .05) and $| {\hat{β}}_{2} | \geq 0.426$ and either: (a) $| {\hat{β}}_{2} | < 0.638$ , or (b) $| β_{2} |$ is not significantly greater than 0.426 (p $\geq$ .05).

DIF is large (category C) if $| β_{2} |$ is significantly greater than 0.426 (p < .05) and $| {\hat{β}}_{2} | \geq 0.638$ .

The classification presented above is defined on the log-odds scale the LR model parameters are on instead of the Delta scale used in the original ETS proposal and on the formula by Monahan et al. (2007a).

As mentioned above, the Wald statistic is used to determine the statistical significance of $β_{2}$ . When it is used to check individual parameters, such as in this case, the Wald statistic is equal to the quotient of the estimated parameter of interest divided by its estimated standard error:

Wald = \frac{\hat{β}}{σ_{\hat{β}}}

(5)

It follows a standard normal distribution under the hypothesis that β = 0. This statistic can also be calculated as Wald = ${(\hat{β} / σ_{\hat{β}})}^{2}$ in which case, it would follow an asymptotic χ² distribution with 1 df.

If the model which best fits the data is the Model 3, the regression coefficient of interest is ${\hat{β}}_{3}$ . Unfortunately, parameter $β_{3}$ does not admit simple interpretation in terms of direct comparison of the focal and reference groups. Therefore, there are no criteria for interpreting its magnitude as there are for parameter β₂.

This disadvantage may be avoided using an estimator of the magnitude of DIF that provides a similar interpretation of uniform and non-uniform DIF. Among the proposals are measures similar to the coefficient of determination (R²) in linear regression, such as the Cox-Snell’s R² or Nagelkerke’s R² (Jodoin & Gierl, 2001; Zumbo & Thomas, 1997). Using this last statistic, an estimation of the size of DIF may be found by comparing Nagelkerke’s R² of the full model with that of the nested model. Thus non-uniform DIF is equal to the difference in R² between the non-uniform and uniform DIF models: $Δ R_{N}^{2} = R^{2} (model 3) - R^{2} (model 2) .$ And uniform DIF is equal to: $Δ R_{U}^{2} = R^{2} (model 2) - R^{2} (model 1)$ . Jodoin and Gierl (2001) proposed a classification criteria based on the SIBTEST effect size measure as a predictor of Nagelkerke’s R². The guidelines proposed by Jodoin and Gierl (2001) to quantify the magnitude of DIF are as follows:

Type A items – negligible DIF: $Δ$ R2 < 0.035

Type B items – moderate DIF: 0.035 ≤ $Δ R^{2}$ ≤ 0.070

Type C items – large DIF: $Δ R^{2}$ > 0.070

Following the criteria of Jodoin and Gierl (2001), an item is considered to have DIF if the probability of either 1-df test was less than .05, and the corresponding $Δ R^{2}$ ≥ .035.

Mantel–Haenszel procedure

The MH procedure was applied to data in order to have a reference point with which to compare detection of uniform DIF using the LR. To apply the MH procedure, the information with the test taker answers to the item must be arranged in 2 × 2 Q contingency tables, where Q is the number of intervals in which the total test score is divided (h =1,…, Q). Thus there is a 2 × 2 contingency table for each h score level, with the group (reference/focal) as one of the entries and the answer to the item (right/wrong) as the other, as observed in Table 1. The values in Cells A_h, B_h, C_h, and D_h denote the number of test takers in each category, and N_h is the total number of test takers on h score level (N_h = A_h + B_h + C_h + D_h ).

Table 1.

The 2 × 2 contingency table for the hth score level.

Group	Right (1)	Wrong (0)
Reference	A_h	B_h
Focal	C_h	D_h

The MH procedure has an estimator of the magnitude of the DIF (the Mantel–Haenszel common log odds ratio estimator, ${\hat{λ}}_{M H}$ ), which, as shown below, provides the same type of information as the ${\hat{β}}_{2}$ in LR (Model 2). Taking the notation in Table 1 and assuming population data, it may be established that if the variables group and item response are independent, the population odds of the probability of responding correctly to the item (π) instead of incorrectly (1−π) will be equal in the reference group (R) and the focal group (F). That is

\frac{π_{R}}{1 - π_{R}} = \frac{π_{F}}{1 - π_{F}}, which is equivalent to \frac{A}{B} = \frac{C}{D}

(6)

The above equality may be expressed as a quotient such that the ratio of the odds, referred to as the odds ratio {α = (A/B)/(C/D)} or cross-product odds ratio {α = (A•D)/(C•B)} will be 1. Assuming homogeneity of the odds ratios of each stratum (α₁ = · · · = α_h = · · · = α_Q), the MH measure of association calculated across all 2 × 2 contingency tables is the common odds ratio estimator ( ${\hat{α}}_{M H}$ ), given by

{\hat{α}}_{M H} = \frac{\sum_{h = 1}^{Q} A_{h} D_{h} / N_{h}}{\sum_{h = 1}^{Q} B_{h} C_{h} / N_{h}}, N_{h} = A_{h} + B_{h} + C_{h} + D_{h}

(7)

${\hat{α}}_{M H},$ is an estimator of the magnitude of DIF in a metric which varies from 0 to ∞, where independence of rows and columns (No DIF) shows a value of 1. Because of the skewness of the distribution of ${\hat{α}}_{M H},$ it is more convenient to use the natural logarithm of ${\hat{α}}_{M H}$ $[{\hat{λ}}_{M H} = l n ({\hat{α}}_{M H})]$ . In this case, the absence of uniform DIF corresponds to ${\hat{λ}}_{M H} = 0 .$ Positive values show DIF against the focal group and negative values show DIF against the reference group.

The MH procedure, in addition to providing an estimator of the magnitude of DIF, provides a test of statistical significance. Given that ${\hat{λ}}_{M H}$ follows a normal asymptotic distribution it is possible to use a standard Wald statistic¹ to check the null hypothesis of no DIF:

z = \frac{{\hat{λ}}_{M H} - λ_{0}}{σ_{{\hat{λ}}_{M H}}},

(8)

where $λ_{0}$ is the hypothesized value of the parameter, $σ_{{\hat{λ}}_{M H}}$ denotes the standard error of ${\hat{λ}}_{M H}$ (see Phillips & Holland, 1987 for calculation) and z follows a normal distribution. The criteria for assessing the severity of DIF are the same as for, because ${\hat{β}}_{2}$ and ${\hat{λ}}_{M H}$ estimate the same type of information, although they use different methods of estimation, ${\hat{β}}_{2}$ by maximum likelihood, and ${\hat{β}}_{2}$ by the ln in Equation 7. Therefore, to evaluate DIF by the MH procedure, the ETS DIF classification system described above was applied.

Objectives

Using data acquired from 400 University of Tehran English Proficiency Test takers, this study intended to answer the following questions.

Regardless of sample size, what DIF detection strategies are appropriate for detecting uniform and non-uniform DIF?

Does it make any difference whether LR or the Wald test is used to determine the statistical significance of DIF when the sample is small?

How much equivalence is there between the ETS, Crane et al. (2006), and Jodoin and Gierl (2001) DIF classification systems when the sample is small?

Method

Instrumentation

University of Tehran English Proficiency Test (UTEPT) is a high-stakes test of English developed and administered by English department of University of Tehran. It is a prerequisite for master’s degree holders aiming at participating in PhD exams of University of Tehran. It consists of 100 items with a time limit of 100 minutes. A more detailed description of this test may be found in Rezaee and Shabani (2010). It should be noted that the purpose of this study is not to analyze the test “per se”, but to analyze LR applied to DIF detection. Before proceeding to the analysis of DIF, analyses were done to find out the psychometric properties of the items, as well as the dimensionality and reliability of the scale as a whole. The original scale showed a Cronbach’s alpha reliability coefficient of 0.89, but many of its items had very low discrimination indices (estimated by the corrected item–total point biserial correlation). Items with a discrimination index below 0.20 were eliminated from the analyses. The resulting scale is comprised of 75 items and has a Cronbach’s alpha of 0.90. It is this reduced scale on which the analyses were performed. Two conditional covariance estimation-based procedures were used as follows: (a) to test the null hypothesis that the test data satisfies a unidimensional model (DIMTEST; Stout, 1987; Stout et al., 2001), and (b) to estimate an effect size for the multidimensionality (DETECT; Kim, 1994; Zhang & Stout, 1999). DIMTEST results (T = 1.7693, p = .0384) indicated that the test is essentially unidimensional. The maximum DETECT value (0.3055) indicated that the test showed weak multidimensionality (Monahan, Stump, Finch, & Hambleton, 2007b).

Participants

Most of the groups compared in studies on DIF are defined based on race/ethnicity, gender or other demographic variables, but very few employ academic background (Pae, 2004). Previous studies on DIF in the UTEPT were based on gender (Rezaee & Shabani, 2010) and academic background (Alavi, Rezaee, & Amirian, 2011). As pointed out above, one of the purposes of this study was to examine the behavior of LR with small samples. Since the study by Alavi et al. (2011) and this one share the same size of sample, it was decided to also select the same variable for forming the groups to facilitate the extrapolation of results. The Rezaee and Shabani (2010) study was done with a much larger sample (N = 6.555).

The participants of the present study were 400 test takers randomly selected from the 1600 examinees who took the University of Tehran English Proficiency Test (UTEPT) in November 2010. Passing UTEPT is a prerequisite for them to be allowed to sit for the PhD exam of the University of Tehran. Based on academic background, the sample was divided into a reference group of 200 participants with science and engineering background and a focal group of 200 participants with humanities backgrounds. The science and engineering group comprised students of chemistry, physics, mathematics, biology, mechanical engineering, electrical engineering, and civil engineering and the humanities group consisted of students of social sciences, law, political sciences, management, Persian literature, and foreign languages. There are both male and female test takers in the sample. A sample size of 200 test takers per group was chosen because some studies identified this size as the minimum for adequate power in MH and LR procedures (Güler & Penfield, 2009; Mazor, Clauser, & Hambleton, 1992; Paek & Wilson, 2011; Rogers & Swaminathan, 1993). The power found in simulation studies with smaller sample sizes (50 or 100 test takers per group) is only moderately acceptable when the magnitude of the DIF is high (Fidalgo, Ferreres, & Muñiz, 2004; Fidalgo, Hashimoto, Bartram, & Muñiz, 2007). Many simulation studies show that differences in ability among the groups measured by the test, which is technically called impact, increase Type I error rates in both LR and MH (DeMars, 2009; French & Maller, 2007; Jodoin & Gierl, 2001; Li et al., 2012; Pei & Li, 2010). To find out whether our analyses could be affected by this variable, an independent-samples t-test was conducted to compare UTEPT scores between groups. There was no significant difference in the total test scores for the science and engineering (M = 45.14, SD =11.03) and humanities (M = 42.51, SD = 12.62) groups at α = .01 (t = 2.218, df = 398, p = .027). The coefficient of determination $(R^{2} = \frac{t^{2}}{(t^{2} + d f)}),$ which was calculated based on the results of the t-test, showed that belonging to the group explained only 1.22% of the variability in the test scores. These results suggest that group membership really does not have an effect on test scores.

Analysis

Logistic regression

To use LR for DIF analysis, Models 1, 2, and 3 were fit to the data using SPSS (version 18). The logistic regression procedure uses the item response (1 for a correct response, and 0 for an incorrect response) as the dependent variable, with grouping variable (dummy coded as 1 = Science and engineering/reference group; 0 = Humanities/focal group), total scale score for each subject, and group by total score interaction, as independent variables.

Logistic regression was applied in two stages because of its advantages with regard to Type I and Type II errors (French & Maller, 2007; Navas-Ara & Gómez-Benito, 2002). In the first stage, uniform and non-uniform DIF were evaluated simultaneously using the Likelihood ratio test of Model 3 vs. Model 1 at a .05 alpha level. This detection strategy was chosen for the first stage because it maximizes the probability of identifying both uniform and non-uniform DIF and controls the overall Type I error rate. In the second stage, the test score for each examinee is refined by removing items that were found to show DIF in the first stage. Using the total refined score as the estimate of ability, the following tests were calculated for each item to determine whether there was DIF:

T1 = Likelihood ratio test of Model 3 vs. Model 1 (Test of Uniform and/or Non-Uniform DIF).

T2 = Likelihood ratio test of Model 2 vs. Model 1 (Test of Uniform DIF).

T3 = Wald test of β₂ in Model 2 (Test of Uniform DIF).

T4 = Wald test of β₂ in Model 3 (Test of Uniform DIF).

T5 = Likelihood ratio test of Model 3 vs. Model 2 (Test of Non-Uniform DIF).

T6 = Wald test of β₃ in Model 3 (Test of Non-Uniform DIF).

${\hat{β}}_{2}$ , $Δ R_{U}^{2}, and Δ {\hat{β}}_{1}$ were used as size effect measures of uniform DIF, and as the measure of non-uniform DIF Δ $R_{N}^{2}$ Since LR, like MH, uses the observed score on the total test as an estimate of ability, when an item is being investigated it is included in the matching criterion, even though it showed DIF in the initial analysis (Holland & Thayer, 1988; Tan et al., 2010; Zwick, 1990).

Mantel–Haenszel procedure

Scale purification was conducted in the same manner as LR, that is, the MH statistics were computed in two stages. When an item is being investigated it is included in the matching criterion, though it displayed DIF in the initial analysis (Holland & Thayer, 1988; Tan et al., 2010). The level of significance used in all analyses was .05.

Results

All the data analyzed in this section are part of the second stage results. That is, they were found by applying the statistics employing the purified test score.

Table 2 presents the results of using LR analysis to determine the statistical significance (α = .05) of the parameters of interest. Out of a total of 75 items, 9 items were flagged as DIF items based on the 2-df likelihood ratio test (T1). The number of detections increased notably when the statistical significance of the parameters of interest was tested individually using the 1-df likelihood ratio tests (T2 and T5). Out of a total of 75 items, 18 items were flagged as DIF, of which 13 correspond to uniform DIF, four to non-uniform DIF, and one item (item 24) showed statistical significance in both parameters ( $β_{2}$ and $β_{3}$ ). Furthermore, as observed in Table 2, these results are practically identical to those found using the appropriate Wald test (T3 and T6). As was hypothesized, no item with uniform DIF could be identified when fitting Model 3 to determine the statistical significance of uniform DIF (parameter β₂) using the Wald test (T4).

Table 2.

Item classified as exhibiting uniform and non-uniform DIF by type of test (α = .05).

Item	T1	T2	T3	T4	T5	T6
Uniform DIF
6	.023	.008	.009	–	–	–
17	–	.047	.048	–	–	–
26	–	.044	.047	–	–	–
33	.009	.012	.012	–	–	–
38	–	.039	.041	–	–	–
39	–	.029	.029	–	–	–
40	.021	.007	.008	–	–	–
52	.014	.008	.009	–	–	–
55	.006	.003	.004	–	–	–
71	–	.040	.041	–	–	–
79	–	.031	.032	–	–	–
86	–	.037	.037	–	–	–
92	.035	.021	.023	–	–	–
Non-uniform DIF
1	.018	–	–	.010	.026	.029
24	.001	.007	.008	.002	.008	.010
25	–	–	–	–	.046	.049
29	.013	–	–	.006	.003	.004
69	–	–	–	.022	.018	.020

T1 = Likelihood ratio test of Model 3 vs. Model 1 (Test of Uniform and/or Non-Uniform DIF).

T2 = Likelihood ratio test of Model 2 vs. Model 1 (Test of Uniform DIF).

T3 = Wald test of β₂ in Model 2 (Test of Uniform DIF).

T4 = Wald test of β₂ in Model 3 (Test of Uniform DIF).

T5 = Likelihood ratio test of Model 3 vs. Model 2 (Test of Non-Uniform DIF).

T6 = Wald test of β₃ in Model 3 (Test of Non-Uniform DIF).

Table 3 shows the size effect measures proposed by Jodoin and Gierl (2001) and Crane et al. (2006). Comparing the values corresponding to columns $Δ R_{N}^{2}$ and Δ $R_{U}^{2}$ it may be observed that DIF is predominantly uniform, coinciding with the results found when checking the statistical significance of parameters β₂ and β₃ (see Table 2): 13 items are much higher in Δ $R_{U}^{2}$ than in Δ $R_{N}^{2}$ , four items are the opposite, and one item is similar in both statistics. Table 4 shows both the statistical significance tests and the effect size measures derived from the ETS DIF classification system for the Mantel–Haenszel and the LR methods.

Table 3.

Results of the Jodoin–Gierl DIF classification criterion (Nagelkerke R²) and the Crane et al. (2006) criterion. According to the criteria of Jodoin and Gierl, the only item that would show moderate DIF is Item 24 (Category B).

Item	Nagelkerke R²
	$R_{N}^{2}$	Δ $R_{U}^{2}$	$R_{T}^{2}$	$% Δ \hat{β_{1}}$
Uniform DIF
6	.002	.022	.024	8.4*
17	.000	.011	.011	1.5
26	.000	.018	.018	9.4*
33	.009	.019	.028	4.4
38	.005	.017	.022	10.0*
39	.000	.016	.016	5.3*
40	.002	.024	.026	9.6*
52	.004	.022	.026	8.6*
55	.005	.026	.030	7.6*
71	.001	.015	.016	3.7
79	.001	.014	.015	6.4*
86	.002	.013	.015	3.6
92	.004	.015	.019	3.8
Non-uniform DIF
1	.017	.011	.028	6.6*
24	.022	.024	.046 (B)	10.5*
25	.012	.000	.012	0.6
29	.027	.000	.027	0.4
69	.019	.000	.019	0.5

$Δ R_{T}^{2} = Δ R_{N}^{2} + Δ R_{U}^{2}$ .

Items identified according to the Crane et al. (2006) criterion: $% Δ {\hat{β}}_{1}$ > 5.

Table 4.

Results of the ETS DIF classification system for the Mantel–Haenszel and the LR methods (uniform DIF).

Item	Mantel–Haenszel				LR
	$p (λ_{M H} = 0)$	$p (\| λ_{M H} \| \leq 0.426)$	${\hat{λ}}_{(M H}$ ₎	ETS	$p (β_{2} = 0)$	$p (\| β_{2} \| \leq 0.426)$	${\hat{β}}_{2}$	ETS
6	.022	.334	−0.524	B	.009	.229	−0.592	B
17	.037	.393	0.489	B	.048	.462	0.477	B
24	.005	.165	−0.654	B	.008	.202	−0.621	B
26	–	–	−0.516	A	.047	.226	−0.685	B
33	.026	.372	0.499	B	.012	.295	0.542	B
38	–	–	−0.439	A	.041	.316	−0.556	B
39	.047	.401	0.487	B	.029	.331	0.532	B
40	.009	.146	−0.715	B	.009	.153	−0.696	B
52	.013	.163	−0.702	B	.009	.149	−0.706	B
55	.001	.085	−0.737	B	.004	.168	−0.636	B
71	–	–	−0.362	A	.041	.341	−0.533	B
79	.045	.430	−0.467	B	.032	.410	−0.476	B
86	–	–	0.512	A	.037	.332	0.538	B
92	–	–	−0.469	A	.023	.314	−0.541	B

Note: The items identified only by LR are in gray. The rest of the items have been identified by both procedures.

A correlational analysis of the estimators shows how all the DIF measures are highly related. Spearman rank correlations were found in a range of 0.89 for the correlation between Δ $R_{U}^{2}$ and $| {\hat{λ}}_{M H} |$ , and 0.99 corresponding to the correlation between Δ $R_{U}^{2}$ and $| {\hat{β}}_{2} |$ . In spite of this, there is a noticeable divergence between the results of applying the Jodoin and Gierl (2001) DIF classification criterion and the rest of the criteria. In the case of uniform DIF, no item would be identified according to the Jodoin and Gierl criterion (see Table 3). However, if after fitting Model 2, the ETS classification system is applied using the Wald test (see Table 4), the number of items with moderate DIF identified comes to 14. If the same classification system is applied, but this time using the MH statistic, nine items are classified in Category B. Employing the $% Δ {\hat{β}}_{1}$ statistic with a 5% criterion, 10 items were identified, of which nine were identified when the ETS criterion was used in the LR. On the other hand, the Jodoin and Gierl (2001) criterion is the only one that enables incorporation of an effect size measure when detecting non-uniform DIF. Taking both uniform and non-uniform DIF into consideration, only item 24 would be identified, since it is the only one high enough (Δ $R_{T}^{2}$ > 0.35) in the Nagelkerke R² statistic (see Table 3).

Discussion

This study examined three controversial points concerning use of LR as a DIF detection method: (a) the DIF detection strategy chosen; (b) the type of statistic employed to determine statistical significance; and (c) the extent of equivalence of different DIF classification systems.

Concerning the first point, the Introduction described the most common analytical strategies in the literature and pointed out the risk of some of them being incorrect although they may seem plausible. Moreover, it was hypothesized that the inability of LR to detect uniform DIF in Herrera and Gómez’s (2008) simulation study is the product of a model specification error. The results support the plausibility of this hypothesis, showing that LR has detection rates similar to MH, as long as a proper analysis strategy is employed (see Tables 2 and 4). Since this is a study using real data and neither the magnitude nor the type of DIF in the items is known, the degree of certainty of our conclusion is limited. Therefore, an additional short Monte Carlo study was performed to clarify that point. The largest sample size used by Herrera and Gómez (2008) (1500 examinees per group) was chosen to achieve statistical power that would demonstrate the differences between strategies more clearly. The IRT parameters reported in the Herrera and Gómez study were employed for the 12 items with DIF. Likewise, the remaining 88 items without DIF fit to the following distributions: b≈ N(0, 1), a≈ N(0.5, 0.2), and c = 0.2 for all items. The ability of the reference and focal group was normally distributed, with mean 0 and standard deviation 1 N(0, 1). Data (20 replications) were generated employing the WinGen program (Han, 2007). Table 5 shows the results of the second stage of calculation of the statistics. The main difference between the results of the first and second stages is the substantial reduction in Type I error rate as a consequence of using the purified test score (second stage).

Table 5.

Power and Type I error rate (α = .05). in the MonteCarlo study (N= 3000; 20 replications).

	T1	T2	T3	MH	T4	T5	T6
Type I error rate Power	.045	.045	.046	.045	.051	.051	.051
Uniform DIF	1.000	1.000	1.000	1.000	.125	.263	.263
Non-uniform DIF	.988	.400	.400	.438	.950	.975	.975
Mixed DIF	1.000	1.000	1.000	1.000	.650	.338	.338
Total Power	.996	.800	.800	.813	.575	.525	.525

T1 = Likelihood ratio test of Model 3 vs. Model 1 (Test of Uniform and/or Non-Uniform DIF).

T2 = Likelihood ratio test of Model 2 vs. Model 1 (Test of Uniform DIF).

T3 = Wald test of β₂ in Model 2 (Test of Uniform DIF).

MH = Mantel–Haenszel test.

T4 = Wald test of β₂ in Model 3 (Test of Uniform DIF).

T5 = Likelihood ratio test of Model 3 vs. Model 2 (Test of Non-Uniform DIF).

T6 = Wald test of β₃ in Model 3 (Test of Non-Uniform DIF).

As may be observed in Table 5, when the analytical strategy followed by Herrera and Gómez (2008) is applied, the power of LR for detecting uniform DIF is practically null. However, when an appropriate analytical strategy was used, LR shows uniform DIF detection rates quite similar to those of MH (see statistics T1, T2 and T3) and very much higher than MH in the detection of non-uniform DIF (see statistics T1, T5 and T6). Consequently, check the no uniform DIF hypothesis by fitting Model 3 and applying the Wald (or Likelihood ratio) statistics with 1 df limits the power of this statistic for detecting uniform DIF. This strategy, inadequate from a theoretical viewpoint, has been included as an alternative analysis in other simulation studies (Hidalgo-Montesinos, Gómez-Benito, & Padilla-García, 2005), and in empirical studies (Doğan & Öğretmen, 2012), with results similar to those of Herrera and Gómez (2008).

From an applied viewpoint, the first conclusion that can be arrived at from this study is that performing a differentiated analysis of uniform DIF and non-uniform DIF always involves fitting more than a regression model. The regression models that would have to be fit based on the hypothesis checked and the type of statistic chosen, the likelihood ratio or Wald, are described below.

The second goal refers to the degree of equivalence between the likelihood ratio and the Wald statistic when samples are small. Even though likelihood ratio is theoretically preferable to the Wald statistic (Agresti, 2002; Menard, 2002), our results show practically identical behavior of the two statistics, corroborating the findings of the simulation study by Rogers (1989) which was done with 250 and 500 examinees per group.

The third point refers to the degree of equivalence of the Jodoin and Gierl (2001) DIF classification criterion, Crane et al. (2004) criterion, and ETS classification system (Monahan et al., 2007a). The results show that when samples are small, as they are in this study, there is wide discrepancy between the results of the Jodoin and Gierl (2001) classification system, which only classified one item with DIF, and the other criteria. Thus using the criteria established by Jodoin and Gierl (2001) for Nagelkerke’s R² statistic, only one of the 18 statistically significant items would be classified as having DIF; the Crane criteria identified 9 items, and the ETS classification found 14 items in the category of moderate DIF.

These data have important implications in the field of language assessment. For example, and limiting it to research on DIF with the UTEP, the results of the previous Alavi et al. (2011) study, in which only one item with DIF was identified, should be taken with caution, because the sample size was small and the only criteria of DIF classification employed were based on Nagelkerke’s R². In a more general manner, we can conclude that, in absence of simulation studies that determine the effectiveness of the various classification systems with small samples, more than one DIF detection method should always be applied, as recommended by Hambleton (2006).

Implications and future research

The results of this study make it possible to make a series of recommendations for using LR in DIF research. First, detection of the type of DIF requires adjusting differentiated models. The results show that an error in model specification has drastic results for the statistical power of the LR for DIF detection. Therefore, we warn against the naive idea of establishing equivalence between the type of DIF and the type of parameter, without considering the adjusted model. Second, with a small sample size of only 400 test takers, no difference was found between the Wald statistic and the LR test. Therefore, the Wald statistic may be used to determine statistical significance, with the consequent simplification in terms of adjusting models (see Table 6). Third, when sample sizes are small, more than one DIF detection technique must be applied, owing to the wide divergence in results found among DIF classification systems. Furthermore, the Jodoin criterion, compared to the ETS classification system, seems extremely conservative and simulation studies should be useful to determine their behaviour with small sample sizes and derive more accurate cutoff points.

Table 6.

Logistic regression models that would have to be fit depending on the hypothesis being checked and the type of statistic chosen. The degrees of freedom (df) of each statistic are also shown.

Hypotheses		Likelihood ratio		Wald
		Models	df	Models	df
No DIF	$H_{0} : β_{3} = β_{2} = 0$	M3 & M1	2	M3	2
No non-uniform DIF	$H_{0} : β_{3} = 0$	M3 & M2	1	M3	1
No uniform DIF	$H_{0} : β_{2} = 0$	M2 & M1	1	M2	1

Footnotes

Acknowledgements

We appreciate the reviewers’ very detailed analysis of the manuscript and all the suggestions received. It enabled us to improve the results substantially.

Funding

This work was supported by the Spanish Ministry of Economy and Competitiveness (grant number PSI2009-08529).

Notes

References

Agresti

(2002). Categorical data analysis (2nd ed). New York: Wiley.

Alavi

S. M.

Rezaee

A. A.

Amirian

S. M.

(2011). Academic Discipline DIF in an English Language Proficiency Test. Journal of English Language Teaching and Learning, 7, 39–65.

Camilli

Shepard

L. A.

(1994). Methods for identifying biased test items. Thousand Oaks, CA: SAGE Publications.

Clauser

B. E.

Mazor

K. M.

(1998). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31–44.

Crane

P. K.

Hart

D. L.

Gibbons

L. E.

Cook

K. F.

(2006). A 37-item shoulder functional status item pool had negligible differential item functioning. Journal of Clinical Epidemiology, 59, 478–484.

Crane

P. K.

van Belle

Larson

E. B.

(2004). Test bias in a cognitive test: Differential item functioning in the CASI. Statistics in Medicine, 23, 241–256.

DeMars

C. E.

(2009). Modification of the Mantel–Haenszel and Logistic Regression DIF procedures to incorporate the SIBTEST regression correction. Journal of Educational and Behavioral Statistics, 34, 149–170.

Doğan

Öğretmen

(2012). The comparison of Mantel–Haenszel, chi-square and logistic regression techniques for identifying differential item functioning. Education and Science, 33, 100–112.

Ferne

Rupp

A. A.

(2007). A synthesis of 15 years of research on DIF in language testing: Methodological advances, challenges and recommendations. Language Assessment Quarterly, 4(2), 113–148.

10.

Fidalgo

A. M.

Ferreres

Muniz

(2004). Utility of the Mantel–Haenszel procedure for detecting differential item functioning in small samples. Educational and Psychological Measurement, 64, 925–936.

11.

Fidalgo

A. M.

Hashimoto

Bartram

Muñiz

(2007). Application of an empirical Bayes enhancement of the Mantel–Haenszel procedure for detecting DIF under small-sample conditions. The Journal of Experimental Education, 75, 293–314.

12.

French

B. F.

Maller

S. J.

(2007). Iterative purification and effect size use with Logistic Regression for differential item functioning detection. Educational and Psychological Measurement, 67, 373–393.

13.

Güler

Penfield

R. D.

(2009). A comparison of logistic regression and contingency table methods for simultaneous detection of uniform and non-uniform DIF. Journal of Educational Measurement, 46, 314–329.

14.

Hambleton

R. K.

(2006). Good practices for identifying differential item functioning. Medical Care, 44(11), 182–188.

15.

Han

K. T.

(2007). WinGen: Windows software that generates IRT parameters and item responses. Applied Psychological Measurement, 31(5), 457–459.

16.

Harding

(2012). Accent, listening assessment and the potential for a shared-L1 advantage: A DIF perspective. Language Testing, 29, 163–180.

17.

Herrera

Gómez

(2008). Influence of equal or unequal comparison group simple sizes on the detection of differential item functioning using the Mantel–Haenszel and logistic regression techniques. Quality & Quantity, 42, 739–755.

18.

Hidalgo

M. D.

López-Pina

J. A.

(2004). Differential item functioning detection and effect size: A comparison between logistic regression and Mantel–Haenszel procedures. Educational and Psychological Measurement, 64, 903–915.

19.

Hidalgo-Montesinos

M. D.

Gómez-Benito

Padilla-García

J. L.

(2005). Regresión logística: Alternativas de análisis en la detección del funcionamiento diferencial del ítem (Logistic regression: analytic strategies in differential item functioning detection). Psicothema, 17, 509–515.

20.

Holland

P. W.

Thayer

D. T.

(1988). Differential item functioning detection and the Mantel–Haenszel procedure. In Wainer

Braun

H. I.

(Eds.), Test validity (pp.129–145). Hillsdale, NJ: Lawrence Erlbaum.

21.

Jodoin

M. C.

Gierl

M. J.

(2001). Evaluating type I error and power rates using an effect size measure with logistic regression procedure for DIF detection. Applied Measurement in Education, 14, 329–349.

22.

Kim

H. R.

(1994). New techniques for the dimensionality assessment of standardized test data. Doctoral dissertation. University of Illinois at Urbana-Champaign, Department of Statistics.

23.

Kim

(2001). Detecting DIF across the different language groups in a speaking test. Language Testing, 18, 89–114.

24.

Koo

Becker

B. J.

Kim

(2013). Examining differential item functioning trends for English language learners in a reading test: A meta-analytical approach. Language Testing, DOI: 0265532213496097.

25.

Brooks

G. P.

Johanson

G. A.

(2012). Item discrimination and Type I error in the detection of differential item functioning. Educational and Psychological Measurement, 72, 847–861.

26.

Suen

H. K

(2013). Detecting native language group differences at the subskills level of reading: A differential skill functioning approach. Language Testing, 30, 273–298.

27.

Maldonado

Greenland

(1993). Simulation study of confounder-selection strategies. American Journal of Epidemiology, 138, 923–936.

28.

Mazor

Clauser

B. E.

Hambleton

R. K.

(1992). The effect of sample size on the functioning of the Mantel–Haenszel statistic. Educational Psychological Measurement, 52, 443–452.

29.

Menard

(2002). Applied logistic regression analysis (2nd ed.). Series: Quantitative Applications in the Social Sciences, No. 106. Thousand Oaks, CA: SAGE Publications.

30.

Monahan

P. O.

McHorney

C. A.

Stump

T. E.

Perkins

A. J.

(2007a). Odds ratio, Delta, ETS classification, and standardization measures of DIF magnitude for binary logistic regression. Journal of Educational and Behavioral Statistics, 32, 92–109.

31.

Monahan

P. O.

Stump

T. E.

Finch

Hambleton

R. K.

(2007b). Bias of exploratory and cross-validated DETECT index under unidimensionality. Applied Psychological Measurement, 31, 483–503.

32.

Narayanan

Swaminathan

(1996). Identification of items that show nonuniform DIF. Applied Psychological Measurement, 20(3), 257–274.

33.

Navas-Ara

M. J.

Gómez-Benito

(2002). Effects of ability scale purification on the identification of dif. European Journal of Psychological Assessment, 18, 9–15.

34.

Ockey

G. J.

(2007). Investigating the validity of math word problems for English language learners with DIF. Language Assessment Quarterly, 4, 149–164.

35.

Pae

(2004). DIF for examinees with different academic backgrounds. Language Testing, 21, 53–73.

36.

Pae

(2012). Causes of gender DIF on an EFL language test: A multiple-data analysis over nine years. Language Testing, 29, 533–554.

37.

Paek

(2012). A note on three statistical tests in the logistic regression DIF procedure. Journal of Educational Measurement, 49, 121–126.

38.

Paek

Wilson

(2011). Formulating the Rasch differential item functioning model under the Marginal Maximum Likelihood estimation context and its comparison with Mantel–Haenszel procedure in short test and small sample conditions. Educational and Psychological Measurement, 71, 1023–1046.

39.

Pei

L. K.

(2010). Effects of unequal ability variances on the performance of Logistic Regression, Mantel–Haenszel, SIBTEST IRT, and IRT Likelihood Ratio for DIF detection. Applied Psychological Measurement, 34, 453–456.

40.

Phillips

Holland

P. W.

(1987). Estimators of the variance of the Mantel–Haenszel log-odds-ratio estimate. Biometrics, 43, 425–431.

41.

Rezaee

Shabani

(2010). Gender differential item functioning analysis of the University of Tehran English Proficiency Test. Pazhuhesh-e Zabanha-ye Khareji, 56, 89–108.

42.

Roever

(2007). DIF in the Assessment of second language pragmatics. Language Assessment Quarterly, 4, 165–189.

43.

Rogers

H. J.

(1989). A logistic regression procedure for detecting item bias. Dissertation Abstracts International, 50, 3928A. (University Microfilms No. 90–11,788).

44.

Rogers

H. J.

Swaminathan

(1993). A comparison of logistic regression and Mantel–Haenszel procedures for detecting differential item functioning. Applied Psychological Measurement, 17, 105–116.

45.

Scott

N. W.

Fayers

P. M.

Aaronson

N. K.

Bottomley

de Graeff

Groenvold

Gundy

Koller

Petersen

M. A.

Sprangers

M. A. G.

(2010). Differential item functioning (DIF) analyses of health-related quality of life instruments using logistic regression. Health and Quality of Life Outcomes, 8, 81.

46.

SPSS Incorporated (2009). SPSS 18. Chicago, IL: SPSS, Inc.

47.

Stout

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrica, 52, 589–617.

48.

Stout

Froelich

A. G.

Gao

(2001). Using resampling methods to produce an improved DIMTEST procedure. In Boomsma

van Duijin

M.A.J

Snijders

T.A.B.

(Eds.). Essays on ítem response theory. New York: Springer-Verlag.

49.

Swaminathan

Rogers

H. J.

( 1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27, 361–370.

50.

Swanson

D. B.

Clauser

B. E.

Case

S. M.

Nungester

R. J.

Featherman

(2002). Analysis of differential item functioning (DIF) using hierarchical logistic regression models. Journal of Educational and Behavioral Statistics, 27, 53–75.

51.

Tan

Xiang

Dorans

N. J.

(2010). The value of the studied item in the matching criterion in differential item functioning (DIF) analysis (ETS RR-10–13). Princeton, NJ: Educational Testing Service.

52.

Teresi

J. A.

(2006). Overview of quantitative measurement methods. Equivalence, invariance, and differential item functioning in health applications. Medical Care, 44, 39–49.

53.

Uiterwijk

Vallen

(2005). Linguistic sources of item bias for second generation immigrants in Dutch tests. Language Testing, 22, 211–234.

54.

Waller

N. G.

(1998). EZDIF: A computer program for detecting uniform and non-uniform differential item functioning with the Mantel–Haenszel and logistic regression procedures. Applied Psychological Measurement, 22, 391.

55.

Zhang

Stout

(1999). The theoretical Detect index of dimensionality and its application to approximate simple structure. Psychometrika, 64(2), 213–249.

56.

Zumbo

B. D.

Thomas

D. R.

(1997). A measure of effect size for a model-based approach for studying DIF (Working paper of the Edgeworth Laboratory for Quantitative Behavioral Sciences). Prince George, Canada: University of Northern British Columbia.

57.

Zwick

(1990). When do item response function and Mantel–Haenszel definitions of differential item functioning coincide? Journal of Educational Statistics, 15, 185–197.