Odds Ratio,Delta,ETS Classification,and Standardization Measures of DIF Magnitude for Binary Logistic Regression

The Logistic Regression (LR) Procedure for DIF Detection

In the binary LR model, the probability of endorsing a dichotomously scored item is

and the log odds (or logit) of endorsing the item is modeled as

where ln is the natural logarithm, x is a measure of overall proficiency (usually total score), g is a dummy variable representing group membership (traditionally, 1 = reference group, 0 = focal group), xg is the interaction term between total score and group membership, and β₀ is the intercept (Swaminathan & Rogers, 1990). The 1 df test of β₃ = 0 is a test of nonuniform DIF. If nonuniform DIF is absent, the xg term can be deleted from the model, and then the 1 df test of β₂ = 0 provides a test of uniform DIF.1 Effect sizes complement these tests. We describe two categories of effect sizes distinguished by the metric of defining departures from the null hypothesis: conditional log odds ratios versus conditional differences in proportions.

Effect Sizes for LR Based on the Conditional-Log-Odds-Ratio Definition of DIF

A natural effect size for LR is the odds ratio. LR coefficients (β̂ _j ) are estimated on the log odds scale. The exponential of β̂ _j [i.e., exp(β̂ _j )] yields the maximum likelihood estimated odds ratio of the event of interest for every one-unit increase in the jth predictor, adjusted for other covariates in the LR model (Hosmer & Lemeshow, 2000). Thus, the exponential of β̂₂ provides the reference-to-focal odds ratio of endorsing the item, conditional on proficiency:2

Odds ratios range from 0 to ∞. Values of α̂ _LR further from 1.0 represent greater DIF magnitude. An odds ratio and its reciprocal are equivalent in strength but not symmetrical in distance from the null value of 1.0 (e.g., 4.0 and 0.25).

Another option for an effect size is to transform α̂ _LR to the logistic definition of the delta scale, used by ETS to measure item difficulty. We use the formula Holland and Thayer (1988) used to convert the Mantel-Haenszel (MH) odds ratio (α̂ _MH ) to the MH delta-DIF statistic (MH-D-DIF or Δ̂ _MH ):

It is apparent that LR-D-DIF is a simple linear rescaling of the regression coefficient, β̂₂.3

In addition, one can calculate the ETS classification system (Dorans & Holland, 1993):

Notice that assigning Categories A and B entails using LR to test H _o : β₂ = 0. Assigning Categories B and C requires testing H _o : |LR-D-DIF| ≤ 1.0. Practitioners can perform the latter test in LR by testing H _o : |β₂| ≤.4255 (i.e., Δ _LR of 1.0 equals β₂ of −.4255).

It is also possible for reporting purposes to convert a conditional log-odds-ratio-based index to the metric of differences in item-proportion-correct called the p metric. We use the formula that Dorans and Holland (1993) used to convert α̂ _MH to MH-P-DIF:

where,

The term P _r ^Ψ is the predicted proportion of examinees endorsing the item in the reference group based on α̂ _LR , and P _f is the proportion of examinees endorsing the item in the focal group.

Effect Sizes for LR Based on the Conditional-Difference-in-Proportions Definition of DIF

The contingency-table standardization (STD) procedure defines departures from the null DIF hypothesis with conditional differences in proportions, and the resulting measure is usually reported in the p metric (STD-P-DIF) (Dorans & Holland, 1993; Dorans & Kulick, 1986). One could estimate a LR model-based STD measure of DIF:

where P _fm ^LR and P _rm ^LR are predicted from the LR model.

This index is reminiscent of item response theory (IRT) model-based standardization (Wainer, 1993) except instead of integrating over θ, averaging occurs over total scores. Historically, absolute values between .05 and .10 are inspected to ensure that no possible DIF is overlooked, and absolute values above .10 are considered more unusual and should be examined (Dorans & Kulick, 1986). One could implement a p metric classification system for LR, applicable to LR-STD-P-DIF or LR-P-DIF:

The weights (w _m ) for averaging conditional differences in proportions in the STD procedure have traditionally been based on intuitive rationale. In DIF studies, w _m is often chosen to be the number of focal group examinees at each stratum (N _fm ) (Dorans & Kulick, 1986). Other plausible weights include (Dorans & Holland, 1993; Mosteller & Tukey, 1977) (a) the number of reference group examinees at each stratum (N _rm ), (b) the number of the total examinees at each stratum (N _tm ), or (c) the relative frequency of some real or hypothetical standard group. One could also use Cochran’s (1954) statistically driven weights (Dorans & Holland, 1993):

Another option, available in the STDIF software (Robin, 2001; Zenisky, Hambleton, & Robin, 2003), is the equal weight (w _m = 1), which yields an unweighted average.

Statistical Methods

We modeled Equation 2, without the interaction term, using binary LR with the SAS LOGISTIC procedure. The matching score was the total score including the studied item (Holland & Thayer, 1988). For the purposes of this article, we examined only uniform DIF. Graphs of empirical logits and LOWESS smoothed curves indicated the LR assumption of linearity of the logit was reasonably satisfied for all items (Monahan, 2004). The SOA and EPESE items were approximately unidimensional because the cross-validated DETECT index was .16 and .28, respectively (Monahan, Stump, Finch, & Hambleton, in press; Roussos & Ozbek, 2006). In SOA data, there were 7,822 women and 5,121 men. In EPESE data, there were 2,203 women and 1,198 men. For both data sets, total score was skewed right (less skewed for EPESE), and men had a slightly lower mean and variance than women.

Result of Using LR Statistical Test Alone to Detect Uniform DIF

Table 1 displays DIF effect sizes for functional status items from the SOA data set. Each row (studied item) in Table 1 represents a different LR model. The right-most column shows the observed significance of the two-sided LR Wald chi-square test of uniform DIF. Based on this test alone, even if we used a conservative Bonferroni-adjusted significance level of .00217 (i.e., .05/23), 11 of 23 items would be flagged for DIF (Table 1, bold items). Most DIF studies using LR have relied on Wald tests alone. We now illustrate the importance of effect sizes.

Effect Sizes for LR Based on the Conditional-Log-Odds-Ratio Definition of DIF

We will interpret the effect sizes in Table 1 from left to right, beginning with the LR odds ratio (α̂ _LR ). By sorting on ascending LR-D-DIF (equivalently, descending α̂ _LR ), items were conveniently grouped by direction and magnitude of DIF. Thus, for the 11 statistically significant items, the 6 items at the top were more greatly endorsed by men and the 5 items at the bottom were more greatly endorsed by women after adjusting for total score. The α̂ _LR for these 11 items varied in strength from 1.55 to 2.94 (men displayed greater functional problems after adjustment) and from 0.74 to 0.21 (women displayed greater functional problems after adjustment) (Table 1). For example, the LR model estimated that the odds that men reported having a problem with lifting and carrying 25 pounds was about one fifth (0.21) times the odds that women reported this problem, adjusted for overall functional status. The LR-estimated odds of having a problem with using the telephone was almost 3 (2.94) times greater for men compared to women, controlling for overall functional status (Table 1). However, the odds of having a problem with walking was only 1½ times greater for men. Therefore, α̂ _LR indicates that not all 11 statistically significant items exhibited equally important DIF magnitude.

Likewise, LR-D-DIF (Δ̂ _LR ) for the 11 statistically significant items varied in strength from −1.02 to −2.53 and from 0.71 to 3.63 (Table 1). According to the LR-based ETS classification system (LR-ETS-class in Table 1), only 5 of 11 items displayed moderate to large magnitude of statistically significant DIF (C), and 5 items showed moderate DIF (B). We also used the transformation of α̂ _LR to the p metric (LR-P-DIF in Table 1) to classify items according to the p metric system described earlier (LR-P-DIF-class in Table 1). This LR-P-DIF classification system indicated weaker DIF than the LR-ETS classification system for 5 items and stronger DIF than LR-ETS classification for 2 items (Table 1). This was because the nonlinear relationship between LR-D-DIF and LR-P-DIF depends on the proportion of focal group examinees endorsing the item (P _f ). Notice that P _f for Items 2, 3, 9, 11, and 23 were closer to zero than P _f for Items 14 and 16 (Table 1). Using Equations 4, 5, and 6, it can be shown that for a given value of Δ̂ _LR or α̂ _LR , LR-P-DIF will be less for lowly or highly endorsed items than for moderately endorsed items. Likewise, for a given value of LR-P-DIF, Δ̂ _LR and α̂ _LR will be greater for P _f near zero or one than for P _f near .50 (see Discussion).

Effect Sizes for LR Based on the Conditional-Difference-in-Proportions Definition of DIF

Using the traditional weight for DIF analyses (w _m = N _fm ), the magnitudes of the LR-based standardization index (LR-STD-P-DIF-focal in Table 1) for the 11 statistically significant items were even smaller than the log-odds-ratio-based p metric magnitudes of LR-P-DIF (Table 1). A p metric classification system based on LR-STD-P-DIF-focal (i.e., LR-STD-P-DIF-focal-class in Table 1) resulted in flagging only 1 item as definite DIF (C) and only 1 item as marginal DIF (B). For both data sets, the LR standardization effect size was very similar when two other weights were used [total group distribution (N _tm ) and Cochran (c _m )], differing at most from LR-STD-P-DIF-focal by .02 for any item but usually by .01 or less (data not shown). The standardization index using w _m = 1 (LR-STD-P-DIF-equal in Table 1) differed from the other three LR-STD-P-DIF indices (w _m = N _fm , N _tm , c _m ), generally displaying slightly greater absolute values, resulting in 1 item demonstrating definite DIF (C) and 5 items revealing marginal DIF (B).

Abbreviated Results for EPESE Data

Table 2 shows effect sizes for the CES-D depression items. Again, fewer items were flagged when effect sizes supplemented the LR Wald test. Of five statistically significant items, only one item displayed moderate to large DIF (C) by any classification system (Table 2). The main difference between Table 1 and Table 2 is that compared to functional status items (Table 1), depression items (Table 2) showed less difference between the log-odds-ratio-based p index (LR-P-DIF) and the focal-group standardization p index (LR-STD-P-DIF-focal). This was because depression items revealed less DIF magnitude and less skewness of total score. Specifically, using Equations 5, 6, and 7, where w _m = N _fm , it can be shown that the difference between LR-P-DIF and LR-STD-P-DIF-focal depends on α̂ _LR , P _fm ^LR , and N _fm (P _f is a function of N _fm and P _fm ^LR ; P _rm ^LR is a function of α̂ _LR and P _fm ^LR because the odds ratio is assumed to be constant across strata in uniform-DIF LR). In short, the magnitude of LR-STD-P-DIF-focal increases as P _fm ^LR values near .50 receive greater weight (i.e., larger N _fm ) than P _fm ^LR values near zero or one.

Sensitivity of Results

Results were very similar after deleting examinees at the floor and ceiling. We computed Cochran’s (1954) test criterion by specifying w _m = c _m in LR-STD-P-DIF and by using LR-predicted proportions in the standard error; the observed significance was extremely similar to the LR Wald observed significance for all items in both data sets (differed at most by .008). This is not surprising given that Cochran (1954) derived these weights for a test criterion that would be powerful for detecting an alternative hypothesis of a constant difference on either the logit or probit scale. Thus, in LR, although Cochran weights are an option when computing STD-LR-P-DIF, the Cochran test might be an unnecessary adjunct to the LR Wald test.

Choosing an Effect Size: Pros and Cons

The effect sizes can be contrasted on a number of dimensions. First, as for ease of interpretation, indices reported on the delta and p metric are symmetrical around their null value of zero, which facilitates interpreting DIF in opposite directions. However, those experienced with interpreting odds ratios may find α̂ _LR easier to interpret than LR-D-DIF, which is on the ETS-preferred delta metric. For data conforming to the two-parameter logistic (2PL) IRT model, one advantage of LR-D-DIF is that the MH-D-DIF parameter (Δ_{2 PL} ) can be written as a linear rescaling of the difference between b parameters (Roussos, Schnipke, & Pashley, 1999):4 Δ_{2 PL} = 4a(b _R –b _F ). The α _MH parameter also shares the advantage of being related, although nonlinearly, to IRT b-DIF (Roussos et al., 1999). The p metric is probably the most universally understood metric, conveniently connected to total and true score metrics. Practitioners should choose an effect size that they and their readership can easily interpret.

Second, practitioners should choose an effect size whose metric for defining departures from the null hypothesis supplies the most valid definition of DIF for their purpose. Specifically, relative to conditional odds ratios (α̂ _LR ) or log odds ratios (Δ̂ _LR ), conditional differences between proportions (LR-STD-P-DIF) will be compressed for items with low or high endorsement rates. However, from another perspective, odds ratios and log odds ratios play up small differences in proportions that are near zero or near one. LR-P-DIF is based on the conditional log-odds-ratio definition but is reported on the p metric. Therefore, LR-P-DIF shares some properties with LR-STD-P-DIF (ease of interpretation and lower magnitude, relative to odds ratios, for lowly or highly endorsed items).

Third, in terms of fundamental connections to the LR model, α̂ _LR is a natural estimator, fundamentally connected to a parameter of the LR model. LR-D-DIF is also a simple rescaling of an estimated LR parameter and therefore is fundamentally connected to the LR model. A disadvantage of the LR-STD-P-DIF index is that it is the most removed from the LR parameter estimation method. However, this does not invalidate it as a descriptive measure of DIF.

Fourth, as far as ease of programming, standard software for LR automatically provides α̂ _LR . LR-D-DIF can easily be calculated. LR-P-DIF requires slightly more programming to convert α̂ _LR to LR-P-DIF. LR-STD-P-DIF requires the most additional programming because predicted proportions at each total score level must be first computed and then weighted.

Fifth, the purpose of weights in standardization is not only to standardize according to the distribution of interest but also to yield smaller weight to sparse strata that provide less precise information. Using equal weights could be dangerous if one or more strata are sparse. (None of the values for total score were sparse for the present data sets.) If one chooses an outside (real or hypothetical) standard distribution, one must be careful to not combine large weights with ill-determined differences in proportions (Mosteller & Tukey, 1977).

Recommendations on How to Use the Effect Sizes

First, we recommend using an effect size and a statistical test when deciding whether items exhibit DIF. The effect size prevents flagging unimportant differences in large samples, and the statistical test prevents flagging noise in small samples. Unimportant differences can be significant, as demonstrated here, when using a statistical test alone, even if statistical tests are conservatively adjusted for multiple comparisons. In addition, effect sizes and their classifications help distinguish between levels of nonnegligible DIF (e.g., B vs. C).

Second, practitioners must decide what values of the effect size represent negligible, moderate, and large magnitudes for the intended purpose. For example, ETS uses thresholds of 1.0 and 1.5 on the absolute value of the delta metric, which are equivalent to odds ratios greater than 1.53 (or less than 0.65) and greater than 1.89 (or less than 0.53), respectively. Users of STD procedures often use .05 and .10 thresholds on the absolute value of the p metric. In the medical sciences, the α̂ _LR thresholds of 1.5 and 2.0 are common due to convenient interpretations of one and one-half and twice the odds, respectively. However, smaller thresholds are used (e.g., 1.1) if the exposure is prevalent and disease serious (e.g., when determining whether risk of heart disease is associated with hormone supplements). Interestingly, α̂ _LR thresholds of 1.5 and 2.0 are nearly equivalent to the delta thresholds used in the ETS classification system.

Third, one can take steps to facilitate interpretations. One could calculate the reciprocal of odds ratios less than one. By calculating 1/.74 = 1.35, one can readily see that .74 for Item 21 in Table 1 is not as strong as 1.55 for Items 5 and 16. One can use graphs (e.g., scatter, line, and bar), which help discern relative distances between DIF magnitudes. In addition, sorting items by direction and magnitude of DIF in tables, as we did here, aids interpretation.

Fourth, these effect sizes can be used to facilitate comparisons of DIF procedures. One could compare MH, LR, SIBTEST, STD, and IRT procedures on the p metric (using LR-P-DIF or LR-STD-P-DIF for LR). Likewise, one could compare procedures on the odds ratio or delta metric, where STD-P-DIF and the latent-true-score adjusted difference in proportions of SIBTEST are converted using a formula similar to Equation 22 in Dorans and Holland (1993).

Limitations

The present analyses employed large sample sizes. In smaller sample sizes, the discrepancy between statistical significance versus flagging items with the combination of effect sizes and statistical significance should not be as great. The degree of discrepancy observed here between LR-D-DIF, LR-P-DIF, and LR-STD-P-DIF may differ for other data sets.

Conclusions

These effect sizes and classification systems have received little attention in the DIF literature for binary LR: the adjusted odds ratio (α̂ _LR ), LR-D-DIF (Δ̂ _LR ), LR-P-DIF, LR model-based standardization indices (LR-STD-P-DIF), and the ETS and p metric classification systems. The present examples demonstrate that these effect sizes are quite useful for preventing practically unimportant DIF from being flagged, especially in large samples. There are various pros and cons for choosing among these effects sizes. When steps are taken for their proper use, these effect sizes should be of great benefit to practitioners.