A Framework for Detecting Both Main Effect and Interactive DIF in Multidimensional Forced-Choice Assessments

Forced-Choice Formats and Item Response Theory Models for Multidimensional Forced-Choice Tests

To reduce response biases, psychometric researchers have long advocated for the use of multidimensional forced-choice (MFC) tests (e.g., Anguiano-Carrasco et al., 2015; Calderón Carvajal et al., 2021; Christiansen et al., 2005; Guenole et al., 2018; Jackson et al., 2000; Merk et al., 2017; Ng et al., 2020). In MFC tests, respondents are given two or more statements with similar appeals, and are asked to choose the one that fits them best or to rank them in order of fit (for an overview of different MFC formats, see Hontangas et al., 2015). Numerous studies have shown that the MFC form significantly reduces socially desirable responding and acquiescence (e.g., Heggestad et al., 2006; Jackson et al., 2000). Some research also demonstrated that MFC tests have similar or higher construct and criterion-related validity than Likert-type personality scales (Bartram, 2007; Salgado et al., 2015). Currently, examples of widely used MFC questionnaires include the International Personality Item Pool-Multidimensional Forced Choice (IPIP-MFC; Heggestad et al., 2006), the Occupational Personality Questionnaire (ipsative shorted format; OPQ32r; Brown & Bartram, 2009), and the Personality and Preference Inventory (Cubiks, 2010). According to Tett et al. (2011), almost 30% of U.S. companies use MFC tests for talent selection in organizational contexts.

Although MFC tests have great advantages, traditional scoring formats of these tests have been criticized for yielding ipsative or partially ipsative scores (Baron, 1996; Brown & Maydeu-Olivares, 2011; Cattell, 1944). Cattell (1944) first used the term ipsative (from the Latin ipse: he, himself) to name a type of scale in which a score on one attribute is relative to scores on the person's other attributes. The scores generated when individuals respond to such a scale are ipsative scores. Ipsative scores only reflect relative levels of the measured traits within an individual because the sum of scores is a constant for each respondent (Hontangas et al., 2015). Accordingly, ipsative scores are not directly comparable across individuals (Loo, 1999). Brown and Maydeu-Olivares (2013) also indicated that ipsative scores have problematic psychometric properties, including reliability, criterion-related validity, and construct validity. For a period of time, this limited the application of MFC tests.

Fortunately, this problem has been addressed using modern psychometric theories. Specifically, researchers have proposed a group of multidimensional forced-choice item response theory (MFC-IRT) models to analyze MFC item response data and generate reliable trait scores that can be compared across individuals. Research has revealed that IRT-estimated scores effectively overcome the limitation of ipsative data and offer higher measurement precision than ipsative scores (Brown & Maydeu-Olivares, 2013). In the past few decades, various MFC-IRT models have been proposed to establish the relationship between the overt responses and the latent traits assessed by MFC tests. To classify existing MFC-IRT models, Brown (2014) introduced a unified framework comprising three dimensions: (1) the MFC format used, (2) the measurement model for the relationships between items and measured traits, and (3) the decision model for choice behavior. In the framework, measurement models used for personality assessments are categorized into two main types: 1) Dominance Models and 2) Ideal Point Models, depending on assumptions regarding respondents’ response processes. Dominance models posit a monotonic increase in the likelihood of endorsing an item with an increasing underlying trait level. Conversely, ideal point models assume a nonmonotonic relationship where the probability of endorsement peaks when the respondent's trait level matches the item's location. As for decision models, four primary types have been employed to model MFC response data: Thurstone's Law of Comparative Judgment (Thurstone, 1927), Andrich's Forced Endorsement Model (Andrich, 1989), Luce's Choice Axiom (Luce, 1977) and Bradley–Terry Model (Bradley & Terry, 1952). Following the Brown's (2014) framework, existing MFC-IRT models are classified as depicted in Table 1.

OPEN IN VIEWER

Table 1.

Existing MFC-IRT Models and Their Categorization in the Brown's (2014) Framework.

MFC-IRT Models	Authors	MFC Formats	Measurement Model	Decision Model
Zinnes-Griggs (Z-G) model	Zinnes and Griggs (1974)	PICK	IPM	BT
Multi-unidimensional pairwise-preference model (MUPP-GGUM)	Stark et al. (2005)	PICK	IPM	AFE
Thurstonian IRT (TIRT) model	Brown and Maydeu-Olivares (2011)	PICK, MOLE, RANK	DM	TLCJ
Generalized graded unfolding-RANK (GGUM-RANK) model	Hontangas et al. (2015) and Lee et al. (2019)	PICK, MOLE, RANK	IPM	AFE
Multi-unidimensional pairwise preference two-parameter logistic (MUPP-2PL) model	Morillo et al. (2016)	PICK	DM	AFE
Rasch ipsative model (RIM)	Wang et al. (2017)	PICK	DM	AFE
Bayesian random block IRT (BRB IRT) model	Lee and Smith (2020a)	PICK, MOLE, RANK	DM	AFE
Zinnes-Griggs pairwise preference item response theory model	Joo et al. (2021)	PICK	IPM	TLCJ
Forced-choice ranking models	Hung and Huang (2022)	MOLE, RANK	IPM	AFE
Generalized Thurstonian unfolding model (GTUM)	Zhang et al. (2023)	PICK, MOLE, RANK	IPM	TLCJ

Note: PICK = choose or pick the statement that is most descriptive of them, MOLE = choose the most and least descriptive statements, RANK = rank all statements by their descriptiveness, DM = dominance model, IPM = ideal point model, AFE = Andrich's Forced Endorsement Model, TLCJ = Thurstone's Law of Comparative Judgment, LUCE = Luce's axiom of choice, BT = Bradley-Terry Model.

Recent advancements in IRT models for analyzing MFC response data have created two noteworthy and innovative research avenues. First, the development of novel MFC-IRT models that incorporate response time (e.g., Bunji & Okada, 2020, 2022; Guo et al., 2023) has provided a deeper understanding of the MFC item response process. This integration of response modeling and response time modeling holds promise for enhancing measurement accuracy. Second, researchers have delved into modeling aberrant response behavior specific to MFC tests (e.g., Frick, 2022; Peng et al., 2023). Expanding the scope of MFC-IRT models to accommodate diverse aberrant response patterns contributes significantly to enhancing the validity of derived trait estimates. Looking ahead, by continuing to advance MFC-IRT methodology and expand the scope of modeled response behaviors, researchers can further unlock the richness of MFC response data to obtain enhanced insights into the cognitive processes underlying test performance and more valid measurement of the noncognitive constructs of interest.

In summary, modern IRT approaches facilitate robust modeling and scoring of MFC tests, providing versatile options for analyzing MFC response data and generating reliable trait scores. Despite the abundance of MFC-IRT models, this study chose the TIRT model for three reasons: (a) it can deal with common MFC formats including choose or pick the statement that is most descriptive of them (PICK), choose the most and least descriptive statements (MOLE), and rank all statements by their descriptiveness (RANK); (b) its adaptability and widespread use in recent research (e.g., Brown & Maydeu-Olivares, 2011; Bürkner et al., 2019; Peng et al., 2023) have established it as a reliable analysis framework; (c) the development of MFC item blocks for dominance models seems easier and incur a lower cost than for ideal point models (Brown & Maydeu-Olivares, 2010), and some studies pointed out that the majority of existing personality assessments seem to fit a dominance model better than an ideal point model (e.g., Cho et al., 2015).

Why Detect Differential Item Functioning for MFC Tests?

As delineated in the Principles for the Validation and Use of Personnel Selection Procedures (2018), a fair selection procedure enables all test takers to show their status on a construct without undue advantage or disadvantage due to other personal attributes (e.g., age, race, ethnicity and gender). As an impactful assessment tool in high-stakes personnel selection procedure, MFC tests must also inevitably consider the issue of test fairness. In psychometric contexts, a frequently used concept intimately tied to test fairness is measurement invariance (MI). MI reflects the comparability of the trait estimates by respondents with divergent demographic characteristics (e.g., males and females). MI at the item level is also referred to as being free from differential item functioning (DIF; see Magis et al., 2010 for an overview). In MFC tests, MFC DIF refers to how individuals with matched latent trait levels but from different subgroups may respond to the same MFC item block ¹ in a different way. As highlighted by the Standards of Educational and Psychological Testing (AERA et al., 2014) and Zieky (2015), DIF analysis has been a routine practice in test construction to ensure the fairness and accuracy of test outcomes. Hence, to ensure the fairness and validity of newly developed MFC tests in high-stakes personnel selection environments, it is essential to conduct MFC DIF analysis.

Compared to the traditional DIF detection with single-stimulus items, detecting DIF in MFC tests faces unique challenges. Specifically, the manifestation of DIF is more complex in MFC tests than in traditional tests. First, each item block in the MFC test contains at least two statements, which increases the chances that DIF appears in MFC tests. Additionally, following the views of Qiu and Wang (2021) and Lee et al. (2021), there are multiple possible sources of DIF in MFC tests, including statements, pairwise comparisons, or both. Finally, DIF in MFC tests may cancel or accumulate depending on how the statements are perceived relative to each other (Lee et al., 2021). This complexity requires unique detection methods for DIF in MFC tests, necessitating research in developing DIF detection methods tailored specifically for MFC tests.

Existing MFC DIF Detection Methods and Their Limitations

The current literature on MFC DIF detection methods is thin, with only four studies published to date to our knowledge. Existing methods can be classified into three categories, depending on whether they address differential functioning at the statement level, the item block level, or the whole test level.

Detecting MFC Differential Functioning at the Statement Level. Chen and Wang (2014) argued that a statement in a MFC item block may exhibit different utilities for different respondents. Therefore, they extended the logistic regression method (LR; Swaminathan & Rogers, 1990) to detect MFC statement-level differential functioning. Qiu and Wang (2021) introduced the term differential statement functioning (DSF) to define the differential functioning manifested at the statement level in MFC tests. Based on the Rasch ipsative model, they proposed three methods for detecting DSF: equal-mean-utility, all-other-statement, and constant-statement. Although the feasibility of these methods has been well-supported, they do not take into account an important issue, according to the views of Lee et al. (2021), that DSF may be canceled or accumulated at the item block level. More specifically, a MFC item block might exhibit DIF even when all the statements do not exhibit significant DSF; conversely, statements that exhibit DSF could be combined to produce a DIF-free item block. Therefore, performing MFC DIF detection only at the statement level is insufficient.

Detecting MFC Differential Functioning at the Item Block Level. Lee et al. (2021) argued that it may be more prudent to detect MFC DIF at the item block level directly, as DSF can be canceled or accumulated within an item block. To achieve this, they changed their unit of differential functioning analysis from a statement to an item block, and then they proposed a method called omnibus Wald tests (OWT) to test MFC DIF at the item block level for the TIRT model. Take a MFC triplet as an example: the basic idea of this method is to perform omnibus 6-df Wald tests on the three loading parameters and the three threshold parameters of each item block. A limitation of this method is that it cannot locate the sources that lead to DIF, whether it is the statement level, the pairwise level, or both.

Detecting MFC Differential Functioning at the Whole Test Level. Lee and Smith (2020b) introduced the multiple group confirmatory factor analysis (CFA) method based on the TIRT model to detect differential functioning on the whole test level. This work found that absolute changes (Δ) in alternative fit indices (e.g., CFI and NCI) worked well in testing measurement invariance for MFC tests. They also proposed cutoffs for changes in model fit indices to better detect metric and scalar non-invariance. However, this approach can only detect whether the whole MFC test maintains measurement invariance across subgroups, and cannot further provide information about which specific item blocks exhibit MFC DIF, which limits its practical application.

Overall, all the above existing methods for detecting MFC DIF employed a multigroup approach, a technique also utilized in traditional single-stimulus DIF detection. This approach requires that the covariates used for DIF detection be categorial, frequently with only two groups, like male and female respondents. While these established MFC DIF detection methods have demonstrated effectiveness, none adequately tackle two crucial issues.

First, in organizational settings, for those ordered or multi-category covariates (e.g., country, ethnicity, race, socioeconomic status, educational level, department/division, job grade, and cultural background) and continuous covariates (e.g., working years, language proficiency, and salary level), if without protected classes prescribed by laws or regulations, the subgroups are defined arbitrarily using a-priori cutoffs, commonly dichotomizing these covariates by their respective medians calculated from the collected data (Bourion-Bédès et al., 2015; Strobl et al., 2013). However, as cautioned by MacCallum et al. (2002), a-priori discretization carries the risk of discarding crucial variation within artificial categories, and may even fail to capture the range where DIF occurs precisely. Moreover, the outcomes of DIF detection are significantly affected by how subgroups are formed. Some researchers also demonstrated that determining the groups arbitrarily a-priori may lead to information loss and a potential decrease in the power to detect DIF (Strobl et al., 2013; Tay et al., 2015).

The second limitation of the existing MFC DIF detection methods is omitting the interaction among multiple covariates that potentially induce DIF, as existing methods only detect MFC DIF by each covariate separately. To distinguish DIF induced by the interaction between multiple covariates from DIF induced by a single covariate investigated in previous MFC DIF studies, we refer to the former interactive DIF, and the latter main effect DIF. Detecting interactive DIF in MFC tests bears many benefits. First, it allows for a more thorough understanding of the complex sources causing differential functioning (Belzak, 2023; Robert et al., 2006; Tay et al., 2015), enabling experts to make targeted modifications. As stated by Collins (1990), there is an increasing recognition that identity encompasses the intersection of multiple background characteristics. Second, it helps to more effectively ensure test fairness. Revealing more hidden measurement bias through interactive DIF detection helps to prevent hiring discrimination. Finally, it further enhances test validity and the overall quality of the tests. In contrast to main effect DIF, interactive DIF is particularly insidious, and it only becomes apparent when multiple covariates are considered simultaneously in a single round of DIF testing. But interactive DIF can be commonly present. For example, this measurement bias caused by the interaction between covariates is quite evident in the assessment of adolescent delinquent behavior (Bauer, 2017). Over the past decade, researchers have shown great research interest in the interactive DIF and developed methods to detect interactive DIF in traditional single-stimulus tests (e.g., Bollmann et al., 2018; Strobl et al., 2013; Tutz & Berger, 2016). However, methods to detect interactive DIF in MFC tests are lacking.

In conclusion, the current methods for detecting MFC DIF face limitations in dichotomizing the grouping covariates arbitrarily a-priori, and omitting the investigation of interactive DIF. This paper proposes a new method based on the recursive partitioning technique to address these limitations.

A Brief Overview of Recursive Partitioning Technique and Its Application in IRT-Based DIF Detection

Advances in machine learning (ML) have provided new opportunities to overcome the constraints of conventional DIF detection approaches. As a multivariate data mining technique in ML, recursive partitioning (RP) has proven remarkably effective in analyzing multivariate data across diverse scientific fields, including genetics (e.g., Díaz-Uriarte & Alvarez de Andrés, 2006), ecology (e.g., Cutler et al., 2007), medicine (e.g., Khalilia et al., 2011), and psychology (e.g., Hayes et al., 2015). RP has its origins in classification and regression tree (CART) models introduced by Breiman et al. (1984). The RP algorithm works by building a tree-like structure, with covariates being the nodes. The tree starts with a single node, and then branches into two mutually exclusive child nodes that maximize within-node homogeneity and between-node heterogeneity. This branching recurs for each new node.

In recent years, RP has emerged as a promising ML approach for IRT-based DIF detection in single-stimulus tests. One major advantage is the ability to incorporate both multi-category and continuous covariates into the partitioning, with the branching cutoffs determined organically based on the data. RP is expected to uncover complex and intricate interactive DIF, which traditional techniques cannot. Owing to these strengths, RP has gained increasing research attention over the past decade as an innovative tool for detecting DIF in traditional single-stimulus tests (e.g., Bollmann et al., 2018; Finch et al., 2015; Strobl et al., 2013; Tutz & Berger, 2016). Such RP algorithm involves the following five key steps:

Starting with the full sample, perform grouping parameter calibration using the current sample of all respondents based on each covariate of interests.

The Present Study

To our knowledge, published studies on DIF detection using RP techniques have been limited to single-stimulus tests. While promising results have been demonstrated, its effectiveness in MFC tests has not yet been thoroughly investigated. In this study, we designed a RP-based method to detect both the main effect DIF and interactive DIF in MFC tests and evaluated its efficacy with simulation. The proposed method seeks to enhance the power of detecting MFC DIF related to ordered and continuous covariates without predefined protected classes through data-optimized grouping. Furthermore, it is designed to simultaneously identify item blocks and covariates responsible for DIF, holding significant practical implications for improving the validity and fairness of MFC tests in high-stakes personnel selection.

In what follows, we first introduce the Thurstonian IRT model, a statistical model used to fit forced-choice response data in this study. Second, we provide detailed information on the procedures of the proposed method. Third, we evaluate the proposed method's performance using three Monte Carlo simulation studies. In the fourth section, we present an empirical example that illustrates the applicability of the proposed method using a dataset derived from a well-validated MFC test collected in a real-world setting. Lastly, we discuss the limitations of the current study and offer recommendations for future research aimed at enhancing the capabilities of the proposed method.

Differential Item Functioning for the Thurstonian IRT Model

In the TIRT model, we can model the DIF by specifying separate equations for the two groups of interest using the following expression:

P ( y j = 1 | θ a , θ b , γ j g , λ i g , λ k g , ψ i g , ψ k g , g ) = Φ ( − γ j g + λ i g θ a − λ k g θ b ψ i g 2 + ψ k g 2 ) , g = 1 , 2 ,

(6)

η j = λ i g θ a − λ k g θ b − γ j g Ψ i g 2 + Ψ k g 2 , g = 1 , 2 ,

(7)

According to Lee et al. (2021), DIF in the TIRT model can manifest in one of three scenarios: only loading parameters (λ), only threshold parameters (γ), or both loading and threshold parameters. These scenarios correspond to DIF at the statement level, pairwise comparison level, and both levels simultaneously, respectively. When examining DIF at the MFC item block level, it is important to note that DIF may occur even when all statements within the block are individually invariant. Therefore, a robust approach is to perform simultaneous DIF detection for both the loading and threshold parameters of each item block, as testing one parameter alone may lead to inaccurate conclusions.

The Procedures of the Proposed Method: the Block-Based Sequential Recursive Partitioning Trees

This paper proposes the Block-based Sequential Recursive Partitioning Trees method to detect MFC DIF in MFC tests, abbreviated as MFC-BSRPT. This method combines recursive partitioning technique with the TIRT model to grow a DIF tree for each studied item block. We first estimate the item parameters of the item block based on response data from respondents of different two subgroups (i.e., focal and reference groups) at each splitting point, then, based on binary splits, sequentially grow a recursive partitioning tree for each item block using a selected split criterion, and finally generate a tree diagram that displays where DIF is present by which cutoff of which covariate and in what ways. To aid understanding, we provide a flow chart of the procedure (see Figure 1). Below we elaborate on the detailed procedures and technical details of the proposed method taking a single studied item block as an example.

OPEN IN VIEWER

Figure 1.

A flow chart of the fitting procedures for the proposed method.

Step 1: Locate All Possible Cut Points in All Available Covariates

Two key issues must be considered when growing a tree: (1) how to select a covariate for a node and identify the best cut point for the selected covariate, and (2) when to stop growing the tree. These issues are explained in subsequent sections.

To grow a tree, one must identify all possible cutpoints within the covariate space. The covariate space refers to all levels of all covariates of interest, and these covariates could be categorial, ordered, or continuous. Notably, while all possible levels with multi-category and ordered covariates are countable, the levels for a continuous covariate have infinite choices. Therefore, in order to locate cutpoints suitable for dichotomizing the continuous covariate to facilitate DIF detection, we need to determine countable and meaningful levels for continuous covariates. In general, a common approach is to consider the number of decimal places reserved for each cutoff. For example, Strobl et al. (2013) took each possible cutoff value that was used to dichotomize the continuous covariate to two decimal places.

At each node, a covariate is selected and we divide all respondents of the full set A into two subgroups (i.e., focal and reference groups), that is, two subsets A₁ and A₂, which are denoted by the following equation:

A 1 = A ∩ { x m ≤ c } and A 2 = A ∩ { x m > c } ,

(8)

where c is one of the levels in the covariate x m .

Let x p T = ( x p 1 , … , x p m ) denote a person-specific covariate vector of length m. To detect MFC DIF for an item block, one should examine all the covariates and all possible splits in each covariate using the following linear predictor of the TIRT model,

{ η j = θ a ⋅ node λ i − θ b ⋅ node λ k − node γ j node Ψ i 2 + node Ψ k 2 , node λ i = [ λ i l ⋅ I ( x m ≤ c m ) + λ i r ⋅ I ( x m > c m ) ] , node λ k = [ λ k l ⋅ I ( x m ≤ c m ) + λ k r ⋅ I ( x m > c m ) ] , node γ j = [ γ j l ⋅ I ( x m ≤ c m ) + γ j r ⋅ I ( x m > c m ) ] , node Ψ i 2 = [ Ψ i l 2 ⋅ I ( x m ≤ c m ) + Ψ i r 2 ⋅ I ( x m > c m ) ] , node Ψ k 2 = [ Ψ k l 2 ⋅ I ( x m ≤ c m ) + Ψ k r 2 ⋅ I ( x m > c m ) ] .

(9)

In Equation 9, I ( ⋅ ) is the indicator function with I ( d ) = 1 if d is true and I ( d ) = 0 otherwise; c m is a possible split point for the mth covariate; the subscripts l and r are abbreviations for the words left and right, respectively; λ i l is the loading parameter of the statement i in the left node ( x m ≤ c m ) and λ i r is the loading parameter of the statement i in the right node ( x m > c m ) . The focal and reference groups were constructed using the two split regions (i.e., { x m ≤ c m } and { x m > c m } ) obtained by the mth covariate at a split point c m . Therefore, Equation 9 can be regarded as an alternative representation of Equation 6.

Step 2: Search for the First Best Split Covariate and the Best Split Point Based on a Split Criterion

After locating all possible cut points in all available covariates, the algorithm searches for the optimal split of each covariate by applying a selected split criterion. According to Tutz and Berger (2016) and Strobl et al. (2009), there are two commonly accepted split criteria in tree-based modeling approaches: (a) impurity measures such as the Gini Index or Shannon Entropy, and (b) test-based splits such as the log-likelihood test statistic. Test-based splits assess whether the current child node requires further partitioning based on the corresponding test statistic. Because this study utilizes the TIRT model, we chose the test-based split criterion for the proposed method.

In general, any statistic which has been shown to perform well in identifying MFC DIF can serve as a test statistic. In this study, we chose the Wald statistic. Based on Lee et al.'s (2021) study, the Wald statistic ( χ 2 ) is a potential tool in this regard, and it can be expressed as follows:

χ 2 = ( ξ R − ξ F ) T ( Σ R + Σ F ) − 1 ( ξ R − ξ F ) ,

(10)

where ξ R and ξ F represent the vectors of item parameter estimators of the reference group (R) and focal group (F) for a single item block, respectively; Σ R and Σ F denote the asymptotic variance and covariance matrices for ξ R and ξ F , respectively. The test statistic utilized in the proposed method follows a chi-square distribution.

One issue that must be addressed before using the Wald statistic for MFC DIF detection is how to obtain comparable item parameter estimates for the different subgroups of respondents. In this study, we used a well-validated method named the free baseline DIF testing approach employed by Lee et al. (2021). The procedures of this approach are as follows. First, an anchor subset consisting of one or more DIF-free item blocks should be identified, which ensures a common metric between groups when conducting TIRT parameter calibration. Then, conduct parameter calibration. For anchor items, get parameter calibrated with the TIRT model in which the parameters of all anchor item blocks are constrained to be identical between groups; for all the remaining item blocks, parameters for these item blocks were freely estimated across groups. Finally, examine the equivalence of parameters using the omnibus 6-df Wald tests in the current block. Lee et al. (2021) found that, as the sample size and DIF magnitude increased, power approached 1.0 and Type I error rates approached the nominal level (.05). Therefore, the Wald statistic based on free baseline testing is adopted as the selection criterion for the proposed method.

To identify the first best split covariate and its best cut point, the algorithm fits the TIRT model to different grouped response datasets, and these datasets are generated by splitting the original response data based on all possible cut points in the covariates being examined. After parameter calibration, the algorithm employs the null hypothesis H₀: λ i l = λ i r , λ k l = λ k r , γ j l = γ j r to evaluate the significance of the Wald statistic calculated from each dataset. It is worth noting that when locating the best split covariate and its best cut point, the proposed method only takes the numerical magnitude of the calculated Wald statistic as the basis for the selection. Specially, the combination of the covariate and its cut point with the maximum Wald statistic is selected as the initial split for constructing the left and right child nodes. These two child nodes are denoted as I ( x m ≤ c m ) and I ( x m > c m ) , respectively. As to how the significance of the Wald statistic is handled, it will be taken into account in the subsequent Step 4, which we will describe in detail therein.

In general RP applications, a covariate is allowed to be reused for different nodes of the same tree. However, in the proposed method, we set a restriction that each covariate is not allowed to be reused. More specifically, once the first best split covariate m is identified, it is excluded from further consideration during subsequent search iterations within the same block. This restriction was performed for two main reasons: (1) to control for Type I errors due to the multiple testing on the same covariate and (2) to reduce the complexity of the tree structure to improve interpretability of the proposed method.

Step 3: Search for Further Splits Based on Step 2

After identifying the first best split covariate and its best cut point, the method will continue to search for the subsequent best split covariates from the remaining covariates and their best split point. Specifically, the algorithm will re-calibrate the TIRT model item parameters using the subset of response data that belong to the current node only, and then re-compute the corresponding Wald statistics. Then, the algorithm locates the optimal combination of covariate and split point corresponding to when the Wald statistic is maximized.

Take the established right child node (i.e., I ( x m > c m ) ) as an example, the further possible nodes of the variable s are denoted as I ( x m > c m ) ⋅ I ( x s ≤ c s ) and I ( x m > c m ) ⋅ I ( x s > c s ) . Then, the adjusted linear predictor of the TIRT model can be expressed as:

{ η j = θ a ⋅ node λ i − θ b ⋅ node λ k − node γ j node Ψ i 2 + node Ψ k 2 , node λ i = [ λ i l ⋅ I ( x m ≤ c m ) + λ i r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s ≤ c s ) + λ i r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s > c s ) ] , node λ k = [ λ k l ⋅ I ( x m ≤ c m ) + λ k r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s ≤ c s ) + λ k r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s > c s ) ] , node γ j = [ γ j l ⋅ I ( x m ≤ c m ) + γ j r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s ≤ c s ) + γ j r [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s > c s ) ] , node Ψ i 2 = [ Ψ i l 2 ⋅ I ( x m ≤ c m ) + Ψ i l 2 [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s ≤ c s ) + Ψ i r 2 [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s > c s ) ] , node Ψ k 2 = [ Ψ k l 2 ⋅ I ( x m ≤ c m ) + Ψ k l 2 [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s ≤ c s ) + Ψ k r 2 [ n ] ⋅ I ( x m > c m ) ⋅ I ( x s > c s ) ] .

(11)

where λ i r [ n ] denotes the loading parameter corresponding to the new child node [ n ] in the grown tree and so on for the rest.

Step 5: Terminate DIF Detection and Output a Tree Diagram Presenting the MFC DIF Form of the Current Item Block

It is crucial to note that tree diagrams are unnecessary for DIF-free item blocks on all the covariates that are incorporated into the MFC DIF detection. However, for DIF item blocks, a tree diagram is essential to illuminate which covariates generate DIF and their corresponding forms of MFC DIF. To aid potential practitioners’ understanding, Figure 2 presents four schematic tree diagrams, each representing a different MFC DIF form. Each child node of the tree represents a subgroup, and all TIRT model item parameters (i.e., λ and γ) for the current item block on the current subgroup are displayed in that child node. Solid or dashed arrows are used to indicate which covariates induce or do not induce MFC DIF, with solid lines signifying the presence of MFC DIF and dashed lines indicating its absence. While both item blocks 1 and 2 exhibit the main effect DIF, the difference lies in that item block 1 exhibits DIF solely on age, while item block 2 exhibits DIF on both the age and gender. In contrast, both item blocks 3 and 4 exhibit the interactive DIF, but there are differences in their MFC DIF forms. Specifically, the DIF in item block 3 is induced solely by the interaction between the two covariates, whereas the DIF in item block 4 exhibits main effect DIF on age in addition to being induced by the interaction between the two covariates.

OPEN IN VIEWER

Figure 2.

Schematic tree diagrams of different MFC DIF forms for different item blocks of a MFC test.

Controlling the Type I Error

Compared to existing MFC DIF detection methods, the potential advantage of the proposed method is in its ability to not only detect MFC DIF by multiple covariates simultaneously in a single algorithm, but also to handle ordered, multi-category and continuous covariates in addition to binary covariates, all with the same algorithm. However, the problem of multiple testing needs to be addressed. For the proposed method, whether performing inter-covariate DIF detection (i.e., simultaneous DIF detection of multiple covariates) and intra-covariate DIF detection (i.e., simultaneous DIF detection at multiple possible levels to dichotomize each multi-category, ordered or continuous covariate), the proposed method involves multiple DIF testing. And it is possible that small or trivial DIF effects may be detected in item blocks that do not actually exhibit DIF in a covariate, especially for the multi-category, ordered and continuous covariate, which may lead to inflated false alarms.

An important concept in the statistical literature on multiple testing is the familywise error rate. According to Benjamini and Hochberg (1995), the familywise error rate refers to the probability of at least one false rejection when a family of null hypotheses is tested simultaneously. In the proposed method, it corresponds to the probability that, for an item block, at least one of the Wald tests conducted among multiple covariates and multiple possible dichotomization levels within each covariate is falsely positive, making a Type I error, and flagging the item block as a DIF item block. Following the suggestion of Tutz and Berger (2016), if one wants to control the familywise error rate by a global significance level α in DIF detection, one has to use much smaller significance levels in the single tests, that is, one should adjust the local significance level based on the global significance level.

To effectively control for the family-wise Type I errors for the proposed method, we implemented the Bonferroni adjustment in this study. The Bonferroni adjustment divides the overall significance level α (here set as 0.05) by the number of hypotheses under examination, yielding the local significance level for the Wald test applied to each item block and covariate. This adjustment accounts for the increased probability of false positives due to multiple testing, thereby ensuring that the probability of incorrectly identifying a DIF-free item block as a DIF block is controlled at or below α. It is worth noting that, in this study, when adjusting the local significance level in the multiple DIF testing using the Bonferroni adjustment, the denominator for α can be set in two ways: (1) the number of covariates requiring DIF testing, and (2) the number of different possible levels at which each covariate requiring DIF testing is dichotomized. For a binary covariate, only the first option is applicable; For multi-category, ordered, or continuous covariates, both options are applicable.

In addition to the Bonferroni adjustment, Nye (2011) noted that incorporating DIF effect size estimation indices, a measure of practical significance, can potentially remedy the weaknesses of hypothesis testing (i.e., statistical significance) and potentially mitigate Type I errors. For the MFC test, Lee et al. (2021) proposed to calculate the MFC DIF effect size for each item block, an adaptation of Nye's (2011) method, as depicted in the equation below.

d D I F = 1 S D j P ∫ … ∫ [ P j R ( θ ) − P j F ( θ ) ] 2 f F ( θ ) d ( θ ) 1 … d θ D

(12)

Furthermore, Nye (2011) pointed out that, within the IRT framework, a DIF effect size of 0.1 is considered small. Therefore, one can consider this value as a threshold to dichotomize the DIF effect size continuum into a binary classification, and then combine both the significance of the p-value and this binary classification to determine whether the current item block exhibits DIF. The procedures for controlling Type I errors using the DIF effect size are as follows. First, the Wald statistic and its p-value are output at each child node created after recursively partitioning the current block using the proposed method. Next, the average DIF effect size estimate for each child node within the current block is calculated using Equation 12. Finally, the significance of each child node's p-value and its average DIF effect size are combined to further determine whether DIF is present in the current block (i.e., the first two rows in Table 2). Specifically, for an item block to be flagged as a DIF item block requires that the following criterion must be met: the p-value on any established child node is below the adjusted local significance level, and the average DIF effect size estimate of that node should also be greater than or equal to 0.1.

OPEN IN VIEWER

Table 2.

Potential Strategies for Controlling Type I Errors in the Proposed Method.

Strategies for Controlling Type I Errors	Criteria for Flagging as DIF
Number of covariates and DIF effect size (NCD)	I ( p j < α M ) ⋅ I ( E S j m ≥ 0.1 )
Number of covariates and their levels and DIF effect size (NCLD)	I ( p j m < α M ( L m − 1 ) ) ⋅ I ( E S j m ≥ 0.1 )
Number of covariates (NC)	I ( p j < α M )
Number of covariates and their levels (NCL)	I ( p j m < α M ( L m − 1 ) )

Note: I(•) = indicator function, with I(d) = 1 if d is true and I(d) = 0 otherwise; p_j= p-value of the Wald statistic for item block j; α = the overall significance level; M = the total number of covariates detected simultaneously; L_m= the total number of levels within the covariate m; ES_j= the average DIF effect size estimation of item block j.

In the present study, four possible strategies for controlling type I errors in the proposed method are outlined in Table 2, depending on the different Bonferroni adjustment perspectives and whether the average DIF effect size estimate is included. Through these strategies, we hope to provide effective guidance to potential users and ensure a more rigorous and meaningful assessment of DIF, while reducing the risk of identifying spurious effects due to the multiple testing.

Study 1

Data Generation

To generate response data for DIF-free item blocks, we followed these steps: (1) Latent trait vectors were drawn from a multivariate normal distribution with a mean vector of 0 and a covariance matrix denoted as:

Σ θ = [ 1 − 0.4 0 − 0.4 1 0.3 0 0.3 1 ] ,

sourced from Lee et al. (2021); (2) Measurement errors were randomly sampled from a univariate standard normal distribution with a standard deviation of 1; (3) We ensured that the latent traits measured within the same block were all different, with only one trait measured by each statement, and an equal number of statements under each trait; (4) Item parameters from Lee et al.'s (2021) study were employed; and (5) MFC responses for individual respondents were derived through the following equation:

{ y j = 1 ( E is preferred to F ) , if y E F * = − γ j + ( λ E η a − λ F η b ) + ( ε E − ε F ) ≥ 0 y j = 0 ( F is preferred to E ) , if y E F * = − γ j + ( λ E η a − λ F η b ) + ( ε E − ε F ) < 0 .

(13)

The response data generated in this study comprised paired comparison data using the TIRT model based on the triplet format. To ensure that the responses adhered to the defined logic governed by the triplet format, a specific simulation approach was employed. For instance, there are three statements (i.e., E, F and G) in an item block, and the first two pairs of responses had to follow the format E > F and E < G, respectively. The third pair was mandated to be F < G due to the constraints of the triplet format. If the response of the third pair could not be inferred from the first two pairs, Equation 13 was once more utilized to generate the response for that pair.

The process of generating data for DIF item blocks followed the same procedure as outlined above, with one exception in Step 4. Under the DIF condition, Step 4 involved manipulating certain item parameters (i.e., λ and/or γ) of the focal group of DIF item blocks. For this study, we created MFC DIF by manipulating the item parameters of Item Block 3, which was designated as the DIF block with a 10% DIF proportion in the MFC test. Table 3 illustrates the simulation of the different DIF forms used to manipulate the item parameters of Block 3.

OPEN IN VIEWER

Table 3.

True Simulated Differences of Item Parameters for Different DIF Forms Under the Comparison of Focal and Reference Groups.

Block code	Differences of Item Parameters
	DIF Form 1	DIF Form 2	DIF Form 3
3	z ⋅ I ( x 2 = 1 )	{ z ⋅ I ( { x 1 = 0 } ∩ { x 2 = 0 } ) z ⋅ I ( { x 1 = 1 } ∩ { x 2 = 1 } )	z ⋅ I ( x 1 = 1 ) + z ⋅ I ( { x 1 = 1 } ∩ { x 2 = 1 } )

Note: z = value of DIF size, DIF form 1 = only the main effect DIF, DIF form 2 = only the interactive DIF, DIF form 3 = both main effect DIF and interactive DIF.

We also provide a simple example illustrating how to generate DIF data to aid understanding. Based on the assumption that the first two loading parameters of Block 3 (i.e., statements 7 and 8) exhibit main effect DIF on the covariates x₁ and x₂, and that these parameters also exhibit interactive DIF due to the interaction between these two covariates, we use the following equation to simulate responses

{ y 7 , 8 * = − γ 7 , 8 + [ ( λ 7 + size ) η 1 − ( λ 8 + size ) η 2 ] + ( ε 7 − ε 8 ) y 7 , 9 * = − γ 7 , 9 + [ ( λ 7 + size ) η 1 − λ 9 η 3 ] + ( ε 7 − ε 9 ) y 8 , 9 * = − γ 8 , 9 + [ ( λ 8 + size ) η 2 − λ 9 η 3 ] + ( ε 8 − ε 9 ) size = 0.3 ⋅ I ( x 1 = 1 ) + 0.3 ⋅ I ( { x 1 = 1 } ∩ { x 2 = 1 } ) .

(14)

Evaluation Criteria

True Positive Rates and False Positive Rates. To assess the effectiveness of the proposed method, we computed the true positive rates (TPRs) and false positive rates (FPRs) for each condition, which are also referred to as power and Type I error rate, respectively. To provide a thorough comparison of the performance of the two methods, we simultaneously assessed two different dimensions of the evaluation criteria: block level and the combination of block and covariate levels.

Let each block be characterized by a vector ξ j T = ( ξ j 1 , … , ξ j m ) , where m is the number of covariates. ξ j m = 1 if block j exhibited differential functioning in the covariate m and ξ j m = 0 otherwise. If one of the components in vector ξ j T is 1, block j is a DIF block. Block j is a DIF-free block only if all the elements within the vector were zero, that is, ξ j T = ( 0 , … , 0 ) . Using the indicator function I ( ⋅ ) , the evaluation criteria are calculated as follows:

1. TPR on the block level (TPR_B)

T P R B = 1 # { j : ξ j ≠ 0 } ∑ j : ξ j ≠ 0 I ( ξ ^ j ≠ 0 )

(15)

2. FPR on the block level (FPR_B)

F P R B = 1 # { j : ξ j = 0 } ∑ j : ξ j ≠ 0 I ( ξ ^ j ≠ 0 )

(16)

3. TPR for the combination of block and variable (TPR_BV)

T P R B V = 1 # { j , m : ξ j m ≠ 0 } ∑ j , m : ξ j m ≠ 0 I ( ξ ^ j ≠ 0 )

(17)

4. FPR for the combination of block and variable (FPR_BV)

F P R B V = 1 # { j , m : ξ j m = 0 } ∑ j , m : ξ p m = 0 I ( ξ ^ j m ≠ 0 ) .

(18)

DIF Effect Size Estimation. For the MFC test, Lee et al. (2021) proposed the calculation of the MFC DIF effect size for each item block, an adaptation of Nye's (2011) method, as depicted in the equation below.

d D I F = 1 S D j P ∫ … ∫ [ P j R ( θ ) − P j F ( θ ) ] 2 f F ( θ ) d ( θ ) 1 … d θ D ,

(19)

where S D j P represents a pooled estimate of the standard deviation of the binary pairwise comparison outcome j for P respondents; P j R ( θ ) and P j F ( θ ) denote the TIRT probabilities of a binary outcome for the reference group (R) and focal group (F); f F ( θ ) signifies a multivariate normal distribution of the latent trait in the focal group, with the mean and variance derived from the focal group's θ distribution; and D stands for the total number of measured latent traits. According to Nye (2011), DIF effect sizes, as calculated with Equation 19, of 0.1, 0.2, and 0.3 correspond to small, medium, and large DIF values, respectively. Effect sizes exceeding 0.2 are generally deemed more significant, whereas those below 0.2 are considered negligible. However, when calculating DIF effect size estimates, the proposed and existing methods have similarities and differences. Both utilize Equation 19 to calculate estimates, reflecting their similarity. For each item block, average effect size estimates were computed across 50 replications for all pairwise comparisons, using Equation 19. They differ in the dimensions accounted for when obtaining DIF effect size estimates. The existing OWT method averages DIF effect size estimates by each DIF-inducing covariate, while the proposed method averages by each established split.

Results of Study 1 for Power and Type I Error Rate

When all the item blocks in the MFC test were DIF-free, the false positive rates (a.k.a., Type I error rates) of the proposed method are presented in Table 4. Under a 10% DIF proportion in the MFC test, due to space constraints we only present the results under the condition of 2,000 per group in Table 5 of the main text. Please refer to Tables S1 to S3 and Tables S13 to S15 in the Supporting Information for the results from conditions with sample sizes of 500 or 1,000 per group. Notably, in Study 1, we found that when the two covariates were all binary, both the methods yielded acceptable Type I error rates even without considering DIF effect size estimation. Therefore, in Tables 4 and 5 of the main text, we only show the results when the strategy for controlling Type I errors was the number of covariates (NC). For simulation results of the proposed method when the strategy for controlling Type I errors was the Number of covariates & DIF effect size (NCD), see Tables S13 to S15 in the Supporting Information. Overall, as shown in Table 5, the performance of the two methods is comparable when the item block exhibits only the main effect DIF, while the proposed method shows better detection performance when the item block exhibits the interactive DIF compared to the conventional omnibus Wald tests. Subsequent sections analyze the impact of each manipulated factor on the experimental results.

OPEN IN VIEWER

Table 4.

Type I Error Rates Under Different Sample Sizes per Group When all Item Blocks in the MFC Test are DIF-Free in Study 1.

Sample Size Per Group	Methods	FPRB	FPRBV
500	OWT	.03	.01
	MFC-BSRPT	.02	.01
1,000	OWT	.03	.01
	MFC-BSRPT	.03	.01
2,000	OWT	.03	.01
	MFC-BSRPT	.02	.01

Note: OWT = omnibus Wald tests, MFC-BSRPT = the proposed method.

OPEN IN VIEWER

Table 5.

Simulation Results of 10% DIF Proportion Under Different MFC DIF Detection Methods and Different MFC DIF Forms When the DIF Size Is Large (Small) and the Sample Size per Group is 2,000 in Study 1.

MFC DIF Form	Results	Indicator	Methods	λ			γ			both
				1	2	3	1	2	3	1	2	3
Only the main effect DIF	Power	TPRB	OWT	1 (.82)	1 (.94)	1 (1)	1 (.76)	1 (.88)	1 (1)	1 (1)	1 (1)	1 (1)
			MFC-BSRPT	1 (.82)	1 (.94)	1 (1)	1 (.76)	1 (.88)	1 (1)	1 (1)	1 (1)	1 (1)
		TPRBV	OWT	1 (.78)	1 (.92)	1 (1)	1 (.76)	1 (.88)	1 (1)	1 (1)	1 (1)	1 (1)
			MFC-BSRPT	1 (.78)	1 (.92)	1 (1)	1 (.76)	1 (.88)	1 (1)	1 (1)	1 (1)	1 (1)
	Type I error	FPRB	OWT	.03 (.02)	.03 (.03)	.03 (.03)	.02 (.02)	.02 (.02)	.03 (.02)	.02 (.02)	.03 (.02)	.02 (.02)
			MFC-BSRPT	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)
		FPRBV	OWT	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
			MFC-BSRPT	.01 (.01)	.02 (.02)	.02 (.01)	.02 (.02)	.01 (.01)	.02 (.01)	.02 (.02)	.02 (.01)	.01 (.01)
Only the interactive DIF	Power	TPRB	OWT	.04 (.06)	.02 (.10)	.04 (.06)	.10 (.06)	.10 (.04)	.06 (.04)	.06 (.06)	.06 (.08)	.06 (.06)
			MFC-BSRPT	1 (.36)	1 (.70)	1 (.96)	1 (.50)	1 (.62)	1 (1)	1 (.88)	1 (1)	1 (1)
		TPRBV	OWT	.02 (.04)	.02 (.06)	.04 (.02)	.04 (.04)	.04 (.02)	.04 (.02)	.02 (.04)	.04 (.04)	.04 (.04)
			MFC-BSRPT	1 (.34)	1 (.66)	1 (.96)	1 (.50)	1 (.62)	1 (1)	1 (.88)	1 (1)	1 (1)
	Type I error	FPRB	OWT	.03 (.02)	.02 (.02)	.02 (.02)	.02 (.03)	.02 (.02)	.03 (.02)	.03 (.03)	.02 (.02)	.03 (.02)
			MFC-BSRPT	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.04 (.03)	.03 (.03)	.03 (.03)
		FPRBV	OWT	.01 (.01)	.01 (01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
			MFC-BSRPT	.01 (.01)	.01 (.01)	.01 (.01)	.02 (.02)	.02 (.01)	.02 (.01)	.02 (.02)	.02 (.01)	.02 (.01)
Both the main effect DIF and interactive DIF	Power	TPRB	OWT	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)
			MFC-BSRPT	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)	1 (1)
		TPRBV	OWT	.36 (.12)	.38 (.16)	.92 (.42)	.72 (.18)	.92 (.22)	.98 (.50)	.64 (.24)	.86 (.44)	1 (.92)
			MFC-BSRPT	.64 (.26)	.68 (.32)	1 (.80)	.98 (.38)	1 (.56)	1 (.90)	1 (.52)	.98 (.78)	1 (1)
	Type I error	FPRB	OWT	.02 (.02)	.02 (.03)	.03 (.03)	.03 (.03)	.03 (.02)	.03 (.02)	.03 (.02)	.02 (.02)	.03 (.03)
			MFC-BSRPT	.03 (.02)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.03 (.03)	.04 (.03)	.03 (.03)	.03 (.03)
		FPRBV	OWT	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
			MFC-BSRPT	.01 (.01)	.01 (.02)	.01 (.02)	.02 (.02)	.02 (.01)	.02 (.01)	.02 (.02)	.01 (.01)	.01 (.01)

Note: OWT = omnibus Wald tests, MFC-BSRPT = the proposed method. The second rows 1, 2 and 3 in the header of the table indicates the number of noninvariant item parameters under a certain noninvariant item parameter type within an item block is 1, 2, and 3, respectively.

MFC DIF Detection Method and MFC DIF Form. As depicted in Table 5, the results demonstrated the advantages of the proposed method over conventional omnibus Wald tests (OWT) for detecting interactive DIF. For example, with only the interactive DIF present and a sample size of 2,000 per group, a small DIF size of 0.3, and a DIF block proportion of 10%, the TPR ranges were .04∼.10 for OWT versus .50∼1 for the proposed method. Additionally, both methods had FPRs below the nominal level (0.05). With only the main effect DIF present, the two methods had comparable power and low Type I error rates (FPR < 0.05). For example, with a sample size of 2,000 per group, a small DIF of 0.3, and 10% DIF item blocks, the TPR ranges at the block level were .76∼1 for OWT versus .76∼1 for the proposed method, respectively. Taken together, these findings indicate the proposed method had comparable detection performance to OWT for the condition of only the main effect DIF. This was because both methods used the same test statistic for the main effect DIF. With interactive DIF present, OWT had limitations, as it examined covariates independently. In contrast, results in Table 5 suggest that the proposed method has a greater potential to identify interactive DIF. For example, in the condition with only the interactive DIF present, a sample size of 2,000 per group, a large DIF size (0.6), 10% DIF block proportion, and both the loading and threshold parameters were noninvariant, the TPR ranges at the block level were .88∼1 for OWT versus .04 for the proposed method, respectively.

Number of Noninvariant Item Parameters Under a Specific Noninvariant Item Parameter Type Within an Item Block, DIF Size and the Sample Size per Group. The results in Table 5 show that, in most experimental conditions, the detection performance of both the omnibus Wald tests and the proposed method improved as the number of noninvariant item parameters within an item block increased, given a specific noninvariant item parameter type. For example, when the sample size per group was 2,000, the DIF form was only the main effect DIF, the DIF size was small (0.3), and only the loading parameter was noninvariant, TPRs at the block level ranged from .82 to 1 (for one and three noninvariant loading parameters within an item block, respectively) for the omnibus Wald tests, and from .82 to 1 for the proposed method. This can be attributed to the accumulation of the DIF effect when there were more noninvariant item parameters within an item block, making it easier for both methods to identify DIF item blocks. Furthermore, as the DIF size increased, both methods exhibited improved detection performance. Lastly, the detection performance of both methods was further enhanced with an increase in the sample size per group. For example, as shown in Table S1 of Supporting Information, under the condition of only main effect DIF, a 10% DIF proportion, a small DIF size (0.3), and only one noninvariant loading parameter within an item block, TPRs at the block level ranged from .10 (for a sample size per group of 500) to .82 (for a sample size per group of 2,000) for both methods.

Type of Noninvariant Item Parameters of the TIRT Model. The experimental results indicated that when the MFC DIF exhibited in either the loading parameter (λ) or the threshold parameter (γ), neither of the detection methods demonstrated a significant difference in their ability to detect DIF. However, when the MFC DIF exhibited in both the loading and threshold parameters, both methods performed better than the other two factor levels. This was primarily due to the superposition of DIF effects on both parameters within an item block, which increased the likelihood of being correctly flagged as a DIF item block.

To improve the clarity and understanding of the results, we also generated Receiver Operating Characteristic (ROC) curves across different experimental conditions. In this study, the area under the curve (AUC) assesses the effectiveness of different MFC DIF detection methods in identifying DIF item blocks. A larger AUC indicates better detection performance. With a sample size per group of 2,000, a DIF proportion of 10%, a large DIF size (0.6), and two item parameters showing DIF within an item block under a specific noninvariant TIRT item parameter type, Figure 3 presents ROC curves under different conditions. These curves included different conditions such as MFC DIF form, detection method, and noninvariant item parameter type, both at the block level and the block-variable combination level. ROC curves for all other conditions are shown in Figures S1 to S17 in the Supporting Information. Overall, Figure 3 shows the better performance of the proposed method in detecting interactive DIF compared to the existing OWT method.

OPEN IN VIEWER

Figure 3.

ROC curves across various conditions at a sample size per group of 2,000, a 10% DIF proportion, with a large DIF size (0.6) and two item parameters exhibiting DIF within an item block under a specific noninvariant TIRT item parameter type in Study 1.

Study 2

Data Generation

Although the data generation procedures for this study were largely the same as those used in Study 1, there were some notable differences. First, for our latent trait vector generation, we retrieved the true correlation matrix of traits from the NEO-PIR (McCrae & Costa, 1992) and represented it as:

Σ θ = [ 1 − 0.21 − 0.53 − 0.25 0 − 0.21 1 0.27 0 0.4 − 0.53 0.27 2 0.24 0 − 0.25 0 0.24 1 0 0 0.4 0 0 1 ] .

In addition, the loading and threshold parameters were randomly sampled from U(0.8, 1.3) and U(−1, 1), respectively, based on Lee et al. (2021). The generated parameters are shown in the Table S8 of Supporting Information. Finally, the details about how the item parameters of DIF blocks of the focal group were manipulated under different DIF forms were presented in Table 6.

OPEN IN VIEWER

Table 6.

True Simulated Differences of Item Parameters for Different DIF Forms Under Different Groups in Study 2.

DIF block code	Differences of Item Parameters
	DIF Form 1	DIF Form 2	DIF Form 3
{ 3 , 4 } or { 3 , 4 , 5 , 6 , 7 , 8 }	z ⋅ I ( x 4 ≥ 3 )	{ z ⋅ I ( { x 3 = 0 } ∩ { x 4 < 3 } ) z ⋅ I ( { x 3 = 1 } ∩ { x 4 ≥ 3 } )	z ⋅ I ( x 3 = 1 ) + z ⋅ I ( { x 3 = 1 } ∩ { x 4 ≥ 3 } )

Note: z = value of DIF size, DIF form 1 = only the main effect DIF, DIF form 2 = only the interactive DIF, DIF form 3 = both main effect DIF and interactive DIF.

Results of Study 2

The results of the 30% DIF proportion condition are presented in Table 7 due to space limitations, as the 10% and 30% DIF proportions showed similarities. In addition, Table 7 shows the simulation results using only the Number of Covariates and their Levels & DIF Effect Size (NCLD) strategy for Type I error control, as it performs better than other strategies in terms of power and Type I error control. Full results can be found in Tables S17 to S24 in the online supporting information. Overall, the proposed method showed a favorable potential for detecting DIF with the ordered covariate, as shown in Table 7. It also showed promising results in dealing with different DIF forms and in the simultaneous identification of DIF item blocks and DIF-inducing covariates.

OPEN IN VIEWER

Table 7.

Simulation Results Under Different Conditions When the DIF Size is Large (Small) and the Percentage of Blocks That Exhibit DIF Is 30% in Study 2.

MFC DIF Form	Total Sample Size	Results	Indicators	λ			γ			both
				1	2	3	1	2	3	1	2	3
Only the main effect DIF	1,000	Power	TPRB	.41 (.03)	.76 (.13)	.90 (.24)	.50 (.04)	.68 (.07)	.91 (.16)	.81 (.14)	.97 (.27)	.97 (.47)
			TPRBV	.41 (.03)	.76 (.13)	.90 (.23)	.50 (.04)	.68 (.07)	.91 (.16)	.81 (.14)	.97 (.27)	.97 (.47)
		Type I error	FPRB	.02 (.02)	.02 (.02)	.02 (.02)	.02 (.02)	.01 (.02)	.02 (.02)	.02 (.02)	.02 (.02)	.02 (.02)
			FPRBV	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
	4,000	Power	TPRB	.99 (.16)	1 (.79)	1 (.96)	.56 (.03)	.99 (.30)	1 (.66)	1 (.32)	1 (.99)	1 (1)
			TPRBV	.99 (.16)	1 (.79)	1 (.96)	.54 (.03)	.99 (.29)	1 (.66)	1 (.32)	1 (.99)	1 (1)
		Type I error	FPRB	.02 (.02)	.02 (.02)	.02 (.02)	.03 (.03)	.03 (.03)	.02 (.03)	.02 (.02)	.02 (.02)	.02 (.02)
			FPRBV	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
Only the interactive DIF	1,000	Power	TPRB	.04 (.02)	.10 (.04)	.22 (.03)	.10 (0)	.08 (.01)	.24 (.02)	.20 (.01)	.22 (.06)	.34 (.07)
			TPRBV	.04 (.02)	.10 (.03)	.22 (.03)	.10 (0)	.08 (.01)	.24 (.02)	.20 (.01)	.22 (.04)	.34 (.07)
		Type I error	FPRB	.02 (.02)	.02 (.03)	.02 (.02)	.02 (.02)	.02 (.03)	.02 (.02)	.02 (.02)	.02 (.02)	.02 (.02)
			FPRBV	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
	4,000	Power	TPRB	.63 (.17)	.73 (.37)	.86 (.52)	.58 (.16)	.73 (.32)	.88 (.36)	.83 (.33)	.92 (.53)	.98 (.67)
			TPRBV	.63 (.14)	.73 (.36)	.86 (.52)	.57 (.14)	.72 (.30)	.88 (.36)	.83 (.31)	.92 (.52)	.98 (.67)
		Type I error	FPRB	.03 (.02)	.03 (.03)	.03 (.03)	.03 (.02)	.03 (.03)	.03 (.03)	.03 (.02)	.03 (.03)	.03 (.03)
			FPRBV	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.02 (.01)	.02 (.01)
Both the main effect DIF and interactive DIF	1,000	Power	TPRB	.89 (.24)	.97 (.54)	.99 (.77)	.98 (.33)	1 (.40)	1 (.74)	.99 (.66)	1 (.86)	1 (.98)
			TPRBV	.12 (0)	.17 (.02)	.03 (.01)	.04 (0)	.08 (0)	.22 (.01)	.27 (.02)	.24 (.06)	.11 (.03)
		Type I error	FPRB	.02 (.02)	.02 (.01)	.01 (.02)	.02 (.02)	.02 (.03)	.02 (.02)	.02 (.02)	.02 (.02)	.02 (.02)
			FPRBV	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)
	4,000	Power	TPRB	1 (.70)	1 (1)	1 (1)	1 (.30)	1 (.93)	1 (1)	1 (.96)	1 (1)	1 (1)
			TPRBV	.64 (.13)	.88 (.46)	.99 (.66)	.71 (.03)	.97 (.41)	1 (.63)	.87 (.43)	.98 (.81)	.98 (.96)
		Type I error	FPRB	.02 (.03)	.02 (.02)	.03 (.02)	.03 (.03)	.02 (.03)	.02 (.03)	.02 (.02)	.03 (.03)	.03 (.03)
			FPRBV	.01 (.01)	.01 (.01)	.02 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.01)	.01 (.02)	.01 (.02)

Other important findings were as follows: First, the power trends in Study 2 across factor levels mirrored those in Study 1. Second, compared to Study 1, the performance of the proposed method for the ordered covariate in Study 2 deteriorated somewhat for the same total sample size. For example, with a total sample size of 1,000, a small DIF size (0.3) and only main effect DIF, block-level TPRs ranged from .10∼.88 in Study 1 versus .03∼.47 in Study 2. This occurred because the binary covariate would have more cases per group than the ordered covariate, holding the total sample size constant. As noted by Lee et al. (2021), sample size has a significant impact on the performance of MFC DIF detection. Therefore, for multi-category and ordered covariates, the proposed method requires the sample size assigned to each subgroup to be greater than 500 if more robust detection performance is desired. Third, the results showed that the NLCD strategy performed best among all strategies for controlling type I errors. The next best method considered the number of covariates and their levels, followed by the number of covariates and the DIF effect size. The worst control considered only the number of covariates. Two findings can be summarized from such results: (1) multiple comparisons between all levels of a single ordered covariate can potentially increase type I error rates; (2) although the p-value of the Wald statistic for an item block incorrectly flagged as having DIF is lower than the adjusted α, the average DIF effect size for that block would be lower than 0.1 in the vast majority of cases. Therefore, considering both the number of covariates and their levels, as well as DIF effect size estimates, potentially helped reduce type I errors while maintaining acceptable power as much as possible.

Study 3

In Study 3, we performed a follow-up simulation to further validate the DIF detection performance of the proposed method in the presence of the continuous covariate. Specifically, we included a binary covariate x₅ and a continuous covariate x₆. The MFC test is constructed with 10 triplets measuring three latent traits, and each trait was measured by 10 statements. Following the procedure for generating continuous covariates in existing studies (Berger & Tutz, 2016; Strobl et al., 2013), we sample the values of the continuous covariate x₆ from a standard normal distribution. The simulations for the different DIF forms were the same as in Study 2, except for the cutoff location that exhibited DIF in the continuous covariate. In this study, we set the cutoff location that exhibited DIF in the continuous covariate x₆ to 0.1, following the approach of Strobl et al. (2013). It is worth noting that, while all possible levels for multi-category covariates are countable, the levels for a continuous covariate have an infinite number of choices. Therefore, to find suitable cutoff values for dichotomizing the continuous covariate to facilitate DIF detection, we also refer to the study by Strobl et al. (2013), in which they took each possible cutoff value used to dichotomize the continuous covariate to two decimal places. In the present study, we followed this setting. We also used the item parameters and the covariance matrix used in Study 1, and the generation of latent trait vectors and response data remained consistent with Study 1. To ensure that the sample sizes assigned to the extreme levels of the continuous variable x₆ were not too small, we only considered the level of a total sample size of 4,000 used in both Study 1 and Study 2. Considering the efficiency of detecting continuous covariates with our proposed method, we only investigated six boundary conditions as listed in Table 8. Each condition was replicated 50 times.

OPEN IN VIEWER

Table 8.

Simulation Results of the Proposed Method Employing Different Strategies for Controlling Type I Errors in Study 3.

Strategies	Condition	DIF Form	DIF Size	Type of DIF Item Parameters	Number of DIF Item Parameters	The Average DIF Effect Size Estimates			TPRB	TPRBV	FPRB	FPRBV
						1st	2nd-L	2nd-R
NC	1	Form 1	0.6	λ	1	.14	.06	.06	1	1	.32	.32
	2			both	3	.32	.06	.06	1	1	.34	.35
	3	Form 2	0.6	λ	1	.09	.13	.10	.40	.33	.34	.35
	4			both	3	.09	.24	.22	.93	.87	.35	.36
	5	Form 3	0.6	λ	1	.18	.09	.14	1	.83	.32	.33
	6			both	3	.42	.09	.26	1	1	.34	.35
NCL	1	Form 1	0.6	λ	1	.14	.06	.06	1	1	.16	.16
	2			both	3	.32	.06	.06	1	1	.17	.17
	3	Form 2	0.6	λ	1	.09	.13	.10	.40	.33	.16	.16
	4			both	3	.09	.24	.22	.93	.86	.17	.17
	5	Form 3	0.6	λ	1	.18	.09	.14	1	.83	.15	.15
	6			both	3	.42	.09	.26	1	1	.17	.17
NCD	1	Form 1	0.6	λ	1	.14	.06	.06	.97	.97	.01	.01
	2			both	3	.32	.06	.06	1	1	.01	.01
	3	Form 2	0.6	λ	1	.09	.13	.10	.33	.33	.01	.01
	4			both	3	.09	.24	.22	.86	.86	.02	.02
	5	Form 3	0.6	λ	1	.18	.09	.14	1	.07	.01	.01
	6			both	3	.42	.09	.26	1	1	.01	.01
NCLD	1	Form 1	0.6	λ	1	.14	.06	.06	.97	.97	.01	.01
	2			both	3	.32	.06	.06	1	1	.01	.01
	3	Form 2	0.6	λ	1	.09	.13	.10	.33	.33	.0095	.0095
	4			both	3	.09	.24	.22	.86	.86	.0095	.0095
	5	Form 3	0.6	λ	1	.18	.09	.14	1	.07	.0095	.0095
	6			both	3	.42	.09	.26	1	1	.0095	.0095

Note: Strategies = the proposed different strategies for controlling Type I errors, Form 1 = only the main effect DIF, Form 2 = only the interactive DIF, Form 3 = both the main effect DIF and interactive DIF, NC = Number of covariates, NCL = Number of covariates and their levels, NCD = Number of covariates & DIF effect size, NCLD = Number of covariates and their levels & DIF effect size.1^st= The first split, 2^nd-L = The left node of the second split, 2^nd-R = The right node of the second split.

Table 8 shows the detection performance indices of the proposed method using different strategies for controlling Type I errors under each condition. The results for both TPR_B and TPR_BV were acceptable for most conditions. However, for both FPR_B and FPR_BV, there was a large difference between different strategies for controlling Type I errors. Specifically, the control strategy that considered the DIF effect size performed much better in controlling Type I errors than the performance when the DIF effect size was not considered. More importantly, there was no significant decrease in power when the control strategy included the DIF effect size estimate.

Instrument and Data Collection

The present study used a Chinese version of the Multidimensional Forced-choice Five Facets of Mindfulness Questionnaire (MFC-FFMQ) to assess mindfulness. The Five Facets of Mindfulness Questionnaire (FFMQ; Baer et al., 2006) is originally a 39-item Likert-type personality assessment that measures five latent traits: Act, Describe, Nonjudging, Nonreactivity, and Observe. Deng et al. (2011) investigated the psychometric properties of the Chinese version of the FFMQ (Ch-FFMQ), and they found that the Ch-FFMQ has acceptable psychometric properties and is a valid instrument for assessing mindfulness. The MFC-FFMQ is a MFC version of the Ch-FFMQ that includes 13-triplet item blocks and uses the MOLE as the scoring format (i.e., participants choose the statements most and least descriptive to them), and each statement within a triplet measures a different dimension of mindfulness.

Data for the MFC-FFMQ were obtained by recruiting 1,408 participants through a combination of online media and field recruitment. All participants were university students who voluntarily agreed to participate in the study, and their anonymity was ensured. After screening the data for quality, 1,276 valid responses (61.91% female) were retained for subsequent data analysis. Respondents ranged in age from 16 to 35 years, with a mean age of 20.15 years (SD = 1.93). Of the participants, 65.44% were from rural areas and 34.56% were from urban areas.

Data Analysis

To begin, we calculated the empirical reliability index to evaluate the reliability of MFC-FFMQ. Then, we conducted a TIRT five-factor confirmatory factor analysis (CFA) on the MFC-FFMQ's five-factor structure using the R package thurstonianIRT (Bürkner et al., 2019) to evaluate the construct validity. Next, we used the proposed method and the omnibus Wald test to investigate MFC DIF in the MFC-FFMQ, with the goal of comparing the results yielded by each method. Specifically, we examined MFC DIF with respect to two binary covariates: gender (male: 1, female: 2) and household registration (rural population: 1, urban population: 2). To ensure good control of Type I errors, we applied the effective sequential free baseline approach recommended by Lee et al. (2021) to identify an anchor subset consisting of DIF-free item blocks. After establishing anchor subsets, we performed DIF detection using the two previously mentioned methods. All statistical analyses were conducted using R version 4.1.1 (R Development Core Team, 2021), with an overall significance level of 0.05 set for DIF test statistics.

Results

MFC DIF Analysis

The sequential free baseline approach highlighted that MFC Items blocks 1, 8, and 11 were most consistent with Lee et al.'s (2021) anchor item screening criteria, consistently demonstrating DIF-free across all the covariates. Therefore, we selected these item blocks to form the anchor set for the subsequent MFC DIF detection on the remaining 10 item blocks. An overview of the final MFC-FFMQ DIF detection results obtained using the two methods is presented in Table 9, which shows a strong overlap between the two methods. We have also reported the average DIF effect size estimates within each item block for both the proposed method and the existing OWT method. As shown in Table 9, both the proposed method and the omnibus Wald test identified the same four MFC items (i.e., MFC item blocks 5, 7, 12, and 13) as exhibiting MFC DIF. To provide a visual representation of the DIF detection results, we have included the estimated trees for these DIF items in Figure 4. On analyzing the trees, we found that only the gender covariate induced MFC DIF for MFC item blocks 12 and 13, while both covariates gender and household registration induced MFC DIF for MFC item blocks 5 and 7, with an interaction between the two covariates in terms of inducing MFC DIF. MFC item blocks 5 and 7 showed both main effect DIF and interactive DIF across both covariates.

OPEN IN VIEWER

Figure 4.

The tree diagrams for the MFC item blocks 5, 7, 12 and 13 of MFC-FFMQ.

OPEN IN VIEWER

Table 9.

Comparison of MFC DIF Detection Results for the MFC-FFMQ Using the Proposed Method and the Omnibus Wald Test.

Item Block	Dimension	MFC-BSRPT				OWT
		The Detection Results	DIF Effect Size Estimates			The Detection Results	DIF Effect Size Estimates
			1st	2nd-L	2nd-R		x 1	x 2
1	D-A-O	Anchor	0	0	0	Anchor	0	0
2	D-A-O	×	.09	.18	.12	×	.09	.08
3	D-A-J	×	.12	.13	.11	×	.12	.08
4	D-J-R	×	.18	.18	.52	×	.18	.09
5	D-O-R	√	.24	.15	.17	√	.24	.18
6	D-J-R	×	.14	.13	.43	×	.14	.10
7	D-A-R	√	.26	.06	.18	√	.26	.16
8	A-O-J	Anchor	0	0	0	Anchor	0	0
9	D-J-R	√	.11	.13	.60	×	.11	.11
10	A-O-R	×	.13	.16	.13	×	.13	.13
11	O-J-R	Anchor	0	0	0	Anchor	0	0
12	A-O-J	√	.17	.20	.15	√	.17	.13
13	A-O-R	√	.17	.17	.18	√	.17	.11

Note: MFC item block 1, 8, and 11 formed an anchor subset; “Dimension” refers to the personality traits measured by three statements within each triplet of MFC-FFMQ; OWT = omnibus Wald tests, MFC-BSRPT = the proposed method;√ = flagged as a DIF item block; × = flagged as a DIF-free item block; Describe = D; Act = A; Observe = O; Nonjudging = J; Nonreactivity = R.

However, there were differences between the two detection methods in relation to MFC item block 9. The proposed method detected MFC DIF for this item block, but the OWT method did not. Figure 5 shows the recursive partitioning tree for item block 9, indicating MFC DIF caused by an interaction between the two covariates. Two solid arrows connected the two leaf nodes in the lower right corner to the internal node directly above, with dotted arrows connecting all other root nodes to internal nodes or leaf nodes. Thus, MFC item block 9 was classified as exhibiting the only interactive DIF. Notably, the omnibus Wald test did not detect MFC DIF for Item Block 9, as expected, given its limitation in identifying the interactive DIF present in the MFC tests. Specifically, the DIF detection results revealed no significant difference between urban and rural males on Item Block 9, with an average DIF effect size estimate of 0.13 within this block. However, there was a significant difference between urban and rural females on Item Block 9, with an average DIF effect size estimate of 0.6 within this block, exceeding the 0.2 threshold, indicating non-negligible DIF.

According to Costa et al. (2001), social desirability bias can lead to the endorsement of gender-relevant traits by both men and women, and they also suggested that “certain traits, such as fearfulness, may be viewed as less undesirable for women than for men” (p.131). These claims appear to be supported by the results of our study. For example, for MFC item block 12, the value of the loading parameter of the third statement (i.e., Usually when I have distressing thoughts or images, I just notice them and let them go.) was much higher for the female group than for the male group (.745 vs. .320). We found this statement to be highly socially desirable, with the potential to activate gender stereotypes for women and elicit socially desirable responses. Although Lee et al. (2021) have pointed out that the post hoc interpretation of DIF can be problematic, even though it is based on speculation, this does not prevent us from concluding that the proposed method has the ability to identify more MFC DIF forms.

(Figure 5).

OPEN IN VIEWER

Figure 5.

The tree diagram for the MFC item block 9 of MFC-FFMQ. Note: The solid line indicates the presence of MFC DIF in the current node and the dashed line indicates the absence of MFC DIF in the current node. Estimated TIRT-based item parameters (i.e., λ and γ) are given in each leaf of the tree.

Summary of the Present Study

Detecting DIF for MFC tests is important for maintaining the fairness and validity of the tests. However, existing methods to detect DIF for MFC tests all have limitations (see the introduction section for details). The manifestation of DIF in MFC tests is more complex than in traditional single-stimulus tests, making DIF detection in MFC tests more challenging.

In this study, we introduce a MFC DIF detection method called MFC-BSPRT that uses the recursive partitioning technique to address the limitations of current MFC DIF detection approaches. To investigate the performance of the proposed method, we conducted three carefully designed simulation studies. These three studies aim to accomplish the following two tasks: (1) A comprehensive comparison of the detection performance of the existing omnibus Wald tests and the proposed method when the covariates used for MFC DIF detection are all binary. The experimental conditions include varying MFC DIF forms, sample sizes per group, percentages of MFC DIF item blocks, types of noninvariant item parameters of the TIRT model, DIF sizes, and numbers of noninvariant item parameters under a certain noninvariant item parameter type within an item block; (2) A comprehensive evaluation of the detection performance of the proposed DIF detection method for ordered covariates and continuous covariates. The experimental conditions include different percentages of item blocks that exhibit DIF, types of noninvariant item parameters of the TIRT model, DIF sizes, numbers of noninvariant item parameters under a certain noninvariant item parameter type within an item block, MFC DIF forms, and total sample sizes. In addition to the simulation studies, we conducted an empirical study to demonstrate the feasibility of the proposed method in a practical setting, and provided a comprehensive comparison of our framework with the existing OWT approach.

The main findings of this study are as follows. First, simulation studies demonstrated that our proposed method effectively controlled Type I errors and exhibited superior detection power compared to the existing OWT method under most experimental conditions. Second, regarding MFC DIF forms, our proposed method exhibited comparable detection performance to the existing OWT method in identifying main effect DIF, while also demonstrating superior detection power in detecting interactive DIF. Third, concerning the types of DIF-induced covariates that can be handled, the proposed method exhibited a robust detection power identifying MFC DIF for binary covariates, as well as demonstrating good detection power and adequate Type I error control for multi-category, ordered, and continuous covariates, surpassing the existing OWT method's inadequate performance in addressing these effectively. Fourth, in the empirical study on 12 out of 13 item blocks of the MFC-FFMQ, our proposed method displayed the same DIF detection as the existing OWT method.

Advantages of Using the MFC-BSRPT Approach for Developing and Validating MFC Tests

The increased MFC DIF detection power. The MFC-BSRPT approach demonstrates higher sensitivity in identifying both the item blocks and covariates responsible for DIF than the existing OWT method. Specifically, compared to the existing OWT method, MFC-BSRPT not only identifies items exhibiting DIF more accurately, but also flags covariates that cause DIF more effectively. This advancement is particularly valuable as it addresses a limitation seen in the OWT method, where the identification of covariates associated with DIF may not be robust enough.

Recommendations for Applying the MFC-BSRPT Approach in Organizational Applications and Research

Provision of R codes and detailed instructions. To facilitate the use of the proposed method, we provide an online resource (accessed via https://osf.io/b98dy/). The resource includes the following main components: (1) A detailed step-by-step tutorial on how to perform the MFC DIF analysis using our proposed method; (2) annotated syntax for performing MFC DIF analysis in R using our method; and (3) demo response data and demo code used to demonstrate the analysis process.

Recommendation on sample sizes. Consistent with the findings of Lee et al. (2021), a larger sample size is expected to lead to a higher DIF detection power in our method. Our simulation results show that the detection performance of the proposed method is directly affected by the sample size per group, and the performance is acceptable under the vast majority of conditions when the sample size is above 1,000 per group, while the method does not perform as well with small samples of less than 500 respondents per group. We recommend a sample size of 1,000 or more per group be used to achieve adequate detection accuracy. Note that here the recommendation is for per group, and does not automatically translate to a total sample size of 2,000. It is possible that the sample sizes of the two groups of a binary covariate do not come even; and for multi-category, ordered, or continuous covariates, the MFC-BSRPT algorithm can split the samples into two groups of different sizes. These lead to a smaller than 1,000 sample for one of the groups if the total sample size is 2,000. Hence, it is recommended that (1) when all covariates are binary, the total sample size is chosen based on the distributions of the categories to ensure at least 1000 per group; (2) when some covariates are multi-group, ordered, or continuous, because the splits cannot be predicted, use a total sample size larger than 2,000.

Recommendation on selecting MFC DIF detection methods. When selecting a MFC DIF detection method, one should keep in mind that the proposed approach does not universally outperform the omnibus Wald test. Therefore, we recommend choosing the detection method based on the specific situation. First, since the omnibus Wald test can effectively detect the main effect DIF, it is recommended as the preferred method when examining only the main effect DIF. Second, the sample size is a key factor affecting the effectiveness of all MFC DIF detection methods. Regardless of the method used, a sample size too small can lead to misinterpretation of the detection results. When sample sizes are inadequate, over-reliance on DIF results should be avoided. Increasing the sample size remains the optimal way to increase the detection power. Third, the detection method proposed in this study is more complicated than the traditional method (i.e., OWT method) because it considers the more complex interactive DIF. Therefore, researchers can choose the OWT method or the proposed method according to their needs when weighing the detection performance and computational efficiency. For the investigation of interactive DIF, the proposed approach is preferred. Finally, with its strengths in handling multi-category, ordered, or continuous covariates, the proposed method is recommended when analyzing such covariates. Given that the current methods for detecting MFC DIF face limitations such as needing to arbitrarily dichotomize the grouping covariates and omitting interactive DIF, our proposed method is expected to be a more robust alternative overall.

Dealing with the flagged MFC DIF item blocks. According to Lee et al. (2021) and Nye (2011), an item block that exhibits non-negligible MFC DIF would have both of the following characteristics: (1) the average DIF effect size of the block is greater than or equal to 0.2; and (2) the p-value for the Wald statistic of this item block is lower than the adjusted α level. Therefore, for item blocks with the above characteristics, we recommend that potential users substantially modify such item blocks. If the modification is too difficult, removing the block from the test is recommended. As for item blocks that have a significant p-value but an average DIF effect size of less than 0.2, they can be included in the test or used with slight modifications. It should be emphasized, however, that we do not advocate for relying solely on the results of the statistical test to draw conclusions about whether these item blocks exhibit MFC DIF. A more prudent approach is to first use the existing detection methods to initially screen the item blocks most likely to exhibit DIF, then to invite experts to conduct a more deliberate and thorough review of these flagged item blocks, and finally to determine whether the flagged item block does exhibit DIF.

Limitations and Directions for Future Study

Although this study found encouraging results demonstrating the effectiveness of the proposed method, there is room for improvement. First, although the proposed method showed potential in correctly identifying DIF item blocks and DIF-inducing covariates, it does not yet specify the exact level of MFC DIF manifestation, whether it is at the pairwise comparison level, the statement level, or both. Therefore, future research could explore more fine-grained ways to detect and distinguish at which level MFC DIF manifests based on the proposed method.

Second, our proposed method relies on the Wald statistic to split response data, which may not perform as well with smaller sample sizes, as suggested by our simulation results and other recent research (Lee et al., 2021). One possible direction for future research is to develop new test statistics for MFC DIF detection or to adapt existing single-stimulus-based DIF test statistics to improve the detection performance in small sample conditions.

Third, given the difficulty of obtaining representative samples in high-stakes talent selection settings, we used only a sample of university students in our empirical study. University students tend to be younger, more educated, less experienced, and come from a narrower range of backgrounds than real candidates for high-stakes personnel selection. Therefore, findings based only on student samples may not generalize well to real personnel selection contexts. Further research is needed to collect more diverse, representative samples from organizational settings.

Fourth, like the existing studies on IRT-based DIF detection using the recursive partitioning technique, our proposed method currently only performs binary splits. In future studies, researchers can consider the problem of multivariate splits.

Finally, our proposed method currently operates only under the dominance model, while its applicability to the ideal point model is unknown. Therefore, future research can explore the proposed method's performance under those ideal point models, such as the widely used multi-unidimensional pairwise-preference model (MUPP-GGUM; Stark et al., 2005), which has been applied to the MFC computerized adaptive testing algorithm and several personality tests for selection in the U.S. military (Stark et al., 2012).