Random Responding Detection in Two Alternative Forced Choice Tests: l z vs. Optimal Appropriateness Measurement

Abstract

Multidimensional forced choice (MFC) test formats are commonly used as an alternative to traditional rating scale formats to reduce aberrant responding, especially faking in high-stakes settings. However, MFC remains susceptible to random responding, particularly in low-stakes settings where respondents may be insufficiently motivated and in high-stakes settings where some assessments may be viewed as less consequential. To ensure the validity of inferences drawn from MFC data, effective methods for detecting random responding are needed. This research contributes to the MFC literature on aberrant responding detection by evaluating the effectiveness of the item response theory (IRT)-based person fit statistic l_z for detecting random responding in multi-unidimensional pairwise preference (MUPP)-based MFC tests, using optimal appropriateness measurement (OAM) as a theoretical benchmark. Two simulation studies were conducted. Study 1 compared l_z with OAM, and Study 2 examined l_z in a broader simulation design. Results showed that (1) higher proportions of randomly answered items, longer tests, and the use of empirical critical values were associated with greater detection power for l_z, (2) the proportion of aberrant respondents did not affect l_z performance, and (3) OAM outperformed l_z only when the random responding model was correctly specified, a condition that can be realized in simulation but may not hold in applied testing contexts. Overall, the findings support the use of l_z with empirical critical values as a practical method for detecting random responding in MUPP-based MFC tests.

Keywords

aberrant responding detection random responding multidimensional forced choice (MFC)multi-unidimensional pairwise preference (MUPP)optimal appropriateness measurement

Introduction

Noncognitive constructs such as personality are commonly assessed in organizational research and practice because of their criterion-related validities in predicting critical outcomes in the workplace (e.g., Barrick & Mount, 1991) and the reduced adverse impact when including them in selection processes (Ployhart & Holtz, 2008). They are often assessed using Likert-type measures where respondents are presented with a set of single statements and are asked to indicate their level of agreement by choosing response options such as strongly disagree, disagree, agree, or strongly agree. However, aberrant responding has long been a concern with Likert-type measures. Common types of aberrant responding include faking, random responding, patterned responding, and rapid responding. Faking occurs when respondents intentionally distort their responses to present themselves in a favorable (faking good) or unfavorable (faking bad) light. Random responding occurs when respondents select response options arbitrarily regardless of item content. Patterned responding involves choosing response options in a systematic way across items, such as alternating between agree and disagree or agree and strongly agree on consecutive items. Rapid responding occurs when respondents answer items too quickly without carefully processing the content. In high-stakes contexts such as job applications where applicants are motivated to present themselves favorably, faking is likely to occur, particularly on transparent Likert-type items (e.g., “I am a reliable worker”). In low-stakes situations such as research studies where inaccurate responses carry little consequence, participants may engage in random, patterned, or rapid responding, especially when measures are lengthy or participants lack motivation. Aberrant responding affects measurement accuracy because observed scores no longer reflect respondents’ true trait levels. This poses a threat to the construct validity of a measure and consequently the appropriateness of any inferences we draw from its scores. In high-stakes selection contexts, such inferences typically involve the rank order of respondents, which can lead to incorrect selection decisions that may turn out to be costly. In low-stakes research contexts, inferences may include inaccurate conclusions about theoretical or empirical relationships between constructs.

Over the past two decades, multidimensional forced choice (MFC) test formats have emerged as a viable alternative to Likert-type measures to prevent aberrant responding. In MFC tests, respondents are presented with items consisting of two or more statements and are instructed to pick the statement in each item that best represents them (PICK), select one statement that most represents them and another that least represents them (MOLE), or rank the statements from most to least represent them (RANK). An example MFC triplet item using the RANK response instruction is shown below:

Rank the statements from most like you (1) to least like you (3).

• I get along well with others.

• I set high personal standards.

• I like new challenges.

Within MFC items, statements are often matched on social desirability and extremity. By forcing respondents to choose or rank between similarly appealing statements within items, faking becomes more difficult. Although MFC test formats have been shown to be more resistant to faking (Cao & Drasgow, 2019; Wetzel et al., 2020), they are not immune to other types of aberrant responding. Similar to Likert-type measures, in low-stakes situations where respondents are not fully motivated to respond accurately or in high-stakes settings where respondents must complete multiple assessments and may view some as less consequential, they may select statements randomly.

A variety of methods have been developed to detect aberrant responding in Likert-type measures, with outlier analyses and consistency indices among the most commonly used approaches (e.g., Arthur et al., 2021; Meade & Craig, 2012). Outlier analyses flag individual response patterns that differ substantially from the distribution of normal responses, whereas consistency indices assess the internal consistency of responses across items within each respondent based on the expectation that items measuring the same construct should be positively correlated. Unlike normal responses that reflect the underlying construct being measured, aberrant responding produces response patterns that differ from normal responses and reduces internal consistency, making outlier analyses and consistency indices effective for detection. Common examples of these approaches include the Mahalanobis distance and the even-odd index. However, these methods are less suitable for MFC measures due to the comparative and multidimensional nature of MFC test formats. Because MFC response data are typically analyzed using item response theory (IRT) models to derive normative scores that allow comparisons and rankings across respondents, detection methods developed within the IRT framework are especially appropriate. This research therefore focuses on IRT-based methods.

Within the framework of IRT, various appropriateness measurement indices, now referred to as person fit statistics, have been developed to detect aberrant responding. These methods examine the consistency between each respondent’s item responses and the responses expected by a chosen IRT model and they are sensitive to deviations from expected response probabilities. One such approach is optimal appropriateness measurement (OAM; Levine & Drasgow, 1988), which uses the likelihood ratio test to classify a response pattern as normal or aberrant based on the IRT model chosen for parameter estimation and a hypothesized model for aberrant responding. The likelihood ratio test has a well-established foundation in statistical hypothesis testing (Drasgow et al., 1987; Levine & Drasgow, 1984; Neyman & Pearson, 1933). Under the Neyman-Pearson lemma, it is the most powerful test for distinguishing between two simple hypotheses when the competing models are correctly specified. In the context of person fit analysis, OAM operationalizes this principle by directly contrasting the likelihood of responses under a “normal” responding model and an “aberrant” responding model. However, the theoretical power of OAM only materializes when both the normal and aberrant responding models are correctly specified. This may limit its practical use because the type(s) of aberrant responding that respondents engage in while completing tests is rarely known a priori, and it may differ across respondents and points within a single testing session. Nevertheless, when there is a strong basis for specifying the selected models, OAM has been used to address applied questions in large-scale testing programs, such as detecting unusual response patterns in Scholastic Aptitude Test (SAT) data (Drasgow et al., 1985), exploring the effects of unmotivated responding on the dimensionality of the Certified Public Accountants (CPA) Exam (Stark et al., 2005a, 2005b), and predicting soldier attrition using the U.S. Army’s Assessment of Individual Motivation (AIM; Stark et al., 2011; White & Young, 1998).

An alternative approach is the l_z statistic (Drasgow et al., 1985), a standardized likelihood based on a “normal” responding model. l_z is among the most commonly used person fit statistics and has demonstrated effectiveness in detecting various types of aberrant responding (e.g., Conijn et al., 2014, 2016; Drasgow et al., 1987; Felt et al., 2017; Nering, 1997). Although widely used, only a limited number of studies have investigated l_z in MFC contexts (Kim & Moses, 2018; Lee et al., 2024). This research extends that literature by examining the efficacy of l_z relative to OAM in MFC tests, with a focus on random responding. OAM achieves optimal detection power for aberrant responding when both the normal and aberrant responding models are correctly specified, a condition that can be realized in simulation. Comparing l_z to OAM in simulation studies therefore allows us to assess how closely l_z approaches optimal detection performance in MFC tests. Thus, the purpose of this comparison is not to promote OAM as an operational tool but to use it as a theoretical benchmark for assessing the effectiveness of l_z in detecting random responding. To compare the two methods across diverse conditions, we manipulated the proportion of items answered randomly, test composition, critical value type, nominal alpha level, and the aberrant responding model used in OAM. The Multi-Unidimensional Pairwise Preference model (MUPP; Stark et al., 2005a, 2005b) was used to characterize the MFC response process, and the dichotomous version of the Generalized Graded Unfolding Model (GGUM; Roberts et al., 2000) was used to characterize the response process for single statements composing MFC items. The MUPP and GGUM IRT models, as well as the implementation of OAM and l _z within the MUPP framework, are described in detail below.

The MUPP Model

MFC tests consist of pairs of statements (s and t) and respondents are asked to choose the statement in each pair that is more like themselves. The preference decision is operationalized as agreeing with one statement and disagreeing with the other, with these agree/disagree probabilities depending on respondents’ trait levels and the statement parameters estimated using an appropriate unidimensional IRT model. Suppose X is the item response such that X = 1 indicates that the respondent prefers statement s to statement t and X = 0 indicates otherwise, then the corresponding preferential choice probability, $P_{(s > t)}$ , according to the MUPP model is defined as

P_{(s > t)} (X = 1 | θ_{d_{s}}, θ_{d_{t}}) = \frac{P_{s t} (1, 0)}{P_{s t} (1, 0) + P_{s t} (0, 1)} \approx \frac{P_{s} (1) P_{t} (0)}{P_{s} (1) P_{t} (0) + P_{s} (0) P_{t} (1)}

(1)

where

θ_{d_{s}}

and

θ_{d_{t}}

are the respondent’s latent trait values on dimension

d_{s}

and

d_{t}

, respectively, and

d_{s}

and

d_{t}

are latent trait dimensions associated with statements s and t, respectively.

P_{s t} (1, 0)

is the joint probability of endorsing statement s and not endorsing statement t, whereas

P_{s t} (0, 1)

is the joint probability of endorsing statement t and not endorsing statement s.

P_{s} (1)

and

P_{t} (1)

are the probabilities of endorsing statements s and t, respectively, and were computed using the dichotomous version of the GGUM described below. The probabilities of not endorsing the statements were calculated as

P_{s} (0) = 1 - P_{s} (1)

and

P_{t} (0) = 1 - P_{t} (1)

The GGUM

The GGUM is an ideal point IRT model that assumes the probability of agreeing with a statement is a function of the distance between a respondent’s latent trait level and the statement’s location on the trait continuum. The closer a statement is to a respondent’s latent trait level, the more likely the statement is to be endorsed. Although the GGUM is often used with rating scales that have an ordered polytomous response format, a simplified version for dichotomous (agree/disagree) responses is most commonly used with the MUPP. The probability of agreeing with a statement, (Z = 1), is

P (Z = 1 | θ) = \frac{\exp (α [(θ - δ) - τ]) + \exp (α [2 (θ - δ) - τ])}{γ}

(2)

where

γ

1 + \exp (α [3 (θ - δ)]) + \exp (α [(θ - δ) - τ]) + \exp (α [2 (θ - δ) - τ])

. Note that θ is the respondent’s latent trait value on the dimension measured by the statement, and

α

δ

, and

τ

are the statement discrimination, location, and threshold parameters, respectively. For more comprehensive descriptions of the MUPP and GGUM IRT models, readers are referred to Stark et al., (2005a, 2005b) and Roberts et al. (2000), respectively.

OAM

In OAM, both normal and aberrant responding models must be specified. The conditional likelihoods of a response pattern for the two models denoted as ${L (u | θ_{d_{s}}, θ_{d_{t}})}_{N o r m a l}$ and ${L (u | θ_{d_{s}}, θ_{d_{t}})}_{A b e r r a n t}$ are computed as follows:

{L (u | θ_{d_{s}}, θ_{d_{t}})}_{N o r m a l} = \prod_{i = 1}^{I} {P_{(s > t) N o r m a l} (X_{i} = 1 | θ_{d_{s}}, θ_{d_{t}})}^{X_{i}} {({1 - P}_{(s > t) N o r m a l} (X_{i} = 1 | θ_{d_{s}}, θ_{d_{t}}))}^{1 - X_{i}}

(3.1)

{L (u | θ_{d_{s}}, θ_{d_{t}})}_{A b e r r a n t} = \prod_{i = 1}^{I} {P_{(s > t) A b e r r a n t} (X_{i} = 1 | θ_{d_{s}}, θ_{d_{t}})}^{X_{i}} {({1 - P}_{(s > t) A b e r r a n t} (X_{i} = 1 | θ_{d_{s}}, θ_{d_{t}}))}^{{1 - X}_{i}}

(3.2)

where

u

is the item response vector, I is the total number of MFC items,

P_{(s > t) N o r m a l}

is the MUPP response probability computed via Equations (1) and (2), and

P_{(s > t) A b e r r a n t}

is .50 for items hypothesized to be answered randomly. The conditional likelihood equations are used to calculate the marginal likelihoods denoted as

{L (u)}_{N o r m a l}

and

{L (u)}_{A b e r r a n t}

as follows:

{L (u)}_{N o r m a l} = \int \dots \int {L (u | θ_{d_{s}}, θ_{d_{t}})}_{N o r m a l} f (θ_{1}) \dots f (θ_{D}) d θ_{1} \dots d θ_{D}

(3.3)

{L (u)}_{A b e r r a n t} = \int \dots \int {L (u | θ_{d_{s}}, θ_{d_{t}})}_{A b e r r a n t} f (θ_{1}) \dots f (θ_{D}) d θ_{1} \dots d θ_{D}

(3.4)

Finally, the ratio of the two marginal likelihoods is the OAM for each respondent.

λ (u) = \frac{{L (u)}_{A b e r r a n t}}{{L (u)}_{N o r m a l}}

(3.5)

The calculated likelihood ratio is compared with a critical value. If it is larger than the critical value, the respondent is identified as an aberrant respondent.

The l_z Statistic

The l_z statistic is a standardized version of the l₀ index (Levine & Rubin, 1979). To obtain l_z, the loglikelihood of a response pattern $l_{0}$ is calculated as follows:

l_{0} = \sum_{i = 1}^{I} X_{i} \log P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}) + (1 - X_{i}) \log (1 {- P}_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}))

(4.1)

where

P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}})

is the MUPP response probability with

{\hat{θ}}_{d_{s}}

and

{\hat{θ}}_{d_{t}}

are estimated trait scores for a respondent on dimensions

d_{s}

and

d_{t}

, respectively. The approximated expectation and variance of the loglikelihood are then:

E (l_{0}) = \sum_{i = 1}^{I} P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}) \log P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}) + (1 {- P}_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}})) \log (1 {- P}_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}))

(4.2)

V a r (l_{0}) = \sum_{i = 1}^{I} P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}}) (1 {- P}_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}})) {\log \frac{P_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}})}{(1 {- P}_{(s > t)} (X_{i} = 1 | {\hat{θ}}_{d_{s}}, {\hat{θ}}_{d_{t}})}}^{2}

(4.3)

Finally, the l_z statistic is then obtained by standardizing l₀:

l_{z} = \frac{l_{0} - E (l_{0})}{\sqrt{V a r (l_{0})}}

(4.4)

The calculated l_z statistic is compared with a critical value. If it is less than the critical value, the respondent is identified as an aberrant respondent.

Study 1

Simulation Study Design

Five factors were manipulated: (1) proportion of items answered randomly (20%, 50%, 100%), (2) test composition (6D-48I, 12D-96I [D = dimensions; I = items]), (3) l_z critical value type (theoretical, empirical), (4) nominal alpha level (.01, .05), and (5) aberrant responding model used in OAM (OAM_unknown, in which all items were modeled as answered aberrantly, OAM_known, in which only the simulated aberrant items were modeled as answered aberrantly). This yielded 48 conditions. Sample size was fixed at 800 and the proportion of aberrant respondents was fixed at 20%. The 20% and 50% random responding conditions were included to reflect realistic scenarios, whereas the 100% condition was included as a boundary case to benchmark the upper limit of detection performance of l_z and OAM. Similarly, the OAM_known condition was included as an ideal scenario to examine the maximum detection performance of OAM under optimal conditions. The proportion of aberrant respondents was fixed because it was expected to have no effect on the performance of l_z and OAM. Trait scores were estimated independently for each respondent using only that respondent’s response data, and statement parameters were pre-estimated from single statement normal response data and treated as fixed when calculating OAM and l_z (Stark et al., 2016). Therefore, the proportion of aberrant respondents affected neither trait score estimation nor statement parameter estimation used in the calculation of l_z and OAM in this study. Due to the long runtime required to calculate OAM, ranging from 4 to 11 hours, 20 replications per condition were performed.

Test Specification

The test specifications developed by Tu et al., (2023a, 2023b) for the 6D-48I and 12D-96I tests were used directly. Specifically, statement location parameters (δ) were sampled from a uniform distribution, U[−2, 2]; threshold parameters (

τ

) were sampled from U[−1.40, −.40]; and discrimination parameters (

α

) were sampled from a combination of uniform distributions, consistent with U[.75, 1.75]. The 6D-48I test was created by systematically selecting from the pool of true statement parameters such that the mean and standard deviation of the discrimination (

α

), location (δ), and threshold (τ) parameters were approximately equal for every dimension measured in the test. The 12D-96I test was created by duplicating the 6D-48I test parameters and modifying the dimension numbers to reflect the higher dimensionality. To ensure that each dimension was measured similarly well, each dimension was represented the same number of times within a test. Tables 1 and 2 provide detailed information on the true statement parameters and test specifications.

Table 1.

Test Specifications for the 6D-48I MUPP Test

Item	s	t	$α_{s}$	$δ_{s}$	$τ_{s}$	$α_{t}$	$δ_{t}$	$τ_{t}$	Item	s	t	$α_{s}$	$δ_{s}$	$τ_{s}$	$α_{t}$	$δ_{t}$	$τ_{t}$
1	1	6	1.81	.60	−1.37	.75	−1.78	−1.17	25	1	5	.75	1.82	−1.37	.78	−1.61	−1.12
2	1	3	1.83	−1.99	−.55	1.88	−.69	−1.24	26	4	5	1.80	−.68	−1.31	2.13	−1.03	−.91
3	3	4	.81	−1.49	−.55	1.31	−1.12	−.41	27	2	4	.97	.56	−.98	1.88	1.23	−1.12
4	2	4	1.77	1.36	−.65	1.83	−1.37	−1.31	28	3	6	2.06	−.1	−.97	1.94	.32	−.51
5	3	4	1.98	.85	−.76	1.51	1.78	−1.28	29	2	5	1.98	−1.25	−1.12	1.92	−.29	−.68
6	3	6	1.33	1.60	−.51	1.25	1.45	−1.12	30	4	5	.82	−1.91	−1.22	1.48	.98	−.65
7	3	6	1.76	1.92	−.94	1.45	−1.08	−.95	31	1	2	2.00	1.25	−.73	1.34	.91	−1.21
8	1	2	1.35	1.08	−.69	1.49	−.59	−.53	32	1	3	2.01	−1.63	−.80	1.52	−.48	−1.13
9	1	5	1.56	−.22	−1.22	1.26	1.35	−.46	33	2	6	1.94	1.93	−1.13	1.97	−1.54	−.94
10	2	6	1.88	−1.71	−1.20	1.83	−.49	−.56	34	4	6	1.85	.37	−.58	2.08	.99	−.69
11	1	4	1.78	−1.16	−.91	2.11	1.45	−.46	35	2	5	1.76	−1.47	−.95	1.92	−1.72	−1.33
12	5	6	1.84	1.89	−.54	1.93	1.87	−.97	36	3	5	1.76	−1.87	−.85	1.75	.18	−1.22
13	5	2	1.81	.60	−1.37	.75	−1.78	−1.17	37	5	1	.75	1.82	−1.37	.78	−1.61	−1.12
14	5	4	1.83	−1.99	−.55	1.88	−.69	−1.24	38	3	1	1.80	−.68	−1.31	2.13	−1.03	−.91
15	4	3	.81	−1.49	−.55	1.31	−1.12	−.41	39	6	3	.97	.56	−.98	1.88	1.23	−1.12
16	6	3	1.77	1.36	−.65	1.83	−1.37	−1.31	40	4	3	2.06	−.10	−.97	1.94	.32	−.51
17	4	3	1.98	.85	−.76	1.51	1.78	−1.28	41	6	1	1.98	−1.25	−1.12	1.92	−.29	−.68
18	4	2	1.33	1.60	−.51	1.25	1.45	−1.12	42	3	1	.82	−1.91	−1.22	1.48	.98	−.65
19	4	2	1.76	1.92	−.94	1.45	−1.08	−.95	43	5	6	2.00	1.25	−.73	1.34	.91	−1.21
20	5	6	1.35	1.08	−.69	1.49	−.59	−.53	44	5	4	2.01	−1.63	−.80	1.52	−.48	−1.13
21	5	1	1.56	−.22	−1.22	1.26	1.35	−.46	45	6	2	1.94	1.93	−1.13	1.97	−1.54	−.94
22	6	2	1.88	−1.71	−1.20	1.83	−.49	−.56	46	3	2	1.85	.37	−.58	2.08	.99	−.69
23	5	3	1.78	−1.16	−.91	2.11	1.45	−.46	47	6	1	1.76	−1.47	−.95	1.92	−1.72	−1.33
24	1	2	1.84	1.89	−.54	1.93	1.87	−.97	48	4	1	1.76	−1.87	−.85	1.75	.18	−1.22

Note. s and t indicate the dimensions represented in the respective pairwise preference items, and $α_{s}$ , $δ_{s}$ , $τ_{s}$ , $α_{t}$ , $δ_{t}$ , and $τ_{t}$ show the corresponding statement parameters.

Table 2.

Test Specifications for the 12D-96I MUPP Test

Item	s	t	$α_{s}$	$δ_{s}$	$τ_{s}$	$α_{t}$	$δ_{t}$	$τ_{t}$	Item	s	t	$α_{s}$	$δ_{s}$	$τ_{s}$	$α_{t}$	$δ_{t}$	$τ_{t}$
1	1	6	1.81	.60	−1.37	.75	−1.78	−1.17	49	1	5	.75	1.82	−1.37	.78	−1.61	−1.12
2	1	3	1.83	−1.99	−.55	1.88	−.69	−1.24	50	4	5	1.80	−.68	−1.31	2.13	−1.03	−.91
3	3	4	.81	−1.49	−.55	1.31	−1.12	−.41	51	2	4	.97	.56	−.98	1.88	1.23	−1.12
4	2	4	1.77	1.36	−.65	1.83	−1.37	−1.31	52	3	6	2.06	−.10	−.97	1.94	.32	−.51
5	3	4	1.98	.85	−.76	1.51	1.78	−1.28	53	2	5	1.98	−1.25	−1.12	1.92	−.29	−.68
6	3	6	1.33	1.60	−.51	1.25	1.45	−1.12	54	4	5	.82	−1.91	−1.22	1.48	.98	−.65
7	3	6	1.76	1.92	−.94	1.45	−1.08	−.95	55	1	2	2.00	1.25	−.73	1.34	.91	−1.21
8	1	2	1.35	1.08	−.69	1.49	−.59	−.53	56	1	3	2.01	−1.63	−.80	1.52	−.48	−1.13
9	1	5	1.56	−.22	−1.22	1.26	1.35	−.46	57	2	6	1.94	1.93	−1.13	1.97	−1.54	−.94
10	2	6	1.88	−1.71	−1.20	1.83	−.49	−.56	58	4	6	1.85	.37	−.58	2.08	.99	−.69
11	1	4	1.78	−1.16	−.91	2.11	1.45	−.46	59	2	5	1.76	−1.47	−.95	1.92	−1.72	−1.33
12	5	6	1.84	1.89	−.54	1.93	1.87	−.97	60	3	5	1.76	−1.87	−.85	1.75	.18	−1.22
13	5	2	1.81	.60	−1.37	.75	−1.78	−1.17	61	5	1	.75	1.82	−1.37	.78	−1.61	−1.12
14	5	4	1.83	−1.99	−.55	1.88	−.69	−1.24	62	3	1	1.80	−.68	−1.31	2.13	−1.03	−.91
15	4	3	.81	−1.49	−.55	1.31	−1.12	−.41	63	6	3	.97	.56	−.98	1.88	1.23	−1.12
16	6	3	1.77	1.36	−.65	1.83	−1.37	−1.31	64	4	3	2.06	−.10	−.97	1.94	.32	−.51
17	4	3	1.98	.85	−.76	1.51	1.78	−1.28	65	6	1	1.98	−1.25	−1.12	1.92	−.29	−.68
18	4	2	1.33	1.60	−.51	1.25	1.45	−1.12	66	3	1	.82	−1.91	−1.22	1.48	.98	−.65
19	4	2	1.76	1.92	−.94	1.45	−1.08	−.95	67	5	6	2.00	1.25	−.73	1.34	.91	−1.21
20	5	6	1.35	1.08	−.69	1.49	−.59	−.53	68	5	4	2.01	−1.63	−.80	1.52	−.48	−1.13
21	5	1	1.56	−.22	−1.22	1.26	1.35	−.46	69	6	2	1.94	1.93	−1.13	1.97	−1.54	−.94
22	6	2	1.88	−1.71	−1.20	1.83	−.49	−.56	70	3	2	1.85	.37	−.58	2.08	.99	−.69
23	5	3	1.78	−1.16	−.91	2.11	1.45	−.46	71	6	1	1.76	−1.47	−.95	1.92	−1.72	−1.33
24	1	2	1.84	1.89	−.54	1.93	1.87	−.97	72	4	1	1.76	−1.87	−.85	1.75	.18	−1.22
25	7	12	1.81	.60	−1.37	.75	−1.78	−1.17	73	7	11	.75	1.82	−1.37	.78	−1.61	−1.12
26	7	9	1.83	−1.99	−.55	1.88	−.69	−1.24	74	10	11	1.80	−.68	−1.31	2.13	−1.03	−.91
27	9	10	.81	−1.49	−.55	1.31	−1.12	−.41	75	8	10	.97	.56	−.98	1.88	1.23	−1.12
28	8	10	1.77	1.36	−.65	1.83	−1.37	−1.31	76	9	12	2.06	−.10	−.97	1.94	.32	−.51
29	9	10	1.98	.85	−.76	1.51	1.78	−1.28	77	8	11	1.98	−1.25	−1.12	1.92	−.29	−.68
30	9	12	1.33	1.60	−.51	1.25	1.45	−1.12	78	10	11	.82	−1.91	−1.22	1.48	.98	−.65
31	9	12	1.76	1.92	−.94	1.45	−1.08	−.95	79	7	8	2.00	1.25	−.73	1.34	.91	−1.21
32	7	8	1.35	1.08	−.69	1.49	−.59	−.53	80	7	9	2.01	−1.63	−.80	1.52	−.48	−1.13
33	7	11	1.56	−.22	−1.22	1.26	1.35	−.46	81	8	12	1.94	1.93	−1.13	1.97	−1.54	−.94
34	8	12	1.88	−1.71	−1.20	1.83	−.49	−.56	82	10	12	1.85	.37	−.58	2.08	.99	−.69
35	7	10	1.78	−1.16	−.91	2.11	1.45	−.46	83	8	11	1.76	−1.47	−.95	1.92	−1.72	−1.33
36	11	12	1.84	1.89	−.54	1.93	1.87	−.97	84	9	11	1.76	−1.87	−.85	1.75	.18	−1.22
37	11	8	1.81	.60	−1.37	.75	−1.78	−1.17	85	11	7	.75	1.82	−1.37	.78	−1.61	−1.12
38	11	10	1.83	−1.99	−.55	1.88	−.69	−1.24	86	9	7	1.80	−.68	−1.31	2.13	−1.03	−.91
39	10	9	.81	−1.49	−.55	1.31	−1.12	−.41	87	12	9	.97	.56	−.98	1.88	1.23	−1.12
40	12	9	1.77	1.36	−.65	1.83	−1.37	−1.31	88	10	9	2.06	−.10	−.97	1.94	.32	−.51
41	10	9	1.98	.85	−.76	1.51	1.78	−1.28	89	12	7	1.98	−1.25	−1.12	1.92	−.29	−.68
42	10	8	1.33	1.60	−.51	1.25	1.45	−1.12	90	9	7	.82	−1.91	−1.22	1.48	.98	−.65
43	10	8	1.76	1.92	−.94	1.45	−1.08	−.95	91	11	12	2.00	1.25	−.73	1.34	.91	−1.21
44	11	12	1.35	1.08	−.69	1.49	−.59	−.53	92	11	10	2.01	−1.63	−.80	1.52	−.48	−1.13
45	11	7	1.56	−.22	−1.22	1.26	1.35	−.46	93	12	8	1.94	1.93	−1.13	1.97	−1.54	−.94
46	12	8	1.88	−1.71	−1.20	1.83	−.49	−.56	94	9	8	1.85	.37	−.58	2.08	.99	−.69
47	11	9	1.78	−1.16	−.91	2.11	1.45	−.46	95	12	7	1.76	−1.47	−.95	1.92	−1.72	−1.33
48	7	8	1.84	1.89	−.54	1.93	1.87	−.97	96	10	7	1.76	−1.87	−.85	1.75	.18	−1.22

Normal Response Data Generation

On each replication, person parameters ( $θ$ ) for the designated sample size and dimensionality were sampled from a multivariate standard normal distribution with zero means and zero correlations between dimensions. The correlation was set to 0 because previous studies (e.g., Tu et al., 2023a, 2023b) found that the correlation between dimensions had no effect on statement or person parameter recovery. Generated statement and person parameters were used to compute MUPP response probabilities for all items and respondents. The MFC item response data were then generated from the binomial distribution with the MUPP probabilities.

Statement Parameter Estimation

Statement parameters were pre-estimated from single statement (i.e., rating scale) response data (N = 800) and treated as “fixed.” Specifically, dichotomous single statement response data were generated based on the GGUM. The GGUM probability was computed using the corresponding true single statement parameters and person parameters. After the GGUM probabilities were obtained for all statements and respondents, single statement response data were generated from the binomial distribution with the GGUM probabilities. The generated response data for the statements representing each dimension were then calibrated, one dimension at a time, using the GGUM MCMC program (Joo et al., 2017).

Random Response Data Generation

On each replication, 20% of respondents were randomly selected as aberrant respondents. For these respondents, aberrant response data were generated by replacing their normal responses with random responses on the final portion of the test, corresponding to the designated proportion of randomly answered items. For example, in conditions where 50% of items were designated to be answered randomly, the last 50% of each aberrant respondent’s item responses were replaced with random responses, while the remaining responses were identical to the originally generated normal response data. Each random item response was generated from the binomial distribution with a probability of .50.

OAM and l_z Implementation

OAM was implemented using the R packages rstan (Stan Development Team, 2023) and bridgesampling (Gronau et al., 2017). Specifically, both the normal and aberrant responding models were fit to each respondent’s data using the stan function in rstan, with pre-estimated statement parameters held fixed. The default sampling algorithm, Hamiltonian Monte Carlo (HMC), and the default standard normal prior were used for scoring. The resulting stanfit objects were then passed to the bridge_sampler function in bridgesampling to calculate marginal likelihoods, as in Equations (3.1)–(3.4). The bf function from bridgesampling was used to compute the likelihood ratio for OAM, as in Equation (3.5). A respondent was flagged as aberrant if their likelihood ratio exceeded the critical value. Person parameter estimates from the normal responding model stanfit object were used to calculate l_z for each respondent, as in Equations (4.1)–(4.4). A respondent was flagged as aberrant if their l_z value fell below the critical value.

Critical Values

Both theoretical and empirical critical values were used. For l_z, theoretical critical values corresponding to the 1st and 5th percentiles were obtained from the inverse normal probability table. Empirical critical values were obtained by generating a new normal response dataset (N = 800) using true statement and person parameters, calculating l_z for each respondent using statement parameters pre-estimated from single statement response data and person parameters estimated from the generated normal MFC response data, and identifying the values at the 1st and 5th percentiles. The theoretical critical values for l_z were the same across all conditions, whereas the empirical critical values varied by test composition. Because the distribution of the likelihood ratio for OAM was far from normal, only empirical critical values were used for OAM (Drasgow & Levine, 1986; Levine & Drasgow, 1988). They were obtained by calculating likelihood ratios from a new set of normal response data (N = 800) and identifying the values at the 1st and 5th percentiles. OAM empirical critical values varied by test composition, aberrant responding model used in OAM, and proportion of items answered randomly. Table 3 presents all critical values. The sample size of 800 used to derive empirical critical values is comparable to the sample size of 1000 used in previous l_z studies with FC tests (e.g., Lee et al., 2014).

Table 3.

Critical values for l_z and OAM

Method	CV type	Test comp	AI	Model type	1%	5%
l _z	Theoretical				−2.33	−1.64
	Empirical
	Study 1	6D-48I			−1.81	−.99
		12D-96I			−1.45	−.60
	Study 2	6D-48I			−1.22	−.63
		12D-96I			−.82	−.11
OAM	Empirical	6D-48I	20%	Unknown	1947.26	66.16
			20%	Known	4966.82	913.54
			50%	Unknown	1947.26	66.16
			50%	Known	2106.89	383.89
			100%	Unknown	1947.26	66.16
			100%	Known	1947.26	66.16
		12D-96I	20%	Unknown	6734.38	50.10
			20%	Known	655,114.13	78,669.84
			50%	Unknown	6734.38	50.10
			50%	Known	53,978.94	5935.38
			100%	Unknown	6734.38	50.10
			100%	Known	6734.38	50.10

Note. 1% = A nominal alpha level of .01; 5% = A nominal alpha level of .05; Test comp = Test composition (6D-48I = 6 dimensions 48 items, 12D-96 = 12 dimensions 96 items); AI = Proportion of items answered randomly; CV Type = Type of critical value (Empirical = Empirical critical values, Theoretical = Theoretical critical value); Model Type = Type of aberrant responding model used in OAM (OAM_unknown = All items were modeled being answered randomly in the aberrant responding model, OAM_known = A known set of items were modeled being answered randomly in the aberrant responding model).

Analyses

Type I error was computed as the proportion of normal respondents incorrectly flagged as aberrant. Power was computed as the proportion of aberrant respondents correctly identified as such.

Study 1 Results

Table 4 presents the results for detecting random responding in MUPP tests at the nominal alpha level of .01. Several patterns are noteworthy. First, detection performance was affected by the proportion of items answered randomly. For both l_z and OAM, power increased as the proportion of randomly answered items increased. This pattern is expected because a larger proportion of randomly answered items increases the total number of random responses, making aberrant response patterns easier to distinguish from normal response patterns. Second, detection performance was also affected by test length. For both l_z and OAM, power was higher in the longer 12D-96I tests than in the shorter 6D-48I tests. This improvement may be due to two reasons. First, when the proportion of randomly answered items is fixed, longer tests contain more randomly answered item responses, making random responding easier to detect. Second, longer tests may lead to more accurate person parameter estimation, which in turn can improve the performance of l_z. Third, the performance of l_z for random responding detection depended on critical value type. Across conditions, l_z showed higher power when empirical rather than theoretical critical values were used. For example, in the 12D-96I condition with 20% of items answered randomly, power was .34 with empirical critical values but only .14 with theoretical critical values. This finding is consistent with prior research showing that empirical critical values can improve the performance of l_z (e.g., Lee et al., 2014). Fourth, the performance of OAM depended on whether the aberrant responding model was correctly specified. When only a subset of items was answered randomly, OAM_known, which correctly specified the proportion of randomly answered items, showed higher power than OAM_unknown, which assumed that all items were answered randomly. For example, in the 12D-96I condition with 20% of items answered randomly, power was .80 for OAM_known but only .30 for OAM_unknown. However, when all items were answered randomly, OAM_unknown and OAM_known performed similarly, with power of .97, because both correctly modeled the 100% random responding condition.

Table 4.

Study 1: Power and Type I Error Rates of l_z and OAM to Detect Random Responding_1%

Test comp	AI	l _z				OAM_unknown		OAM_known
		Empirical		Theoretical		Empirical		Empirical
		Power	Type I error	Power	Type I error	Power	Type I error	Power	Type I error
6D-48I	20%	.21	.01	.12	.00	.17	.00	.51	.01
	50%	.63	.01	.50	.00	.59	.00	.84	.01
	100%	.92	.01	.86	.00	.96	.01	.96	.01
12D-96I	20%	.34	.01	.14	.00	.30	.01	.80	.01
	50%	.90	.01	.74	.00	.90	.01	.99	.01
	100%	1.00	.01	.99	.00	.97	.01	.97	.01

Note. 1% = A nominal alpha level of .01; Test comp = Test composition (6D-48I = 6 dimensions 48 items, 12D-96 = 12 dimensions 96 items); AI = Proportion of items answered randomly; Empirical = Empirical critical values; Theoretical = Theoretical critical value; OAM_unknown = All items were modeled being answered randomly in the aberrant responding model; OAM_known = A known set of items were modeled being answered randomly in the aberrant responding model.

l_z vs. OAM

Comparing l_z with OAM further clarified the conditions under which l_z was most effective. OAM_known represents a theoretical upper-bound benchmark because it assumes that both the proportion and exact items answered randomly are known a priori, a condition unlikely to be met in practice. When OAM was correctly specified in this way, it showed higher power than l_z, especially when random responding was difficult to detect. For example, in the 6D-48I condition with only 20% of items answered randomly, power was .21 for l_z with empirical critical values and .51 for OAM_known. However, as the proportion of randomly answered items and test length increased, the difference between OAM_known and l_z became smaller. For example, in the 12D-96I condition with 100% of items answered randomly, power was 1.00 for l_z with empirical critical values and .97 for OAM_known. Thus, l_z approached the theoretical maximum detection power when random responding was easier to detect, such as in the 100% random responding and longer test conditions.

OAM_unknown provides a more realistic benchmark because all items were modeled as random regardless of the true proportion of randomly answered items. This reflects the more practical reality that the items answered randomly are not known a priori. The results showed that the performance of l_z relative to OAM_unknown depended on the critical value type, the proportion of items answered randomly, and test length. In the 100% random responding conditions, both l_z and OAM_unknown were highly effective, exhibiting high power and low Type I error. In the 50% random responding conditions, l_z with empirical critical values and OAM_unknown were comparably effective, although power declined sharply for the shorter 6D-48I tests. In the 20% random responding conditions, both l_z and OAM_unknown had lower power, but l_z with empirical critical values performed better than OAM_unknown. These findings suggest that l_z may be more effective than OAM when the aberrant responding model specified in OAM does not match the true random responding pattern, particularly when random responding is relatively difficult to detect, such as in the 20% random responding and shorter test conditions.

At the nominal alpha level of .05, a similar pattern was observed, except that l_z with empirical critical values and OAM were also similarly effective when all items were modeled as being answered randomly in OAM (i.e., OAM_unknown), as shown in Table 5.

Table 5.

Study 1: Power and Type I Error Rates of l_z and OAM to Detect Random Responding_5%

Test comp	AI	l _z				OAM_unknown		OAM_known
		Empirical		Theoretical		Empirical		Empirical
		Power	Type I error	Power	Type I error	Power	Type I error	Power	Type I error
6D-48I	20%	.42	.04	.25	.01	.41	.04	.71	.03
	50%	.82	.04	.67	.01	.82	.04	.93	.04
	100%	.98	.04	.94	.01	.99	.04	.99	.04
12D-96I	20%	.61	.04	.28	.01	.61	.06	.92	.05
	50%	.97	.05	.88	.01	.98	.06	1.00	.05
	100%	1.00	.05	1.00	.00	.97	.06	.97	.06

Note. 5% = A nominal alpha level of .05; Test comp = Test composition (6D-48I = 6 dimensions 48 items, 12D-96 = 12 dimensions 96 items); AI = Proportion of items answered randomly; Empirical = Empirical critical values; Theoretical = Theoretical critical value; OAM_unknown = All items were modeled being answered randomly in the aberrant responding model; OAM_known = A known set of items were modeled being answered randomly in the aberrant responding model.

Study 1 provided initial evidence regarding the performance of l_z and OAM in detecting random responding in MFC tests. However, several aspects of the simulation design should be noted. First, because the implementation of OAM relies on HMC estimation and is therefore computationally intensive, only 20 replications were performed per condition. Although the observed power and Type I error rates showed little variation across replications, with standard deviations ranging from 0 to .05 for power and from 0 to .02 for Type I error, a larger number of replications is needed to better establish the generalizability of the findings. Second, l_z was calculated using pre-estimated statement parameters from single statement response data rather than the true statement parameters commonly used in simulation studies (e.g., Nering, 1995; Reise, 1995; Snijders, 2001). This approach reflects common practice in large-scale MUPP applications in which statement parameters are first estimated from single statement response data and then treated as fixed during scoring. This two-step approach has been supported by prior work showing high correlations between trait scores estimated using pre-estimated statement parameters and those estimated directly from MFC response data (Stark et al., 2005a, 2005b; Tu et al., 2023a). However, it introduces statement parameter estimation error that may affect the performance of l_z. Third, the proportion of aberrant respondents was fixed at 20%. Although this factor was not expected to affect the performance of l_z in Study 1 because it influences neither the scoring of individual response patterns nor the pre-estimated statement parameters, this needs to be empirically demonstrated. Study 2 was therefore designed to address these aspects by increasing the number of replications, using true statement parameters, and varying the proportion of aberrant respondents.

Study 2

Simulation Study Design

Simulation Procedures

The test specifications, normal response data generation, random response data generation, critical values, and analyses followed Study 1, with four differences. First, the simulated sample size was 1600 rather than 800. Second, true statement parameters were used directly for trait estimation and l_z calculation instead of estimating statement parameters from single statement response data. Third, the proportion of aberrant respondents varied across conditions (20%, 50%) rather than being fixed at 20%. Fourth, empirical critical values were derived from a normal response dataset of N = 1600 rather than 800.

l_z Implementation

Person parameters were estimated using maximum a posteriori (MAP) estimation, and l_z was then computed for each respondent following Equations (4.1)–(4.3). In Study 1, HMC was used because OAM requires marginal likelihoods, which were computed using the R package bridgesampling with fitted objects from rstan, whose default estimation method is HMC. In Study 2, MAP was used instead because it is substantially faster, which was necessary given the expanded design of 48 conditions and 100 replications per condition. Given the computational demands of OAM, which relies on HMC estimation, OAM was excluded from Study 2.

Study 2 Results

Tables 6 and 7 present the results at the nominal alpha levels of .01 and .05, respectively. Overall, by increasing the number of replications, using true statement parameters, and varying the proportion of aberrant respondents, Study 2 extended and strengthened the conclusions drawn from Study 1. Specifically, the power of l_z increased when a higher proportion of items were answered randomly, when test length increased, and when empirical rather than theoretical critical values were used. In addition, power and Type I error were essentially identical between the 20% and 50% aberrant respondent conditions, suggesting that the performance of l_z for random responding detection was not affected by the proportion of aberrant respondents.

Table 6.

Study 2: Power and Type I Error Rates of l_z to Detect Random Responding_1%

Test comp	AP	AI	Empirical		Theoretical
Test comp	AP	AI	Power	Type I error	Power	Type I error
6D-48I	20%	20%	.28	.01	.08	.00
		50%	.71	.01	.43	.00
		100%	.95	.01	.82	.00
	50%	20%	.28	.01	.08	.00
		50%	.71	.01	.42	.00
		100%	.95	.01	.82	.00
12D-96I	20%	20%	.41	.01	.07	.00
		50%	.94	.01	.66	.00
		100%	1.00	.01	.98	.00
	50%	20%	.42	.01	.07	.00
		50%	.94	.01	.66	.00
		100%	1.00	.01	.97	.00

Note. 1% = A nominal alpha level of .01; Test comp = Test composition (6D-48I = 6 dimensions 48 items, 12D-96 = 12 dimensions 96 items); AP = Proportion of aberrant respondents; AI = Proportion of items answered randomly; Empirical = Empirical critical values; Theoretical = Theoretical critical value.

Table 7.

Study 2: Power and Type I Error Rates of l_z to Detect Random Responding_5%

Test comp	AP	AI	Empirical		Theoretical
Test comp	AP	AI	Power	Type I error	Power	Type I error
6D-48I	20%	20%	.45	.04	.18	.01
		50%	.84	.04	.61	.01
		100%	.98	.04	.91	.01
	50%	20%	.46	.04	.18	.01
		50%	.84	.04	.60	.01
		100%	.98	.04	.91	.01
12D-96I	20%	20%	.66	.06	.18	.00
		50%	.98	.06	.81	.00
		100%	1.00	.06	.99	.00
	50%	20%	.66	.06	.18	.00
		50%	.98	.06	.81	.00
		100%	1.00	.06	.99	.00

Note. 5% = A nominal alpha level of .05; Test comp = Test composition (6D-48I = 6 dimensions 48 items, 12D-96 = 12 dimensions 96 items); AP = Proportion of aberrant respondents; AI = Proportion of items answered randomly; Empirical = Empirical critical values; Theoretical = Theoretical critical value.

Discussion

MFC test formats are widely used in high-stakes settings to reduce faking. However, MFC remains susceptible to random responding, particularly in low-stakes settings where respondents may be insufficiently motivated and in high-stakes settings where some assessments may be viewed as less consequential. To support the development and application of MFC tests, effective methods for detecting random responding in MFC are needed to ensure that inferences drawn from MFC data are valid. This research contributes to the MFC literature on aberrant responding detection by evaluating the effectiveness of the IRT-based person fit statistic l_z in detecting random responding and comparing its performance to OAM, a theoretically optimal method for detecting aberrant responding.

Two simulation studies were conducted to examine the efficacy of l_z for detecting random responding in MFC tests. Based on the results, three practical implications for the use of l_z can be drawn. First, l_z can be effective for detecting random responding, particularly when empirical rather than theoretical critical values are used, which is consistent with findings from previous research (Drasgow et al., 1985; Lee et al., 2014, 2024; Molenaar & Hoijtink, 1990; Nering, 1997). This improvement is attributed to the use of estimated rather than true person parameters in calculating l_z, which causes the empirical distribution of l_z to deviate from the standard normal distribution. Consequently, when theoretical critical values based on the standard normal distribution were applied, both power and Type I error rates were reduced. Because true person parameters are unknown in practice and must be estimated from observed response data, empirical critical values are recommended for detecting random responding in MFC tests. Second, shorter tests were associated with lower power for l_z. Researchers and practitioners using short tests should therefore interpret respondents not flagged by l_z cautiously, as a non-flagged response pattern does not necessarily indicate the absence of random responding. Third, the proportion of aberrant respondents had essentially no effect on the power or Type I error rate of l_z. The prevalence of aberrant respondents is therefore not a concern when applying l_z.

The comparison between OAM and l_z provides additional information about the practical utility of l_z for detecting random responding in MFC tests. OAM_known served as a theoretical benchmark because OAM achieves maximum detection power when the normal responding model fits the data and the aberrant responding model is correctly specified. Relative to this benchmark, l_z with empirical critical values approached “optimal” detection under favorable conditions, specifically when the proportion of randomly answered items was higher and the test was longer. When the aberrant responding model was incorrectly specified, as in OAM_unknown, l_z with empirical critical values performed as well as or better than OAM. In practice, it is nearly impossible to know the exact type of aberrant responding respondents engage in, the proportion of items answered aberrantly, or which specific items are affected, making it unlikely that OAM will reach its theoretical maximum power in operational settings. In addition, implementing OAM involves marginal likelihood estimation, which can be computationally intensive and technically demanding. Therefore, although l_z may not always reach the theoretical maximum detection power, its use with empirical critical values is a practical approach for detecting random responding in applied MFC testing because of its simplicity, efficiency, and robustness.

Building on this recommendation, when implementing l_z in MFC tests to detect random responding in practice, researchers and practitioners should first score the MFC response data for each respondent. The l_z statistic can then be calculated for each respondent using the estimated trait scores and the pre-estimated statement parameters based on Equations (4.1)–(4.3). The resulting l_z value is compared to a critical value, and respondents with l_z values smaller than the selected critical value may be flagged. Although researchers and practitioners may choose between theoretical and empirical critical values, the findings of the present study support the use of empirical critical values. To obtain empirical critical values, MFC response data can be simulated under a normal responding model for a large sample (e.g., 800 to 1600 respondents), and l_z can be calculated for each simulated respondent. The critical value is then determined by identifying the values at the 1st and 5th percentiles, depending on the acceptable false positive rate.

Several limitations should be noted. First, this research focused only on random responding. Although MFC formats reduce faking compared to rating scales, they do not fully prevent it. Therefore, effective methods for detecting faking in MFC may also be necessary. Lee et al. (2014) examined the effectiveness of l_z for detecting faking and found it to be ineffective. However, in their simulation, faking response data were simulated by simply increasing respondents’ trait levels, which may have made it difficult for l_z to distinguish between respondents who were faking and those who genuinely had high trait levels. Future research should explore alternative ways of modeling and simulating faking response data and further investigate the effectiveness of l_z and OAM in detecting faking. For example, instead of simulating only the outcome of faking, which is trait score inflation, future studies could simulate the process of faking by considering the social desirability of each statement within MFC items. Second, this research evaluated l_z and compared its performance with OAM, providing a useful starting point for evaluating random responding detection in MFC tests. However, given the comparative and multidimensional nature of MFC tests, whether other person fit statistics evaluated in the unidimensional IRT literature (Karabatsos, 2003) can be extended to the MFC context, and how their performance would compare with l_z in detecting random responding remain open questions. A systematic comparison of l_z with other person fit statistics in MFC tests is therefore an important direction for future research. Third, although the observed power of l_z in this research was comparable to findings from previous research (e.g., Kim & Moses, 2018; Lee et al., 2024), it was relatively low in some conditions. In particular, when the proportion of items answered randomly was small and the test was short, the power of l_z was well below the theoretical maximum detection power achieved by OAM. One potential reason is that l_z is sensitive to person parameter estimation error (Meijer & Tendeiro, 2012). In this research, person parameters were estimated from simulated aberrant response data when calculating l_z, which reflects operational testing practice in which person parameters must be estimated from observed response data that may contain aberrant responding. However, estimating person parameters from response data that may contain aberrant responding introduces estimation error, which can negatively affect the performance of l_z (Kim & Moses, 2018; Nering, 1997). Future research should examine whether improving the accuracy of person parameter estimation, such as by using informative priors or incorporating collateral information (de La Torre & Patz, 2005; Joo et al., 2026; Stark et al., 2005b; Tu et al., 2023a, 2025; Wang & Wu, 2016), can increase the effectiveness of l_z for detecting random responding. Finally, we note that in the 12D test under the 20% randomly answered items condition, the distribution of randomly answered items across dimensions was uneven, as indicated by the test specification in Table 2. Although this does not affect the present simulation results because the detection indices were computed for each respondent across all dimensions, it raises questions about more realistic testing settings in which some dimensions may show higher rates of aberrant responding than others. Future research should systematically examine how dimension-specific patterns of aberrant responding affect the estimation and interpretation of construct dimensions in MFC measures.

Footnotes

ORCID iDs

Naidan Tu

Sean Joo

Ethical Considerations

There are no human participants in this article and informed consent is not required.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data for the study will be available at .

References

Arthur Jr

Hagen

George

J. F.

(2021). The lazy or dishonest respondent: Detection and prevention. Annual Review of Organizational Psychology and Organizational Behavior, 8(1), 105–137. https://doi.org/10.1146/annurev-orgpsych-012420-055324

Barrick

M. R.

Mount

M. K.

(1991). The big five personality dimensions and job performance: a meta-analysis. Personnel psychology, 44(1), 1–26.

Cao

Drasgow

(2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347–1368. https://doi.org/10.1037/apl0000414

Conijn

J. M.

Emons

W. H.

Sijtsma

(2014). Statistic l_z-based person-fit methods for noncognitive multiscale measures. Applied Psychological Measurement, 38(2), 122–136. https://doi.org/10.1177/0146621613497568

Conijn

J. M.

Sijtsma

Emons

W. H.

(2016). Identifying person-fit latent classes, and explanation of categorical and continuous person misfit. Applied Psychological Measurement, 40(2), 128–141. https://doi.org/10.1177/0146621615611164

de La Torre

Patz

R. J.

(2005). Making the most of what we have: A practical application of multidimensional item response theory in test scoring. Journal of Educational and Behavioral Statistics, 30(3), 295–311. https://doi.org/10.3102/10769986030003295

Drasgow

Levine

M. V.

(1986). Optimal detection of certain forms of inappropriate test scores. Applied Psychological Measurement, 10(1), 59–67.

Drasgow

Levine

M. V.

McLaughlin

M. E.

(1987). Detecting inappropriate test scores with optimal and practical appropriateness indices. Applied Psychological Measurement, 11(1), 59–79. https://doi.org/10.1177/014662168701100105

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x

10.

Felt

J. M.

Castaneda

Tiemensma

Depaoli

(2017). Using person fit statistics to detect outliers in survey research. Frontiers in Psychology, 8, 863. https://doi.org/10.3389/fpsyg.2017.00863

11.

Gronau

Q. F.

Singmann

Wagenmakers

E. J.

(2017). Bridgesampling: An R package for estimating normalizing constants. arXiv preprint arXiv:1710.08162.

12.

Joo

Lee

Stark

(2026). Improving latent trait estimation in multidimensional forced choice measures: Latent regression multi-unidimensional pairwise preference model. Applied Psychological Measurement, 50(4-5), 01466216251415189. https://doi.org/10.1177/01466216251415189

13.

Joo

S. H.

Lee

Stark

(2017). Evaluating anchor-item designs for concurrent calibration with the GGUM. Applied Psychological Measurement, 41(2), 83–96. https://doi.org/10.1177/0146621616673997

14.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277–298. https://doi.org/10.1207/s15324818ame1604_2

15.

Kim

Moses

(2018). The impact of aberrant responses and detection in forced‐choice noncognitive assessment. ETS Research Report Series, 2018(1), 1–15. https://doi.org/10.1002/ets2.12222

16.

Lee

Joo

Son

(2024). Detecting careless respondents in multidimensional forced choice data: An application of lz person-fit statistic to the TIRT model. Journal of Business and Psychology, 39(3), 541–564. https://doi.org/10.1007/s10869-023-09911-w

17.

Lee

Stark

Chernyshenko

O. S.

(2014). Detecting aberrant responding on unidimensional pairwise preference tests: An application of l_z based on the Zinnes–Griggs ideal point IRT model. Applied Psychological Measurement, 38(5), 391–403.

18.

Levine

M. V.

Drasgow

(1984). Performance envelopes and optimal appropriateness measurement. No. MEASUREMENTSER845.

19.

Levine

M. V.

Drasgow

(1988). Optimal appropriateness measurement. Psychometrika, 53(2), 161–176. https://doi.org/10.1007/bf02294130

20.

Levine

M. V.

Rubin

D. B.

(1979). Measuring the appropriateness of multiple-choice test scores. Journal of Educational Statistics, 4(4), 269–290. https://doi.org/10.2307/1164595

21.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085

22.

Meijer

R. R.

Tendeiro

J. N.

(2012). The use of the l_z and l_z* person-fit statistics and problems derived from model misspecification. Journal of Educational and Behavioral Statistics, 37(6), 758–766.

23.

Molenaar

I. W.

Hoijtink

(1990). The many null distributions of person fit indices. Psychometrika, 55(1), 75–106. https://doi.org/10.1007/bf02294745

24.

Nering

M. L.

(1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19(2), 121–129. https://doi.org/10.1177/014662169501900201

25.

Nering

M. L.

(1997). The distribution of indexes of person fit within the computerized adaptive testing environment. Applied Psychological Measurement, 21(2), 115–127. https://doi.org/10.1177/01466216970212002

26.

Neyman

Pearson

E. S.

(1933). IX. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 231(694-706), 289–337. https://doi.org/10.1098/rsta.1933.0009

27.

Ployhart

R. E.

Holtz

B. C.

(2008). The diversity-validity dilemma: Strategies for reducing racioethnic and sex subgroup differences and adverse impact in selection. Personnel Psychology, 61(1), 153–172.

28.

Reise

S. P.

(1995). Scoring method and the detection of person misfit in a personality assessment context. Applied Psychological Measurement, 19(3), 213–229. https://doi.org/10.1177/014662169501900301

29.

Roberts

J. S.

Donoghue

J. R.

Laughlin

J. E.

(2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001

30.

Snijders

T. A.

(2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66(3), 331–342. https://doi.org/10.1007/bf02294437

31.

Stan Development Team . (2023). RStan: The R interface to Stan. R Package Version 2.21.8. https://mc-stan.org/

32.

Stark

Chernyshenko

O. S.

Drasgow

(2005a). An IRT approach to constructing and scoring pairwise preference items involving stimuli on different dimensions: The multi-unidimensional pairwise-preference model. Applied Psychological Measurement, 29(3), 184–203. https://doi.org/10.1177/0146621604273988

33.

Stark

Chernyshenko

O. S.

Drasgow

(2005b). Identifying and understanding the effects of unmotivated examinees on test dimensionality using optimal appropriateness measurement. International Journal of Testing, 5(3), 247–263. https://doi.org/10.1207/s15327574ijt0503_4

34.

Stark

Chernyshenko

O. S.

Drasgow

(2016, July). A multidimensional forced choice approach to high stakes personality testing [Presentation]. 81st Annual International Meeting of the Psychometric Society, Asheville, NC, United States.

35.

Stark

Chernyshenko

O. S.

Drasgow

Lee

W. C.

White

L. A.

Young

M. C.

(2011). Optimizing prediction of attrition with the US Army's assessment of Individual Motivation (AIM). Military Psychology, 23(2), 180–201. https://doi.org/10.1080/08995605.2011.550234

36.

Joo

Lee

Stark

(2023a). Comparison of parameter estimation approaches for multi-unidimensional pairwise preference tests. Behavior Research Methods, 55(6), 2764–2786. https://doi.org/10.3758/s13428-022-01927-z

37.

Joo

Stark

(2025). Examining the tradeoffs of exposure control and collateral information with multidimensional forced-choice computerized adaptive testing. Behavior Research Methods, 57(7), 207. https://doi.org/10.3758/s13428-025-02712-4

38.

Zhang

Angrave

Sun

Neuman

(2023b). Estimating the multidimensional generalized graded unfolding model with covariates using a Bayesian approach. Journal of Intelligence, 11(8), 163. https://doi.org/10.3390/jintelligence11080163

39.

Wang

W. C.

S. L.

(2016). Confirmatory multidimensional IRT unfolding models for graded-response items. Applied Psychological Measurement, 40(1), 56–72. https://doi.org/10.1177/0146621615602855

40.

Wetzel

Frick

Brown

(2020). Does multidimensional forced-choice prevent faking? Comparing the susceptibility of the multidimensional forced-choice format and the rating scale format to faking. Psychological Assessment, 33(2), 156–170. https://doi.org/10.1037/pas0000971

41.

White

L. A.

Young

M. C.

(1998, August). Development and validation of the assessment of individual motivation (AIM). Paper presented at the Annual Meeting of the American Psychological Association, San Francisco, CA.

Random Responding Detection in Two Alternative Forced Choice Tests: l z vs. Optimal Appropriateness Measurement

Abstract

Keywords

Introduction

The MUPP Model

The GGUM

OAM

The l z Statistic

Study 1

Simulation Study Design

Test Specification

Normal Response Data Generation

Statement Parameter Estimation

Random Response Data Generation

OAM and l z Implementation

Critical Values

Analyses

Study 1 Results

lz vs. OAM

Study 2

Simulation Study Design

Simulation Procedures

lz Implementation

Study 2 Results

Discussion

Footnotes

ORCID iDs

Ethical Considerations

Funding

Declaration of conflicting interests

Data Availability Statement

References

The l_z Statistic

OAM and l_z Implementation

l_z vs. OAM

l_z Implementation