Compound Optimal Design for Online Item Calibration Under the Two-Parameter Logistic Model

Abstract

Under the theory of sequential design, compound optimal design with two optimality criteria can be used to solve the problem of efficient calibration of item parameters of item response theory model. In order to efficiently calibrate item parameters in computerized testing, a compound optimal design is proposed for the simultaneous estimation of item difficulty and discrimination parameters under the two-parameter logistic model, which adaptively focuses on optimizing the parameter which is difficult to estimate. The compound optimal design using the acceptance probability can provide ability design points to optimize the item difficulty and discrimination parameters, respectively. Simulation and real data analysis studies showed that the compound optimal design outperformed than the D-optimal and random design in terms of the recovery of both discrimination and difficulty parameters.

Keywords

computerized adaptive testing two-parameter logistic model D-optimal design compound optimal design online calibration

Introduction

Computerized adaptive testing (CAT) is a tailored test that adapts to ability level of each examinee (Chang & Ying, 2007). Compared with a pen-and-paper or linear test, CAT has the advantages of high accuracy, short test, and real-time report. Quellmalz and Pellegrino (2009) report in Science that many large-scale assessment programs and formative assessments have implemented online testing or CAT (Chang, 2004; Chang et al., 2016; Chang & Ying, 2007; Quellmalz & Pellegrino, 2009; van der Linden & Glas, 2000; Yan et al., 2014). In order to take full advantages of CAT, it is indispensable for CAT to have a large-scale item bank with already calibrated parameters of item response theory models (Berger et al., 2019; Huebner, 2010; Reckase, 2010; Stocking, 1994; Wainer et al., 2000; Weiss & Kingsbury, 1984; You et al., 2010). In the process of continuously building, maintaining, and expanding an item bank of CAT, online calibration of item parameters of new test items is required (Chang & Lu, 2010; Stocking, 1994). Online calibration of item parameters is accomplished by “seeding” new items into the adaptive test sessions with previously calibrated items (Wainer et al., 2000; Wainer & Mislevy, 1990). Because it is an economical and effective way to collect item response data and expand item bank, online calibration (Chang & Lu, 2010; Chen, 2016; Chen et al., 2017; Chen & Wang, 2016) has long been concerned by researchers in CAT.

Online calibration method and online calibration design are two important aspects of online calibration (Chen & Xin, 2014). The online calibration method aims to estimate the item parameters of the new items, while online calibration design is intended to design the way the new items are assigned to test-takers. As early as the 1990s, several online calibration methods have been proposed under the unidimensional item response theory (UIRT) model, such as Method A and Method B (Stocking, 1988), and the marginal maximum likelihood estimate with one EM cycle (OEM; Wainer & Mislevy, 1990). Method A fixes the estimated abilities obtained from the responses to the operational items during the field test and applies the conditional maximum likelihood estimation to estimate item parameters of new items based on the collected item responses and the fixed abilities. Method B fixes the estimated abilities to estimate item parameters of new items and anchor items and uses the scale transformation obtained from the initial estimates and re-estimates for the anchor items to place the item parameters of new items on the scale of item pool. As a variation of the OEM method, the marginal maximum likelihood estimate with multiple EM cycles (MEM) has been proposed in later studies (Ban et al., 2001, 2002). In recent years, online calibration is an ongoing topic in the field of CAT. You et al. (2010) proposed the conditional maximum likelihood estimation method, which is an alternating iteration algorithm for estimating discrimination and difficulty parameters. The OEM and MEM was extended from dichotomous item response theory models to the generalized partial credit model (Zheng, 2016). For considering the measurement errors in ability estimation, Chen (2016) applied a full function maximum likelihood estimation (FFMLE) and an estimator which exploited the consequences of sufficiency (ECSE) to correct the estimation error of ability before using Method A for online calibration. He et al. (2017) introduced the modified Lord’s bias-correction method to correct the deviation of maximum likelihood ability estimates in Method A, namely, MLE-LBCI-Method A. Moreover, Method A and its variations have been generalized to multidimensional item response theory (MIRT; Chen, 2017; Chen & Wang, 2016; Chen et al., 2017; Hassan & Miller, 2024). Method A will be used in our study, because it is the relatively simplest and most straightforward method among all available calibration methods in UIRT (Chen & Wang, 2016; He et al., 2017).

Beside online calibration methods, two kinds of online calibration designs are developed in the UIRT. The first kind requires a static examinee pool, where examinees with suitable ability values will be selected to calibrate new items (He et al., 2020). This kind is suitable for a parallel testing situation (Ul Hassan & Miller, 2019). The design points or examinees’ abilities in a static examinee pool can be obtained from the D-optimal design, which is based on a determinant criterion of item parameter information matrix (Berger, 1992; Chang & Lu, 2010; Jones & Jin, 1994; Ul Hassan & Miller, 2019). The second kind of examinee pool is dynamic, because examinees are allowed to leave immediately after their operational test sessions (He & Chen, 2020) and every time only an examinee reaches a seeding location (Zheng, 2016). van der Linden and Ren (2015) proposed an reformed D-optimal design (denoted as D-VR design) for online calibration based on Bayesian optimality criteria. The alternative two-point D-optimal design (denoted as D-Tp) was introduced for selecting the new item with the greatest increase in the determinant among all new items to the current examinee (Ren et al., 2017). He and Chen (2020) proposed a new online design (denoted as D-c design) to select the new item for which the current examinee would yield the greatest decrease in the determinant of the covariance matrix of item parameter estimates. Moreover, a lot of online designs were developed based on the excellence degree criterion (He et al., 2020), the maximum $θ$ Fisher information (MTFI) and the match-b selection (MATB; Zheng, 2016), and the ability-related background variables (Berger et al., 2019).

It is not difficult to find out from the above studies that random design and D-optimal design have been adapted into static and dynamic scenarios. Random design has the advantage of being easy to implement, but its accuracy is not as good as optimal design. The basic goal of optimal design is to select as few control variables as possible to estimate unknown parameters of the model (Kitsos, 2013; Silvey, 1980). For example, the maximum Fisher information (MFI) method in CAT (Lord, 1971) is a fully sequential D-optimal design method for selecting item parameter design points in order to accurately estimate the ability of examinees. Sequential design allows for iteratively selecting informative experiments based on current knowledge about the parameters (e.g., Cao et al., 2024). Thus, CAT is more efficient (e.g., shorter test length) and accurate (e.g., higher precision for all examinees) than a linear test. Similar to the MFI method used in CAT, instead of items, the optimal calibration design is to select the most suitable ability design points for a new item according to a certain criterion, so as to solve the problem of item parameter calibration with a limited number of examinees.

Now in the framework of online calibration, the D-optimal design still has some problems. From the relevant study, it can be seen that the D-optimal design has the problem of large error in the estimation of discrimination parameters of high discrimination items and difficulty parameters of low discrimination items (Chang & Lu, 2010). The high estimation error of discrimination parameters in the small sample calibration conditions leads to the overestimation of the precision of the ability estimates in CAT, which is called as the error capitalization on chance (COC) problem (Olea et al., 2012; Patton et al., 2013). Although cross-validation techniques are regarding as possible solutions (van der Linden & Glas, 2000), these methods only suppress symptoms without addressing underlying causes. There is still a need to develop parameter estimation procedures for reducing the estimation errors of the item parameters (Olea et al., 2012).

To fundamentally tackle the problem from its root, it is necessary to propose optimal designs and calibration methods to improve the accuracy of item parameter estimates. The main purpose of the study is to explore an optimal design for online calibration, optimizing the parameter which is difficult to estimate. The new design is first introduced under the static examinee pool. Then, the new design is extended to the dynamic examinee pool from the idea of the D-Tp, because it has been generalized from the static examinee pool to the dynamic examinee pool and performs very well (He & Chen, 2020; Ren et al., 2017).

The rest of this article is organized as follows. First, the UIRT model and online calibration method used in this article are described, followed by detailed introduction of design for a static examinee pool and a dynamic examinee pool, respectively. Then, simulation studies and real data analysis are conducted for illustrating the performance of the new optimal design. Finally, we conclude with a discussion. The detailed results for the static examinee pool are given in the appendix.

Method

Two-Parameter Logistic Model

The item response function of the two-parameter logistic model (2PLM) can be written as

P_{i j} = P (U_{i j} = 1 | θ_{i}, a_{j}, b_{j}) = {1 + \exp (- D a_{j} (θ_{i} - b_{j}))}^{- 1},

(1)

where

U_{i j}

is the response for examinee

i

to item

j

P_{i j}

is the probability of a correct response for examinee

i

on item

j

a_{j}

and

b_{j}

are discrimination and difficulty parameters, and

θ_{i}

is the ability parameter. The constant

D

is taken as 1.

Online Calibration Method

As described above, online calibration methods mainly include Method A, OEM, and MEM. Since this study focuses on online optimal design, Method A as a simple and effective online calibration method was used and introduced below.

Let $U_{j} = (u_{1 j}, u_{1 j}, . . ., u_{N_{j}})$ be a vector of item responses on new item $j$ , where $N_{j}$ is the calibration sample size for item $j$ . Let $\hat{θ} = ({\hat{θ}}_{1}, {\hat{θ}}_{2}, . . ., {\hat{θ}}_{N_{j}})$ denote a vector of ability estimates of examinees. The ability of each examinee usually can be estimated based on calibrated item parameters of an operational item bank and his or her item responses in the field test. From the local independence, the likelihood function and its natural logarithm are, respectively, given by

L (a_{j}, b_{j} | U_{j}, \hat{θ}) = \prod_{i = 1}^{N_{j}} P_{i j}^{u_{i j}} {(1 - P_{i j})}^{1 - u_{i j}},

(2)

and

l (a_{j}, b_{j} | U_{j}, \hat{θ}) = \sum_{i = 1}^{N_{j}} (u_{i j} \ln (P_{i j}) + (1 - u_{i j}) \ln (1 - P_{i j})) .

(3)

Since maximizing the logarithm of likelihood function with respect to item parameter vector $γ_{j} = {(a_{j}, b_{j})}^{T}$ is a non-linear problem, it needs to use an iterative algorithm to estimate $γ_{j}$ . Given a second-order derivative (Hessian matrix) of the log-likelihood function

H (γ_{j}) = [\frac{\partial^{2} l (γ_{j} | U_{j}, \hat{θ})}{\partial γ_{j} \partial {γ_{j}}^{T}}] .

(4)

The iteration formula of the Newton-Raphson algorithm is

{\hat{γ}}_{j}^{(t + 1)} = {\hat{γ}}_{j}^{(t)} - {[H ({\hat{γ}}_{j}^{(t)})]}^{- 1} \frac{\partial l ({\hat{γ}}_{j}^{(t)} | U_{j}, \hat{θ})}{\partial {\hat{γ}}_{j}} .

(5)

Using the information matrix $I ({\hat{γ}}_{j}) = - E [H ({\hat{γ}}_{j})]$ instead of the Hessian matrix, we can get an iterative formula for the Fisher scoring approach

{\hat{γ}}_{j}^{(t + 1)} = {\hat{γ}}_{j}^{(t)} + {[I^{(N)} ({\hat{γ}}_{j}^{(t)})]}^{- 1} \frac{\partial l ({\hat{γ}}_{j}^{(t)} | U_{j}, \hat{θ})}{\partial {\hat{γ}}_{j}} .

(6)

For 2PLM, the first-order derivative and information matrix in equation (6) can be written as respectively

\frac{\partial l ({\hat{γ}}_{j}^{(t)} | U_{j}, \hat{θ})}{\partial {\hat{γ}}_{j}} = \sum_{i = 1}^{N_{j}} (u_{i j} - P_{i j}) [\begin{array}{c} D ({\hat{θ}}_{i} - b_{j}) \\ {- D a}_{j} \end{array}],

(7)

and

I^{(N_{j})} ({\hat{γ}}_{j}) = \sum_{i = 1}^{N_{j}} I ({\hat{γ}}_{j}; {\hat{θ}}_{i}) = \sum_{i = 1}^{N_{j}} P_{i j} (1 - P_{i j}) [\begin{array}{c} D^{2} {({\hat{θ}}_{i} - b_{j})}^{2} & - D^{2} a_{j} ({\hat{θ}}_{i} - b_{j}) \\ - D^{2} a_{j} ({\hat{θ}}_{i} - b_{j}) & D^{2} a_{j}^{2} \end{array}] .

(8)

Design for a Static Examinee Pool

Random Design

Random design, that is, completely randomized design, refers to random assignment of examinees to new items in the field test for online calibration. In other words, examinees are randomly assigned or selected to answer new items, and the position of new items are also randomly seeded in a field test. The random design is very simple to implement, but it does not lead to either an optimal match between item’s difficulty and examinee’s ability or the adaptive of CAT.

D-Optimal Design

As can be seen from Section “Online Calibration Method,” the online calibration of item parameters is a non-linear problem. For the non-linear model or problem, because optimal design relies on unknown parameters, sequential D-optimal design (Chang & Lu, 2010) and optimal Bayesian adaptive design (van der Linden & Ren, 2015) are usually used. That is, the calibration of the unknown parameters is performed while collecting the sample data, and then the optimal design point is selected according to the current estimation of the unknown parameters, and the process is repeated until a certain termination condition reached.

The D-optimal design is one of the commonly used designs under the fully sequential design. The performance of the D-optimal design for CAT online item calibration has been investigated on test items from the synthesized data and National Assessment of Educational Progress (Chang & Lu, 2010). Parameter calibration is one of the basic tasks of statistics or measurements. To accurately calibrate item parameters of a new item with a limited number of examinees, it is necessary to select examinee ability or design point according to certain criteria, so that the design point can minimize estimation error of item parameters for the new item. For the current sample size of $n_{j}$ , the estimation error of the maximum likelihood estimate of item parameter vector ${\hat{γ}}_{j} = {({\hat{a}}_{j}, {\hat{b}}_{j})}^{T}$ is usually be measured by the information matrix $I^{(n_{j})} ({\hat{γ}}_{j})$ of item parameters. In order to minimize the confidence ellipsoid area of item parameters, the determinant of the information matrix should be maximized. That is, the D-optimal design is designed to sequentially select the next design point satisfied the following condition

θ^{(n_{j} + 1)} = \underset{θ \in R}{arg min} - [\det (I^{(n_{j})} ({\hat{γ}}_{j}) + I ({\hat{γ}}_{j}; θ)) - \det (I^{(n_{j})} ({\hat{γ}}_{j}))],

(9)

where

\det

denotes the determinant of a matrix. The right site of equation (9) is a function of

θ

. The D-optimal design is to find design point

θ = θ^{(n_{j} + 1)}

with the maximum value of the gain in the determinant due to the assignment of the next examinee for item

j

D-Tp Design

Suppose that the current estimates of item parameters are known, 2PLM can be regarded as a logistic regression model. For the two-point design, the previous studies suggested that the D-optimal design points should satisfy $P (U_{i j} = 1 | θ^{(n_{j} + 1)}, {\hat{a}}_{j}, {\hat{b}}_{j}) = 0.176$ and $P (U_{i j} = 1 | θ^{(n_{j} + 2)}, {\hat{a}}_{j}, {\hat{b}}_{j}) = 0.824$ , and thus the solution for the two-point D-optimal design is (see e.g., Chang & Lu, 2010):

(θ^{(n_{j} + 1)}, θ^{(n_{j} + 2)}) = (- 1.5434 / (D {\hat{a}}_{j}) + {\hat{b}}_{j}, 1.5434 / (D {\hat{a}}_{j}) + {\hat{b}}_{j}) .

(10)

Equation (9) is slightly different from equation (10), and the former is a one-point design and the latter is a two-point design.

Compound Optimal Design

The design is based on the diagonal elements of Fisher information matrix to find the design points based on the following two optimality criteria

θ_{a_{j}}^{(n_{j} + 1)} = \underset{θ \in R}{arg max} I_{11} ({\hat{γ}}_{j}; θ) = \underset{θ \in R}{arg max} D^{2} {(θ - {\hat{b}}_{j})}^{2} P_{i j} (1 - P_{i j}),

(11)

and

θ_{b_{j}}^{(n + 1)} = \underset{θ \in R}{arg max} I_{22} ({\hat{γ}}_{j}; θ) = \underset{θ \in R}{arg max} D^{2} \hat{a}_{j}^{2} P_{i j} (1 - P_{i j}) .

(12)

One solution for a specific variable $θ$ of equation (11) can be obtained by solving the non-linear equation, which can be recorded as one of the solutions being $θ_{a_{j}}^{(n_{j} + 1)}$ , and the other solution being $θ_{a_{j}}^{(n_{j} + 2)} = - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}$ from the symmetry of $I_{11} ({\hat{γ}}_{j}; θ)$ . A specific variable $θ$ of equation (12) can be directly solved and two points of the optimal design for the difficulty parameter are both $θ_{b_{j}}^{(n_{j} + 1)} = θ_{b_{j}}^{(n_{j} + 2)} = {\hat{b}}_{j}$ . If a two-point design, says $θ_{a_{j}}^{(n_{j} + 1)}$ and $- θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}$ , is used, it would be reasonable to conclude that the information matrix $I ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) + I ({\hat{γ}}_{j}; - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j})$ is a diagonal matrix. This is confirmed by the following derivation. If two-points $θ_{a_{j}}^{(n_{j} + 1)}$ and $- θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}$ are substituted for $θ_{a}$ in the information matrix, two resulting information matrices at these two points are respectively

\begin{array}{l} I ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) = {1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1} (1 - {1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1}) \\ [\begin{array}{c} D^{2} {(θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j})}^{2} & - D^{2} {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}) \\ - D^{2} {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}) & D^{2} \hat{a}_{j}^{2} \end{array}], \end{array}

(13)

and

\begin{array}{l} I ({\hat{γ}}_{j}; - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}) = {1 + \exp (- D {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}))}^{- 1} (1 - {1 + \exp (- D {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j})}^{- 1}) \\ [\begin{array}{c} D^{2} {(- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j})}^{2} & - D^{2} {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}) \\ - D^{2} {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}) & D^{2} \hat{a}_{j}^{2} \end{array}] . \end{array}

(14)

Since the following three equations are established

{1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1} = 1 - {1 + \exp (- D {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}))}^{- 1},

(15)

1 - {1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1} = {1 + \exp (- D {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}))}^{- 1},

(16)

- D^{2} {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}) = D^{2} {\hat{a}}_{j} (- θ_{a_{j}}^{(n_{j} + 1)} + {\hat{b}}_{j}) .

(17)

I ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) + I ({\hat{γ}}_{j}; - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j})

is a diagonal matrix as follows

\begin{array}{c} I ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) + I ({\hat{γ}}_{j}; - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}) \\ = {1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1} (1 - {1 + \exp (- D {\hat{a}}_{j} (θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j}))}^{- 1}) \\ [\begin{array}{l} 2 D^{2} {(θ_{a_{j}}^{(n_{j} + 1)} - {\hat{b}}_{j})}^{2} & 0 \\ 0 & 2 D^{2} \hat{a}_{j}^{2} \end{array}] . \end{array}

(18)

It should be noted that although the sum of these two matrices is a diagonal matrix, $I^{(n)} ({\hat{γ}}_{j})$ is not diagonal, because the estimated difficulty parameter ${\hat{b}}_{j}$ will change and the information matrix based on designs derived for different ${\hat{b}}_{j}$ will therefore not be diagonal.

There are two optimality criteria in equations (11) and (12). Obviously, the corresponding designs do not dominate each other, in that one design will yield a higher efficiency with respect to one criterion but not the other. An approach to simultaneously maximize two criteria was proposed in Eccleston and Whitaker (1999). The design point is accepted with a probability. It is similar to the acceptance probability which refers to the likelihood of a candidate solution being accepted based on its quality and often used in simulated annealing algorithms or Metropolis–Hastings algorithms (Givens & Hoeting, 2013). This idea is implemented in our study to develop a new way of combining these two opposing criteria, which can be regarded as a compound design criterion (McGree et al., 2008). Thus, the two-points are accepted with the following probability

(θ_{j}^{(n_{j} + 1)}, θ_{j}^{(n_{j} + 2)}) = {\begin{cases} (θ_{a_{j}}^{(n_{j} + 1)}, - θ_{a_{j}}^{(n_{j} + 1)} + 2 {\hat{b}}_{j}), & r < p_{j}, \\ ({\hat{b}}_{j}, {\hat{b}}_{j}), & r \geq p_{j}, \end{cases}

(19)

where

\begin{array}{l} p_{j} = (1 / I_{11} ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)})) / ((1 / I_{11} ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) + (1 / I_{22} ({\hat{γ}}_{j}; θ_{b_{j}}^{(n_{j} + 1)}))) \\ = I_{22} ({\hat{γ}}_{j}; θ_{b_{j}}^{(n_{j} + 1)}) / (I_{11} ({\hat{γ}}_{j}; θ_{a_{j}}^{(n_{j} + 1)}) + I_{22} ({\hat{γ}}_{j}; θ_{b j}^{(n_{j} + 1)})), \end{array}

(20)

and

r

is a uniformly distributed random number over the interval (0, 1). Here, the

p_{j}

is the ratio of the maximum Fisher information of difficulty to the sum of the maximum Fisher information of discrimination and difficulty for item

j

. For example, if

p_{j}

is larger than 0.5, it means that the optimal design point can provide more information on difficulty than on discrimination. It implies that the item discrimination is more difficult to estimate than the item difficulty, and the estimation of item discrimination requires a larger sample relative to the estimation of item difficulty. Thus, equation (19) takes a higher probability to select optimal design points for item discrimination. Therefore, the design using the acceptance probability adaptively focuses on optimizing the parameter which is difficult to estimate in order to implement the simultaneous estimation of item difficulty and discrimination parameters.

Design for a Dynamic Examinee Pool

Let $j = 1, 2, . . ., J$ denote the new items, and let $n_{j}$ be the current sample size for item $j$ . Suppose the total number of new items answered by each examinee is less than $f$ . Let $θ_{i}$ be the ability of the current active examinee $i$ . Let $S_{i}$ denote the set of the new items that need to be calibrated for $n_{j} < N_{j}$ , and the current active examinee $i$ has not answered. Next, the random and optimal designs are introduced to select a new item from the set of $S_{i}$ assigned to the current examinee $i$ .

Random Design

A new item is randomly selected from the set of $S_{i}$ and assigned to the current examinee $i$ .

D-optimal Design

For the current active examinee $i$ , the D-optimal design is designed to select the new item with the following condition (He & Chen, 2020; Ren et al., 2017)

j = \underset{j \in S_{i}}{arg min} - [\det (I^{(n_{j})} ({\hat{γ}}_{j}) + I ({\hat{γ}}_{j}; {\hat{θ}}_{i})) - \det (I^{(n_{j})} ({\hat{γ}}_{j}))] .

(21)

D-Tp Design

The two-point D-optimal design selects the new item that satisfies (He & Chen, 2020; Ren et al., 2017)

j = \underset{j \in S_{i}}{arg min} ({| - 1.5434 / (D {\hat{a}}_{j}) + {\hat{b}}_{j} - {\hat{θ}}_{i}_{i} |}^{δ_{j}} \times {| 1.5434 / (D {\hat{a}}_{j}) + {\hat{b}}_{j} - {\hat{θ}}_{i} |}^{1 - δ_{j}}) .

(22)

where the indicator function

δ_{j}

takes value 1 if

n_{j}

is odd for item

i

and value 0 otherwise.

Compound Optimal Design

First, the two design points are calculated by equation (19) for each item $j$ of $S_{i}$ . Then, the two-point design selects the new item that satisfies:

j = \underset{j \in S_{i}}{arg min} ({| θ_{j}^{(n_{j} + 1)} - {\hat{θ}}_{i}_{i} |}^{δ_{j}} \times {| θ_{j}^{(n_{j} + 2)} - {\hat{θ}}_{i} |}^{1 - δ_{j}}),

(23)

where

δ_{j}

is the indicator function as in equation (22).

Simulation Studies

Two simulation studies were conducted to investigate the performance of four designs of random design (R), the single point D-optimal design (D), the two-point D-optimal design (D-Tp), and the compound optimal design (C) under the static examinee pool and dynamic examinee pool, respectively.

Design

According to the design of related studies (Chang & Lu, 2010; He & Chen, 2020; Ren et al., 2017), three factors were considered here, including item parameters, sample size, and ability estimate error. The simulation design mainly comes from the study of Chang and Lu (2010). They have applied the similar design to investigate the performance of D-optimal design for online calibration via variable length CAT. For the static examinee pool, the values of the discrimination parameter were a = 0.5, 1, 1.5, 2, and 2.5, and three values of the difficulty parameter were only considered, b = 0, 1, and 2, where b = −2 and −1 need not to consider since these cases are symmetric to 1 and 2, respectively. The full factorial design formed a total of 15 item parameter combinations. For the dynamic examinee pool, the difficulty of −1 and −2 (i.e., b = −1 and −2) were considered in the study, and formed a total of 25 item parameter combinations. The sample size was divided into three levels, $N = 120$ , 180, and 300.

We did not conduct simulations within a CAT framework or a computerized test, because many factors (such as item bank size, item parameter distribution, item selection, ability estimation, seeding location, test length, or stopping rule) influence the standard error of ability estimates, which is difficult to be manipulated. Thus, we directly manipulated four levels of the standard error of ability estimates based on the approach of Chang and Lu (2010) to create four scenarios that cover a range of outcomes, from ideal to realistic. The standard deviation of the ability estimation error contains three fixed levels, σ = 0, 0.2, and 0.5, where σ = 0 denoted the true or simulated ability without adding error to estimate item parameters, σ = 0.2 denoted a predetermined standard error criterion for variable length CAT (Choi et al., 2011), whose corresponding information of the ability estimation was 25, and σ = .5 denoted the condition of the fixed large error of ability. When information function of an item pool with uniform difficulty parameters is relatively flat (Babcock & Weiss, 2012), or the optimal item pool with sufficient numbers of good quality items that are most informative at a series of ability levels (He & Reckase, 2014), the variable length CAT can achieve a predetermined level of measurement precision and yield near equivalent measurement precision across the entire range of ability (Choi et al., 2011). If the operational item pool has difficulty parameters centered around a specific range of the ability scale (e.g., −1 < θ < 1) and has fewer items appropriate for administration for extreme ability levels, the operational CAT can yield better measurement precision for examinees at the middle of the ability scale than at the extremes of the ability scale. The study of He and Reckase (2014) shows that conditional error of ability estimates for the retied operational item pool has the shape of a parabolic bowl with its bottom at origin (He & Reckase, 2014). Because conditional standard errors of ability estimates for CAT are almost in the range of 0.2 and 0.65 on an interval $[- 3, 3]$ (Babcock & Weiss, 2012; He & Reckase, 2014; Magis et al., 2017), the shape of the parabolic bowl is given as $σ (θ) = 0.05 θ^{2} + 0.2$ . It implies that the standard error has a minimum value of 0.2 when $θ = 0$ , and the standard error is 0.65 when $θ = \pm 3$ . This condition as the variation in conditional error across different levels of ability was also considered in the simulation design to enhance its realism and applicability to real-world scenarios. For each condition, the number of replications is 100 for the static and dynamic examinee pool, respectively.

Procedures

The Static Examinee Pool

The specific steps for estimating item parameters of each new item in four designs were as follows:

Step 1 Choose one combination of item parameters in turn to produce results for each combination.

Step 2 Obtain the initial estimation of item parameters. First 50 ability values, denoted as $θ = (θ_{1}, θ_{2}, . . ., θ_{50})$ , were randomly generated from the standard normal distribution N (0,1). The initial sample size is the same as in the study of Chang and Lu (2010, p. 149). Because the distribution U [−3.6, 3.6] (Chang & Lu (2010, p. 147) is not typical in real-world assessment, it was replaced with N (0, 1). Item responses were generated using the inverse transform method. To accomplish this, item response probability $P_{i j}$ was first obtained by substituting the true value of ability $θ_{i}$ and item parameters $γ_{j}$ into equation (1) and a random uniform value $r_{i j}$ was generated from uniform distribution $U (0, 1)$ , and then comparing $r_{i j}$ with $P_{i j}$ , set $U_{i j} = 1$ if $r_{i j} \leq P_{i j}$ , and $U_{i j} = 0$ otherwise. Let ${\hat{θ}}_{i}$ denote an estimate of ability, which is the sum of the true ability and the error of ability estimate, ${\hat{θ}}_{i} = θ_{i} + ε$ , $ε \sim N (0, σ^{2})$ . The simulated item responses and ability estimates are denoted by $U = (u_{1}, u_{2}, . . ., u_{50})$ and $\hat{θ} = ({\hat{θ}}_{1}, {\hat{θ}}_{2}, . . ., {\hat{θ}}_{50})$ , respectively. Based on $\hat{θ}$ and $U$ , item parameters were estimated by the Fisher scoring approach, setting ${\hat{γ}}_{j} = (1, 0)$ as the initial values of item parameters in equation (6).

Step 3 Select the ability design points based on the random design or the optimal designs, respectively. For the random design, each element of estimated ability vector ${\hat{θ}}^{(1)}$ was randomly generated from the standard normal distribution of $N (0, 1)$ . While for optimal designs, the ability design point or estimated ability of ${\hat{θ}}^{(1)}$ was produced by the D, D-Tp, and C design, respectively, which utilized the current estimates of item parameters. The true ability was then generated by $θ_{i}^{(1)} = {\hat{θ}}_{i}^{(1)} + ε$ ,where the element of $ε$ was randomly drawn from $N (0, σ^{2})$ . Item responses were simulated based on true item parameters and the true ability $θ_{i}^{(1)}$ in exactly the same way as we described above in Step 2. The current item responses, ability vector, and the corresponding estimates were denoted by $U = (\begin{array}{l} U & U^{(1)} \end{array})$ , $θ = (\begin{array}{l} θ & θ^{(1)} \end{array})$ , and $\hat{θ} = (\begin{array}{l} \hat{θ} & {\hat{θ}}^{(1)} \end{array})$ , respectively.

Step 4 Update the estimates of item parameters. The $\hat{θ}$ and $U$ were used to estimate item parameters using equation (6). If the number of item responses on the new item is less than the specified sample size $N$ , repeat steps 3 and 4. Otherwise, set the final estimates of ${\hat{a}}_{j}$ and ${\hat{b}}_{j}$ , and stop the estimation of item parameters for item $j$ .

The Dynamic Examinee Pool

The specific steps for estimating item parameters for all new items in four designs were as follows:

Step 1 Simulate the abilities of the examinees and their item responses on the new items. The number of new items is $J = 25$ . The number of new items that each examinee answered is set to be no more than $f = 5$ . The number of examinees is $I = N J / f$ . The true ability is drawn from $θ_{i} \sim N (0, 1)$ , and the estimated ability is ${\hat{θ}}_{i} = θ_{i} + ε$ ,where $ε \sim N (0, σ^{2})$ and $i = 1, 2, . . ., N$ . The true ability and true item parameters of the new items are used to simulate the item response matrix, denoted by $U = (u_{i j})$ . The estimated abilities and the item response matrix will be selected sequentially below.

Step 2 Obtain the initial estimation of item parameters. In the initial stage, each new item should receive $n_{j} = 50$ responses. Then, the initial estimation required the number of examinees equal to $N_{0} = (J n_{j}) / f = 250$ . The new items are divided into 5 blocks and each contains five different items. The estimated abilities of the first $N_{0}$ examinees are selected sequentially, that is, ${\hat{θ}}_{i}$ , $i = 1, 2, . . ., N_{0}$ . One block selected in turn is assigned to the examinee ${\hat{θ}}_{i}$ , $i = 1, 2, . . ., N_{0}$ . The corresponding item responses are then selected from the $U$ , denoted by $U_{j} = (\begin{array}{l} u_{i_{1} j} & u_{i_{2} j} & . . . & u_{i_{50} j} \end{array})$ and ${\hat{θ}}_{j} = (\begin{array}{l} {\hat{θ}}_{i_{1} j} & {\hat{θ}}_{i_{2} j} & . . . & {\hat{θ}}_{i_{50} j} \end{array})$ , $j = 1, 2, . . ., J$ . The item parameters are estimated by the Fisher scoring approach. Let $i = N_{0} + 1$ .

Step 3 Select the current active examinee with the estimated ability ${\hat{θ}}_{i}$ .

Step 4 Select the new item $j$ based on the random design or the optimal designs, respectively. If $n_{j} > N$ or item $j$ has been answered by the current active examinee, the criterion is set to 1000 so that item $j$ couldn’t be selected unless in case of $S_{i} = \emptyset$ . The reason is that the value of the criterion for D-optimal design, D-Tp design, or compound optimal design is small and far less than 1000. Item response for the current examinee on item $j$ was selected. Then, the item responses and the ability estimates $U_{j} = (\begin{array}{l} U_{j} & u_{i j} \end{array})$ and ${\hat{θ}}_{j} = (\begin{array}{l} {\hat{θ}}_{j} & {\hat{θ}}_{i} \end{array})$ are used to estimate the item parameters. Let $n_{j} = n_{j} + 1$ and update the set of $S_{i}$ . Repeat this step five times.

Step 5 Let $i = i + 1$ , and repeat Step 3 and 4 until $i > N$ , otherwise, obtain the final estimates of ${\hat{a}}_{j}$ and ${\hat{b}}_{j}$ .

Evaluation Criteria

The bias, absolute deviation (Abs), and mean square error (MSE) of item parameters estimator were used to measure the accuracy of item parameters’ estimates. Their calculation formulas are given by

B i a s ({\hat{a}}_{j}) = \frac{1}{R} \sum_{r = 1}^{R} ({\hat{a}}_{r j} - a_{r j}), B i a s ({\hat{b}}_{j}) = \frac{1}{R} \sum_{r = 1}^{R} ({\hat{b}}_{r j} - b_{r j}),

(24)

A b s ({\hat{a}}_{j}) = \frac{1}{R} \sum_{r = 1}^{R} | {\hat{a}}_{r j} - a_{r j} |, A b s ({\hat{b}}_{j}) = \frac{1}{R} \sum_{r = 1}^{R} | {\hat{b}}_{r j} - {\hat{b}}_{r j} |,

(25)

M S E (a_{j}) = \frac{1}{R} \sum_{j = 1}^{R} {({\hat{a}}_{r j} - a_{r j})}^{2}, M S E ({\hat{b}}_{j}) = \frac{1}{R} \sum_{j = 1}^{R} {({\hat{b}}_{r j} - b_{r j})}^{2} .

(26)

where

R = 100

Results

Tables 1 –6 show the results from the dynamic examinee pool. Tables A1–A3 in the Appendix shows the similar results from the static examinee pool. First, the influence of ability estimation errors on the precision of item parameter estimates under four designs was presented. Table 1 shows the mean squared errors across two factors of item parameters and sample size. It can be seen from Table 1 that no matter which of the above online designs was employed, the mean square error of item parameter estimates increased as the ability estimation error increased, implying that the precision of estimated item parameters decreased. The precision of item parameter estimates was less affected by relatively small error of the ability estimates (i.e., fixed standard error of 0.2 or conditional standard error the shape of the parabolic bowl). While when the standard error of the ability estimation error increased to 0.5, the precision of item parameter estimates was greatly affected by it. Furthermore, the ability estimation error had a relatively large influence on the mean squared errors under four online designs. The ability estimation error did also affect the precision of difficulty parameter estimates.

Table 1.

Estimation Precision of Item Parameters for Four Designs Under Different Ability Errors.

Ability Estimation Errors	MSE ( $\hat{a}$ )				MSE ( $\hat{b}$ )
Ability Estimation Errors	R	D-Tp	D	C	R	D-Tp	D	C
0.0	.536	.493	.485	.276	.239	.223	.227	.138
0.2	.579	.521	.494	.294	.266	.245	.239	.144
0.5	.707	.662	.655	.401	.388	.382	.378	.255
$0.05 θ^{2} + 0.2$	.576	.534	.495	.307	.273	.263	.265	.167
Mean	.599	.553	.532	.319	.291	.278	.277	.176

Table 2.

Estimation Precision of Item Parameters for Four Designs Under Different Sample Sizes.

Sample Size	MSE ( $\hat{a}$ )				MSE ( $\hat{b}$ )
Sample Size	R	D-Tp	D	C	R	D-Tp	D	C
120	.590	.549	.532	.370	.321	.313	.316	.218
180	.591	.550	.527	.310	.291	.278	.271	.171
300	.618	.559	.538	.277	.262	.244	.245	.138
Mean	.599	.553	.532	.319	.291	.278	.277	.176

Table 3.

Bias of Item Parameters for Four Designs With Sample Size of 120 and Ability Error of 0.2.

Discrimination	The Absolute of Difficulty	BIAS( $\hat{a}$ )				BIAS ( $\hat{b}$ )
Discrimination	The Absolute of Difficulty	R	D-Tp	D	C	R	D-Tp	D	C
0.5	0	.002	.020	−.001	.064	−.060	−.084	.011	−.014
	1	−.016	−.023	−.030	.014	.018	−.041	−.046	−.072
	2	−.018	−.061	−.047	−.001	−.001	−.016	.047	.010
1	0	.018	.046	.017	.019	.037	.035	.043	.017
	1	−.011	.005	.000	.013	.015	.007	−.011	.004
	2	−.137	−.152	−.133	−.040	.008	−.058	−.083	−.020
1.5	0	−.084	−.033	−.045	−.079	−.008	−.017	−.004	.001
	1	.007	.031	.046	.013	.004	.001	−.003	.016
	2	−.314	−.394	−.332	−.199	−.020	.044	.003	.051
2	0	−.067	−.049	−.062	−.027	−.004	−.020	−.014	−.020
	1	−.035	−.020	−.010	−.030	.010	.004	.010	.008
	2	−.782	−.863	−.848	−.411	−.037	−.004	−.055	.070
2.5	0	−.149	−.178	−.170	−.182	−.020	−.013	.000	−.005
	1	−.101	−.096	−.090	−.128	−.011	−.011	−.009	−.012
	2	−1.536	−1.166	−1.073	−.624	.023	.020	−.045	.021
Mean		−.215	−.196	−.185	−.107	−.003	−.010	−.010	.004
Standard deviation		.418	.355	.331	.187	.024	.033	.035	.032

Note. The value closest to zero was highlighted in bold.

Table 4.

Abs of Item Parameters for Four Designs With Sample Size of 120 and Ability Error of 0.2.

Discrimination	The AbsoluteDifficulty	ABS( $\hat{a}$ )				ABS( $\hat{b}$ )
Discrimination	The AbsoluteDifficulty	R	D-Tp	D	C	R	D-Tp	D	C
0.5	0	.151	.170	.177	.172	.395	.433	.464	.306
	1	.175	.156	.162	.173	.515	.503	.534	.451
	2	.224	.228	.201	.205	.633	.646	.622	.560
1	0	.205	.211	.181	.242	.181	.161	.166	.160
	1	.215	.216	.221	.213	.292	.245	.243	.206
	2	.385	.417	.393	.323	.494	.496	.469	.404
1.5	0	.259	.248	.257	.282	.116	.119	.122	.116
	1	.277	.289	.280	.276	.177	.148	.134	.132
	2	.745	.729	.712	.578	.525	.532	.500	.398
2	0	.281	.319	.272	.301	.099	.092	.104	.100
	1	.333	.317	.325	.305	.139	.113	.127	.111
	2	1.154	1.091	1.090	.712	.652	.627	.644	.413
2.5	0	.351	.376	.384	.346	.081	.067	.080	.086
	1	.393	.369	.365	.356	.129	.112	.107	.108
	2	1.684	1.377	1.304	.913	.778	.633	.601	.415
Mean		.455	.434	.422	.360	.347	.328	.328	.264
Standard deviation		.428	.357	.345	.212	.236	.227	.221	.162

Note. The minimum value was highlighted in bold.

Table 5.

MSE of Item Parameters for Four Designs With Sample Size of 120 and Ability Error of 0.2.

Discrimination	The AbsoluteDifficulty	MSE ( $\hat{a}$ )				MSE ( $\hat{b}$ )
Discrimination	The AbsoluteDifficulty	R	D-Tp	D	C	R	D-Tp	D	C
0.5	0	.038	.047	.050	.047	.298	.456	.464	.222
	1	.048	.038	.043	.049	.578	.513	.573	.409
	2	.078	.074	.065	.064	.511	.541	.511	.427
1	0	.077	.078	.053	.094	.059	.042	.045	.041
	1	.082	.085	.076	.086	.189	.147	.130	.069
	2	.247	.269	.254	.176	.376	.389	.359	.276
1.5	0	.097	.098	.108	.126	.024	.021	.022	.020
	1	.124	.128	.119	.123	.055	.037	.029	.032
	2	.830	.814	.786	.558	.430	.453	.407	.300
2	0	.120	.156	.121	.140	.017	.013	.015	.016
	1	.173	.152	.167	.150	.033	.020	.026	.021
	2	1.776	1.718	1.681	.907	.580	.561	.573	.311
2.5	0	.178	.197	.195	.175	.010	.008	.009	.011
	1	.206	.192	.186	.183	.028	.029	.021	.021
	2	3.550	2.762	2.619	1.580	.733	.567	.532	.319
Mean		.508	.454	.435	.297	.261	.253	.248	.166
Standard deviation		.956	.774	.741	.422	.254	.242	.241	.160

Note. The minimum value was highlighted in bold.

Table 6.

MSE of Item Parameters for Four Designs With Sample Size of 120 and Conditional Standard Error of $0.05 θ^{2} + 0.2$ .

Discrimination	The AbsoluteDifficulty	MSE(a)				MSE ( $\hat{b}$ )
Discrimination	The AbsoluteDifficulty	R	D-Tp	D	C	R	D-Tp	D	C
0.5	0	.044	.041	.062	.042	.411	.439	.538	.172
	1	.052	.043	.042	.047	.536	.520	.625	.327
	2	.058	.062	.070	.062	.529	.533	.547	.490
1	0	.070	.045	.061	.069	.054	.049	.043	.056
	1	.099	.100	.088	.100	.161	.154	.141	.121
	2	.268	.278	.277	.227	.402	.424	.450	.389
1.5	0	.097	.146	.110	.121	.023	.019	.024	.020
	1	.154	.141	.134	.119	.110	.061	.063	.045
	2	.814	.787	.744	.469	.475	.468	.460	.312
2	0	.131	.146	.137	.136	.015	.014	.018	.013
	1	.185	.184	.199	.202	.036	.033	.038	.030
	2	1.806	1.691	1.477	.903	.615	.562	.485	.324
2.5	0	.168	.181	.178	.164	.011	.008	.009	.012
	1	.204	.190	.196	.212	.025	.021	.025	.024
	2	3.178	2.822	2.688	1.767	.675	.581	.561	.367
Mean		.489	.457	.431	.309	.272	.259	.268	.180
Standard deviation		.874	.782	.729	.460	.252	.243	.252	.169

Note. The minimum value was highlighted in bold.

Second, the impact of sample size on the estimation precision of item parameters for four online designs was considered. Table 2 presents the mean squared errors across two factors of item parameters and ability estimation errors. It was found that the mean square error for both discrimination and difficulty parameters became smaller as the sample size increased for the compound optimal design. The sample size could effectively improve the precision of difficulty parameter estimates for four designs. While for the random design and two D-optimal designs, increasing the sample size did not improve the precision of discrimination parameter estimates.

Tables 3 –5 show the BIAS, ABS, and MSE under the condition that the sample size is 120 and the standard error of the estimated ability is 0.2. Similar results from other conditions were not shown here. It can be seen from the mean for each column or the penultimate row of Tables 3 –5 that for four online designs, the means for BIAS, ABS, and MSE of the item parameter estimation from the compound optimal design were almost the smallest, except that random design has smallest BIAS of difficult parameter than other designs. The performance of the D-Tp and D-optimal designs was very similar and was better than random design. On average, it indicates that the compound optimal design was superior to the other three designs. The variability of the estimation precision for different true item parameters was applied to examine the robustness of four online designs. From the standard deviation for each column or the last row of Table 5, we found that true item parameters had a small impact on the MSE for the compound optimal design. Thus, the compound optimal design performed quite robustly under a variety of item parameters.

From Table 5, for the precision of the estimate of discrimination parameter, the compound optimal design produced smaller MSE ( $\hat{a}$ ) than the D-optimal design and the D-Tp design, when the true value of $a$ was large (i.e., $a \geq 2$ ), or the true value of $a$ was small (i.e., $a \leq 1.5$ ) and the true value of $b$ was larger (i.e., $| b | = 2$ ). For the precision of the estimate of difficult parameter, we found from Table 5 that the compound optimal design obtained smaller MSE ( $\hat{b}$ ) than the D-optimal design or the D-Tp design when the true value of $a$ was small (i.e., $a = 0.5$ ) or the true value of $b$ was larger (i.e., $| b | = 2$ ). For other conditions, the compound optimal design, the D-optimal design, and the D-Tp design achieved comparable results. In addition to the cases with conditional standard error of $0.05 θ^{2} + 0.2$ , similar results are shown in Table 6.

Real Data Analysis for TIMSS Items

In the real data analysis, item responses, item parameters, and abilities are taken from a set of International Mathematical and Scientific Trends Research (TIMSS; Foy et al., 2013). The data comes from a 0–1 scored data set in the CDM package written in R language (Robitzsch et al., 2022). The file name of the data set is “data.timss11.G4.AUT.rda,” which is the fourth-grade Austrian math test data in the 2011 TIMSS. The total sample size is 4,668, which is the same as the number of examinees shown in the related document published on the website (https://timss.bc.edu). The published TIMSS test consists of a total of 175 Items. Codes of 174 items contained in the data set are identical to codes published on the website, while one published item, identified as “M051061Z,” was not included in the R dataset.

The way of the study of You et al. (2010) was used to simulate the procedures of online calibration. The 2PLM was used to fit all item responses in the dataset. The Bilog-MG software was employed to estimate item parameters and ability parameters. Since the TIMSS test was administrated by use of a balanced incomplete block (BIB) design, there is a large number of missing item responses. In the estimation of item parameters, missing item responses were treated as not presented, an option available in Bilog-MG (Du Toit, 2003; Finch, 2008). Only 13 items did not fit 2PLM well because the p-values of the chi-square statistics for these items are less than 0.05. The item parameter estimates were taken as the true values of item parameters. The mean of discrimination and difficulty were 1.002 and −0.0673, respectively. By default, expected posterior estimates of ability from Bilog-MG were regarded as the ability estimates for examinees taking the CAT or the field test, which would be used in the online calibration. Online calibration of item parameters of the 174 items was very similar to the procedures under the dynamic examinee pool in the simulation study, except that the simulation of item responses and the ability estimates were not necessary, because real item responses and the ability outputs from Bilog-MG were available. The active examinee was sequentially selected in the order of the row of item response matrix. It should be noted that measurement errors were not needed to add in the estimation ability because there were errors in the ability outputs. For the design of online calibration of item parameters, the ability design points were generated by one of the four online designs under the dynamic examinee pool and the corresponding item responses without missing of examinees were then selected. The estimation of item parameters and the selection of item responses were repeated until the number of item responses reached 130. Because the sample size is 4,668, the number of items is 174, and 5 items are seeded to each examinee, the mean sample size for each item is $4668 \times 5 / 174 = 134.1$ , which rounded to the nearest ten is 130.

Considering the randomness of design points in the initial estimation stage, Table 7 summarizes the mean square errors of item parameter estimates across the 100 replications. The results show that, on average, the mean square errors of the discrimination and difficulty estimates obtained by the compound optimal design were relatively small. Under most conditions, the new design produced better item parameters estimates than the existing designs. Particularly, when the discrimination was large, the precision of the estimation of the discrimination parameter improved markedly. The precision of item parameters for the new design in the real study was not as good as that of the simulation study when the standard deviation of the estimated ability error was 0.2 and the number of examinees answered the new item was 120. The reason is that each examinee only answered about 25 items in the BIB design resulted in a larger amount error of the ability estimates. For example, the averaged standard error of the ability outputs from Bilog-MG was about 0.44.

Table 7.

Results of Empirical Study for Four Online Designs.

Range of Discrimination	Range of Difficulty	The Number of Items	MSE ( $\hat{a}$ )				MSE ( $\hat{b}$ )
Range of Discrimination	Range of Difficulty	The Number of Items	R	D-Tp	D	C	R	D-Tp	D	C
0<a≤1	\|b\|≤.5	26	.118	.131	.125	.132	.090	.195	.187	.082
	0.5<\|b\|≤1	25	.117	.137	.109	.122	.100	.089	.110	.072
	1<\|b\|≤1.5	18	.119	.162	.145	.124	.295	.322	.324	.248
	1.5<\|b\|≤2	10	.134	.169	.136	.140	.332	.387	.388	.295
	\|b\|>2	15	.193	.262	.256	.263	1.800	1.925	1.898	1.932
1<a≤1.5	\|b\|≤.5	15	.243	.355	.252	.238	.018	.017	.016	.018
	0.5<\|b\|≤1	15	.272	.313	.255	.265	.038	.034	.036	.039
	1<\|b\|≤1.5	14	.286	.328	.316	.317	.070	.087	.083	.069
	1.5<\|b\|≤2	8	.396	.384	.396	.394	.156	.154	.156	.119
	\|b\|>2	11	.693	.752	.704	.638	.394	.399	.409	.393
a>1.5	\|b\|≤.5	7	.545	.528	.480	.394	.011	.007	.007	.008
	0.5<\|b\|≤1	5	.489	.367	.327	.471	.035	.023	.022	.032
	1<\|b\|≤1.5	3	.699	.742	.827	.829	.062	.051	.083	.085
	1.5<\|b\|≤2	2	.829	.638	.533	.445	.224	.112	.178	.162
Sum or mean		174	.367	.376	.347	.341	.259	.272	.278	.254

Note. The minimum value was highlighted in bold.

Conclusion and Discussion

Online calibration of new items in the operational CAT provided an economical and efficient way for maintaining and expanding of item bank. Through the analysis of existing related literatures, it was found that the D-optimal design has some advantages but it results a large estimation error of discrimination parameters for high discrimination items or difficulty parameters for low discrimination items. Moreover, previous studies have shown that the frequent use of high discrimination parameter items with large estimation errors in CAT has the potential effects of capitalization on chance (Patton et al., 2013), which ultimately seriously affects the precision of CAT ability estimation. Therefore, it is very important to investigate a new design for online item calibration under the 2PLM.

The contribution of our study to the literature has two aspects: First, we propose the compound optimal design with two optimality criteria for online calibration of item parameters of new items under the 2PLM. The design using the acceptance probability adaptively selects design points for optimizing the parameter which is difficult to estimate. The new design improves the estimation precision of item difficulty and discrimination parameters in simultaneous. Second, the compound optimal design is suitable for both the static examinees pool and the dynamic examinees pool. The static examinees can be obtained from group-based testing or a whole group (e.g., school class or school grade) of test-takers undergoing assessment concurrently (Bengs et al., 2021). The concurrent test-taking aligns with usual classroom testing or state testing programs, such as the VERA (Vergleichsarbeiten). The VERA are comparison tests that students take in the 3rd and 8th grades (VERA-3 and VERA-8) in Germany. In recent years, several German federal states decided to implement the computer-based testing of the VERA tests (Wagner et al., 2022). For the situation, a static examinee pool is available when there are many students attending computer-based testing in parallel on the operational item pool. In addition, the static examinees can be sampled based on ability-related background variables, such as grades or courses taken in school (Berger et al., 2019), because the background variables are usually related to the ability (Eggen & Verhelst, 2011; Mislevy & Wu, 1996). Furthermore, if the number of students attending computer-based testing in parallel is small, a dynamic examinee pool is available for sequential designs in the testing period.

The simulation study shows that the compound optimal design can achieve an efficient estimation of the discrimination and difficulty parameters at the same time. In other words, the compound optimal design can control the estimation error of discrimination and difficulty parameters simultaneously. With the same sample size, the compound optimal design outperforms the D-optimal design and random design in terms of the precision of item parameters. The compound optimal design also performs well in real data analysis. Thus, the compound optimal design can obtain the simultaneous estimation of item difficulty and discrimination parameters, which can be used for online calibration of item parameters.

The advantage of the compound optimal design is that the practitioner can focus on the estimation of important item parameters that is vital for item selection and ability estimates. It can directly control the estimation error of item parameters through the selection of the ability design points by the definition of the acceptance probability. Whether complex methods for the simultaneous maximization of two objective functions in the optimization of designs can help to improve the performance of the designs which require further study. The fixed sample size, the 2PLM, and one online estimation method were considered in the study. Other optimal design criteria, optimal design method, and stopping rule should be further studied for the replenishment of item bank. For example, we should extend the compound optimal design and compare it with the optimal Bayesian adaptive design (He & Chen, 2020; Ren et al., 2017; van der Linden & Ren, 2015). For the 2PLM, if one of item parameters (i.e., difficulty parameter) has fast achieved a relatively small estimation error, then it will leave a large number of examinees to estimate other item parameter (i.e., discrimination parameter). Thus, the stopping rule needs further study. The 2PLM was used in the study, while optimal designs under other models in the framework of item response theory need to be further considered. Besides, whether optimal designs and online estimation methods would be interacted with each other needs to be further discussed. In the study, ability estimation was only used for online calibration of item parameters on new items. It is worth paying attention to how to estimate and optimize the item parameters of the new item with the help of other auxiliary information, such as response times and related characteristics of test items (He et al., 2021; Kang et al., 2020; Li & Kuang, 2023). How to generalize the new design to multidimensional item response theory (MIRT) is an interesting problem, because some studies have extended related methods to the MIRT (Chen, 2017; Chen et al., 2017; Chen & Wang, 2016).

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the National Natural Science Foundation of China (grant numbers 62067005, 62267004, and 62467003).

ORCID iDs

Lihong Song

Wenyi Wang

Appendix

References

Babcock

Weiss

D. J.

(2012). Termination criteria in computerized adaptive tests: Do variable-length CATs provide efficient and effective measurement. Journal of Computerized Adaptive Testing, 1(1), 1–18. https://doi.org/10.7333/1212-0101001

Ban

J.-C.

Hanson

B. A.

Wang

Harris

D. J.

(2001). A comparative study of on-line pretest item: Calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38(3), 191–212. https://doi.org/10.1111/j.1745-3984.2001.tb01123.x

Ban

J.-C.

Hanson

B. A.

Harris

D. J.

(2002). Data sparseness and on-line pretest item calibration-scaling methods in CAT. Journal of Educational Measurement, 39(3), 207–218. https://doi.org/10.1111/j.1745-3984.2002.tb01174.x

Bengs

Kroehne

Brefeld

(2021). Simultaneous constrained adaptive item selection for group-based testing. Journal of Educational Measurement, 58(2), 236–261. https://doi.org/10.1111/jedm.12285

Berger

M. P. F.

(1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521–538. https://doi.org/10.1007/bf02294418

Berger

Verschoor

A. J.

Eggen

T. J. H. M.

Moser

(2019). Efficiency of targeted multistage calibration designs under practical constraints: A simulation study. Journal of Educational Measurement, 56(1), 121–146. https://doi.org/10.1111/jedm.12203

Cao

Chen

Biegler

L. T.

(2024). Sequential design of experiments for parameter estimation with Markov chain Monte Carlo. In Manenti

Reklaitis

G. V.

(Eds.), Computer aided chemical engineering (Vol. 53, pp. 3199–3204). Elsevier. https://doi.org/10.1016/b978-0-443-28824-1.50534-2

Chang

H.-H.

(2004). Understanding computerized adaptive testing: From Robbins-Monro to Lord and beyond. In Kaplan

D. W.

(Ed.), The Sage handbook of quantitative methodology for the social sciences. Sage Publications, Inc.

Chang

H.-H.

Wang

Ying

(2016). Information theory and its application to testing. In Linden

W. J. v. d.

(Ed.), Handbook of item response theory, volume two, statistical tools (pp. 105–123). CRC Press, Taylor & Francis Group.

10.

Chang

H.-H.

Ying

(2007). Computerized adaptive testing. In Salkind

N. J.

Rasmussen

(Eds.), Encyclopedia of measurement and statistics (Vol. 1, pp. 170–173). Sage Publications, Inc.

11.

Chang

Y.-C. I.

H.-Y.

(2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75(1), 140–157. https://doi.org/10.1007/s11336-009-9133-0

12.

Chen

(2016). Two new online calibration methods for computerized adaptive testing. Acta Psychologica Sinica, 48(9), 1184. https://doi.org/10.3724/sp.j.1041.2016.01184

13.

Chen

(2017). A comparative study of online item calibration methods in multidimensional computerized adaptive testing. Journal of Educational and Behavioral Statistics, 42(5), 559–590. https://doi.org/10.3102/1076998617695098

14.

Chen

Wang

(2016). A new online calibration method for multidimensional computerized adaptive testing. Psychometrika, 81(3), 674–701. https://doi.org/10.1007/s11336-015-9482-9

15.

Chen

Wang

Xin

Chang

H.-H.

(2017). Developing new online calibration methods for multidimensional computerized adaptive testing. British Journal of Mathematical and Statistical Psychology, 70(1), 81–117. https://doi.org/10.1111/bmsp.12083

16.

Chen

Xin

(2014). Online calibration with cognitive diagnostic assessment. In Cheng

Chang

H.-H.

(Eds.), Advancing methodologies to support both summative and formative assessments (pp. 287–313). Information Age.

17.

Choi

S. W.

Grady

M. W.

Dodd

B. G.

(2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71(1), 37–53. https://doi.org/10.1177/0013164410387338

18.

Du Toit

(Ed.), (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Scientific Software International, Inc.

19.

Eccleston

J. A.

Whitaker

(1999). On the design of optimal change-over experiments through multi-objective simulated annealing. Statistics and Computing, 9(1), 37–42. https://doi.org/10.1023/a:1008810109585

20.

Eggen

T. J. H. M.

Verhelst

N. D.

(2011). Item calibration in incomplete testing designs. Psicológica, 32(1), 107–132. https://www.uv.es/psicologica/articulos1.11/7EGGEN.pdf

21.

Finch

(2008). Estimation of item response theory parameters in the presence of missing data. Journal of Educational Measurement, 45(3), 225–245. https://doi.org/10.1111/j.1745-3984.2008.00062.x

22.

Foy

Arora

Stanco

G. M.

(Eds.). (2013). TIMSS 2011 user guide for the international database. TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College, and International Association for the Evaluation of Educational Achievement (IEA).

23.

Givens

G. H.

Hoeting

J. A.

(2013). Computational statistics. John Wiley & Sons, Inc.

24.

Hassan

M. U.

Miller

(2024). Optimal calibration of items for multidimensional achievement tests. Journal of Educational Measurement, 61(2), 274–302. https://doi.org/10.1111/jedm.12386

25.

Reckase

M. D.

(2014). Item pool design for an operational variable-length computerized adaptive test. Educational and Psychological Measurement, 74(3), 473–494. https://doi.org/10.1177/0013164413509629

26.

Chen

(2020). Optimal online calibration designs for item replenishment in adaptive testing. Psychometrika, 85(1), 35–55. https://doi.org/10.1007/s11336-019-09687-0

27.

Chen

(2020). New efficient and practicable adaptive designs for calibrating items online. Applied Psychological Measurement, 44(1), 3–16. https://doi.org/10.1177/0146621618824854

28.

Chen

(2021). Maximum information per time unit designs for continuous online item calibration. British Journal of Mathematical and Statistical Psychology, 74(Suppl 1), 24–51. https://doi.org/10.1111/bmsp.12221

29.

Chen

Zhang

(2017). A new online calibration method based on Lord’s bias-correction. Applied Psychological Measurement, 41(6), 456–471. https://doi.org/10.1177/0146621617697958

30.

Huebner

(2010). An overview of recent developments in cognitive diagnostic computer adaptive assessments. Practical Assessment, Research and Evaluation, 15(3), 1–7. https://doi.org/10.7275/7fdd-6897

31.

Jones

D. H.

Jin

(1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59(1), 59–75. https://doi.org/10.1007/bf02294265

32.

Kang

H.-A.

Zheng

Chang

H.-H.

(2020). Online calibration of a joint model of item responses and response times in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 45(2), 175–208. https://doi.org/10.3102/1076998619879040

33.

Kitsos

C. P.

(2013). Optimal experimental design for non-linear models: Theory and applications. Springer.

34.

Kuang

(2023). The new item selection strategies in CAT in combination with item response time and item discrimination. Journal of Jiangxi Normal University (Nature Science), 47(4), 377–382. https://doi.org/10.16357/j.cnki.issn1000-5862.2023.04.07

35.

Lord

F. M.

(1971). Robbins-Monro procedures for tailored testing. Educational and Psychological Measurement, 31(1), 3–31. https://doi.org/10.1177/001316447103100101

36.

Magis

Yan

von Davier

A. A.

(2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.

37.

McGree

J. M.

Eccleston

J. A.

Duffull

S. B.

(2008). Compound optimal design criteria for nonlinear models. Journal of Biopharmaceutical Statistics, 18(4), 646–661. https://doi.org/10.1080/10543400802071352

38.

Mislevy

R. J.

(1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing (ETS Research Reports Series No. RR-96-30-ONR). Educational Testing Service.

39.

Olea

Barrada

J. R.

Abad

F. J.

Ponsoda

Cuevas

(2012). Computerized adaptive testing: The capitalization on chance problem. Spanish Journal of Psychology, 15(1), 424–441. https://doi.org/10.5209/rev_sjop.2012.v15.n1.37348

40.

Patton

J. M.

Cheng

Yuan

K. H.

Diao

(2013). The influence of item calibration error on variable-length computerized adaptive testing. Applied Psychological Measurement, 37(1), 24–40. https://doi.org/10.1177/0146621612461727

41.

Quellmalz

E. S.

Pellegrino

J. W.

(2009). Technology and testing. Science, 323(5910), 75–79. https://doi.org/10.1126/science.1168046

42.

Reckase

M. D.

(2010). Designing item pools to optimize the functioning of a computerized adaptive test. Psychological Test and Assessment Modeling, 52(2), 127–141. https://ptam-journal.com/wp-content/uploads/2025/01/01_Reckase.pdf

43.

Ren

van der Linden

W. J.

Diao

(2017). Continuous online item calibration: Parameter recovery and item utilization. Psychometrika, 82(2), 498–522. https://doi.org/10.1007/s11336-017-9553-1

44.

Robitzsch

Kiefer

George

A. C.

Uenlue

(2022). CDM: Cognitive Diagnosis Modeling. R package version 8.2-6. https://cran.r-project.org/web/packages/CDM/index.html

45.

Silvey

S. D.

(1980). Optimal design. Chapman and Hall Ltd.

46.

Stocking

M. L.

(1988). Scale drift in on-line calibration (RR-88-28). Educational Testing Service.

47.

Stocking

M. L.

(1994). Three practical issues for modern adaptive testing item pools. Educational Testing Service.

48.

Ul Hassan

Miller

(2019). Optimal item calibration for computerized achievement tests. Psychometrika, 84(4), 1101–1128. https://doi.org/10.1007/s11336-019-09673-6

49.

van der Linden

W. J.

Glas

C. A. W.

(2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53. https://doi.org/10.1207/s15324818ame1301_2

50.

van der Linden

W. J.

Ren

(2015). Optimal bayesian adaptive design for test-item calibration. Psychometrika, 80(2), 263–288. https://doi.org/10.1007/s11336-013-9391-8

51.

Wagner

Loesche

Bißantz

(2022). Low-stakes performance testing in Germany by the VERA assessment: Analysis of the mode effects between computer-based testing and paper-pencil testing. European Journal of Psychology of Education, 37(2), 531–549. https://doi.org/10.1007/s10212-021-00532-6

52.

Wainer

Dorans

N. J.

Eignor

Flaugher

Green

B. F.

Mislevy

R. J.

Thissen

(2000). Computerized adaptive testing: A primer. Lawrence Erlbaum Associates.

53.

Wainer

Mislevy

R. J.

(1990). Item response theory, item calibration, and proficiency estimation. In Wainer

(Ed.), Computerized adaptive testing: A primer (pp. 65–102). Erlbaum.

54.

Weiss

D. J.

Kingsbury

G. G.

(1984). Application of computerized adaptive testing to educational problems. Journal of Educational Measurement, 21(4), 361–375. https://doi.org/10.1111/j.1745-3984.1984.tb01040.x

55.

Yan

von Davier

A. A.

Lewis

(2014). Computerized multistage testing: Theory and applications. CRC Press.

56.

You

X.-F.

Ding

S.-L.

Liu

H.-Y.

(2010). Parameter estimation of the raw item in computerized adaptive testing. Acta Psychologica Sinica, 42(7), 813–820. https://doi.org/10.3724/sp.j.1041.2010.00813

57.

Zheng

(2016). Online calibration of polytomous items under the generalized partial credit model. Applied Psychological Measurement, 40(6), 434–450. https://doi.org/10.1177/0146621616650406