Alternative Smoothing and Scaling Strategies for Weighted Composite Scores

Abstract

In this study, smoothing and scaling approaches are compared for estimating subscore-to-composite scaling results involving composites computed as rounded and weighted combinations of subscores. The considered smoothing and scaling approaches included those based on raw data, on smoothing the bivariate distribution of the subscores, on smoothing the bivariate distribution of the subscore and weighted composite, and two weighted averages of the raw and smoothed marginal distributions. Results from simulations showed that the approaches differed in terms of their estimation accuracy for scaling situations with smaller and larger sample sizes, and on weighted composite distributions of varied complexity.

Keywords

scaling weighted composites smoothing

Introduction

The creation of a composite score as a weighted combination of subscores is a common practice in large-scale testing, arising when the construct of interest is assessed with a battery of subtests and content areas, or when the test contains subtests with items of different formats (e.g., multiple-choice and constructed response items and subtests). The weights of the subscores used in forming the weighted composite have implications for the measurement characteristics of the composite score (i.e., reliability and validity). Other implications involve the smoothing, scaling, and equating procedures used to maintain consistent reported scores across alternate composite forms, or to express equating results for one of the subscores on the scale of the weighted composite. The interest of this study is the influence of subscores’ weights on the smoothing and scaling procedures used to obtain subscore-to-composite scaling results. The next section describes three subscore weighting choices which result in simple and more complex subscore-to-composite scaling situations commonly encountered in practice. For these three situations, four approaches to smoothing the score distributions and converting the scores of one of the subscores to the weighted composite scale are described and then compared in simulations.

Three Examples of Composite Score Weightings and Distributions

Three examples of weighted composites encountered in testing programs’ tests are illustrated using the single group data from von Davier, Holland, and Thayer (2004, p. 115). These data contain the scores on two 20-point tests, X and Y, as taken by one group of 1,453 examinees. The X and Y data are summarized in Tables 1 to 3 and are used in weighted form to produce three hypothetical rounded and weighted composites as $composite = round (w_{X} X + w_{Y} Y)$ . The three weighted composites summarized in Tables 1 to 3 are obtained from weighting systems of $w_{X} = w_{Y} = 1$ (Table 1), $w_{X} = 1$ and $w_{Y} = 1.5$ (Table 2), and $w_{X} = 1.2$ and $w_{Y} = 0.8$ (Table 3). In large-scale testing situations, these three sets of weights might be selected based on several considerations, including the variances or timing of X and Y; the content of X, Y, and the composite; or on producing maximum possible weighted X and Y scores that are desired proportions of the maximum possible composite score. Proportions of maximum possible weighted scores are an especially common and familiar justification of testing programs, resulting in a fixed set of weights that helps maintain consistent and explainable scoring procedures across test administrations. The three sets of weights for the weighted composites in Tables 1 to 3 might be used to supportintensions for the maximum weighted scores of X and Y being 50% of the weighted composite for the $w_{X} = w_{Y} = 1$ weights, 40% and 60% for the $w_{X} = 1$ and $w_{Y} = 1.5$ weights, and 60% and 40% for the $w_{X} = 1.2$ and $w_{Y} = 0.8$ weights.

Table 1.

Descriptive Stats of X, Y, and Composite, $round (w_{X} X + w_{Y} Y)$ , $w_{X} = w_{Y} = 1$ .

	X	Y	Composite
Observed (and possible) min	0 (0)	2 (0)	2 (0)
Observed (and possible) max	20 (20)	19 (20)	39 (40)
Mean	10.818	10.389	21.207
SD	3.807	3.588	6.968
Skew	0.003	−0.006	0.001
Kurtosis	−0.465	−0.513	−0.496
N	1,453	1,453	1,453
	Correlations
X	1.000
Y	0.775	1.000
Composite	0.946	0.939	1.000

Table 2.

Descriptive Stats of X, Y and Composite, $round (w_{X} X + w_{Y} Y)$ , $w_{X} = 1$ and $w_{Y} = 1.5$ .

	X	Y	Composite
Observed (and possible) min	0 (0)	2 (0)	2 (0)
Observed (and possible) max	20 (20)	19 (20)	49 (50)
Mean	10.818	10.389	26.652
SD	3.807	3.588	8.686
Skew	0.003	−0.006	0.000
Kurtosis	−0.465	−0.513	−0.497
N	1,453	1,453	1,453
	Correlations
X	1.000
Y	0.775	1.000
Composite	0.920	0.960	1.000

Table 3.

Descriptive Stats of X, Y and Composite, $round (w_{X} X + w_{Y} Y)$ , $w_{X} = 1.2$ and $w_{Y} = 0.8$ .

	X	Y	Composite
Observed (and possible) min	0 (0)	2 (0)	2 (0)
Observed (and possible) max	20 (20)	19 (20)	39 (40)
Mean	10.818	10.389	21.286
SD	3.807	3.588	7.039
Skew	0.003	−0.006	−0.002
Kurtosis	−0.465	−0.513	−0.495
N	1,453	1,453	1,453
	Correlations
X	1.000
Y	0.775	1.000
Composite	0.965	0.911	1.000

Some implications of the three sets of weights for composite scaling procedures are evident in the three weighted composite score distributions from Tables 1 to 3. Figures 1 to 3 plot two forms of the distributions of the composite scores based on the three sets of X and Y weights and the bivariate XY population distribution used in this study (described in the “Method” section). For Figures 1 to 3, the A-graphs show the expected number of examinees at each weighted composite score and the B-graphs show the total number of score combinations of X and Y that produce each rounded and weighted composite score. Plotting the numbers of composite scores based on the weighted score combinations of X and Y (B) along with the numbers of examinees at each composite score (A) is useful for showing the correspondence of the composite score distributions with the popularity of the composite scores as determined by the sets of weights. For the $w_{X} = w_{Y} = 1$ and $w_{X} = 1$ and $w_{Y} = 1.5$ sets of weights, Figures 1 and 2 show that the weighted composite score distributions of examinees are relatively simple, where the X and Y weights produce weighted composite distributions that are peaked to greater degrees (Figure 1) and lesser degrees (Figure 2). For the $w_{X} = 1.2$ and $w_{Y} = 0.8$ set of weights, Figure 3 shows that the sets of weights used in creating the weighted composite score distribution produces a structure that is essentially two distributions corresponding to more frequently occurring even scores and less frequently occurring odd scores. Altogether, Figures 1 to 3 reflect a range of complexity in weighted composite score distributions that has uncertain performance implications for different approaches to obtaining a subscore-to-compositescaling.

Four Approaches to X-to-Composite Scaling

The approaches of interest for converting the scores of one of the subscores (i.e., X) to the scale of the weighted composite all involve the estimation of an equipercentile relationship for X and the weighted composite,

{\overset{⌢}{e}}_{c} (x_{j}) = {\overset{⌢}{G}}^{- 1} [\overset{⌢}{F} (x_{j})]

where $\overset{⌢}{F} (x_{j})$ and $\overset{⌢}{G} []$ are percentile rank functions obtained from the distributions of X and the weighted composite (Kolen & Brennan, 2004). The four approaches to implementing Equation (1) use different estimates of the distributions of X and the weighted composite.

Raw data

Probably the most direct approach to implementing Equation (1) to produce an X-to-composite scaling is the raw test score data from the group of examinees who take tests X and Y (Raw Data). This approach has the advantage of being relatively simple. However, raw equipercentile results are often not completely “raw.” Equipercentile estimates can be awkward and problematic with only raw data, such as when specific scores of X, Y, and the composite are unobserved in the raw data and the resulting percentile estimates are not unique for every possible score. There are several ad hoc approaches to addressing problems due to unobserved scores for equipercentile conversions, and the one of interest in this study involves estimating the probabilities of X (and similarly for the composite) from the observed data ( $r_{j}$ ) averaged with a small constant (Kolen & Brennan, 2004, p. 48),

{\overset{⌢}{r}}_{R a w, j} = \frac{r_{j} + . 00001}{1 + . 00001 (X_{J} + 1)}

Equipercentile results from a Raw Data approach would be expected to be relatively unbiased because whatever structures might occur in the distributions of X and the composite ought to be visible in the observed data (Moses & Holland, 2007). Raw Data results would also be expected to be highly variable due to the relatively strong influence of sampling variability on the estimated probabilities.

Smoothing the bivariate X–composite distribution

In practice, Equation (1) is usually implemented using smoothed probabilities rather than raw probabilities of the X and weighted composite distributions. The smoothing is intended to improve the accuracy of the estimated equipercentile results by reducing the influence of sampling variability but not increasing estimation bias very much (Moses & Holland, 2007). A common smoothing approach involves fitting a loglinear model to the test score distributions (Holland & Thayer, 2000). The type of loglinear model most likely considered for an X-to-composite situation would relate the log of the expected bivariate probabilities as a linear function of the X and composite scores,

\log_{e} (p_{jl}) = α + \sum_{i = 1}^{I} β_{xi} {(x_{j})}^{i} + \sum_{h = 1}^{H} β_{ch} {(c_{l})}^{h} + β_{xc} (x_{j}) (c_{l}),

where, with maximum likelihood estimation of the $β$ s, the values of I and H result in the first I observed moments of X, the first H observed moments of the weighted composite, and the $β_{xc} (x_{j}) (c_{l})$ term results in the observed covariance of X and the weighted composite being preserved in the modeled distribution. Marginal probabilities for X and the weighted composite for the equipercentile calculations are obtained directly from $p_{jl}$ . This smoothing approach, abbreviated as XComp, is direct, often used in practice and is presumably what would be recommended based on equating texts describing single group data (e.g., von Davier et al., 2004, pp. 113-130). One drawback of this approach is that it can be difficult to capture structures that might arise in weighted composite distributions based on specific subscore weights (e.g., Figure 3). This suggests that the XComp smoothing approach has the potential for estimation bias.

Smoothing the bivariate XY distribution

Another smoothing approach involves fitting a loglinear model to the bivariate distribution of X and Y,

\log_{e} (p_{jk}) = α + \sum_{i = 1}^{I} β_{xi} {(x_{j})}^{i} + \sum_{h = 1}^{H} β_{yh} {(y_{k})}^{h} + β_{xy} (x_{j}) (y_{k}),

and obtaining the marginal distributions of X and the weighted composite by appropriately aggregating the $p_{jk}$ s based on the rounded weighted score combinations of X and Y. This XY smoothing approach would more closely reflect the population distributions regarded as arising from X and Y rather than X and a composite that is partially based on X. However, it is also indirect with respect to the composite that is of interest in the scaling. Loglinear models are often selected based on fitting moments of the score distributions involved in scalings and equatings. The moments of the unrounded weighted composite can be indirectly preserved in the XY loglinear model through specific cross-moments of X and Y. In particular, the $composite = round (w_{X} X + w_{Y} Y)$ relationship means that the noncentral moments of the unrounded composite can be expressed in terms of the noncentral moments and cross-moments of X and Y,

\sum_{j, k} p_{jk} (w_{X} x_{j} + w_{Y} y_{k}) = w_{X} \sum_{j} (\sum_{k} p_{jk}) x_{j} + w_{Y} \sum_{k} (\sum_{j} p_{jk}) y_{k}

\sum_{j, k} {p_{jk} (w_{X} x_{j} + w_{Y} y_{k})}^{2} = \sum_{j, k} p_{jk} [{(w_{X} x_{j})}^{2} + 2 w_{X} w_{Y} x_{j} y_{k} + {(w_{Y} y_{k})}^{2}]

\sum_{j, k} {p_{jk} (w_{X} x_{j} + w_{Y} y_{k})}^{3} = \sum_{j, k} p_{jk} [{(w_{X} x_{j})}^{3} + 3 {(w_{X} x_{j})}^{2} w_{Y} y_{k} + 3 w_{X} x_{j} {(w_{Y} y_{k})}^{2} + {(w_{Y} y_{k})}^{3}]

\sum_{j, k} p_{jk} w_{X} x_{j} (w_{X} x_{j} + w_{Y} y_{k}) = \sum_{j, k} p_{jk} [{(w_{X} x_{j})}^{2} + w_{X} x_{j} w_{Y} y_{k}]

Thus, to fit the observed mean, variance, and skew of the unrounded weighted composite and its observed covariance with X, the model in Equation (4) could be extended to include all the X and Y terms in Equations (5) to (8),

\log_{e} (p_{jk}) = α + \sum_{i = 1}^{I} β_{xi} {(x_{j})}^{i} + \sum_{h = 1}^{H} β_{yh} {(y_{k})}^{h} + β_{xy, 1} (x_{j}) (y_{k}) + β_{xy, 2} {(x_{j})}^{2} (y_{k}) + β_{xy, 3} (x_{j}) {(y_{k})}^{2} .

Weighted averages of the raw and smoothed data

A final smoothing approach is considered that is an attempt to improve the accuracy of the Raw Data approach in Equation (2). By noting that Equation (2) is essentially a weighted average of the observed probabilities of the marginal distributions and the probabilities from uniform distributions, it is possible to consider a more general Weighted Average approach defined as,

w_{R a w} r_{j} + (1 - w_{R a w}) {\overset{⌢}{p}}_{j}, 0 \leq w_{R a w} \leq 1 .

where $r_{j}$ is the observed probability and ${\overset{⌢}{p}}_{j}$ is the estimated probability from a uniform distribution (e.g., Equation 2) or from a loglinear model of the marginal distribution of X,

\log_{e} ({\overset{⌢}{p}}_{j}) = α + \sum_{i = 1}^{I} {\overset{⌢}{β}}_{x i} {(x_{j})}^{i}

(Agresti, 1990; Bishop, Fienberg, & Holland, 1975; Fienberg & Holland, 1973; Moses & Oh, 2009). Equation (10) has been described as a pseudo or empirical Bayes estimate in that standard multinomial assumptions for $r_{j}$ and other assumptions for the prior distribution of population probability $p_{j}$ imply that Equation (10) produces an estimate of the posterior mean of p given the sample size. The pseudo aspect of the estimate in Equation (10) pertains to data-dependent choices of $w_{Raw}$ and ${\overset{⌢}{p}}_{j}$ . Data-adaptive choices for $w_{Raw}$ have been proposed (Fienberg & Holland, 1973) and considered with respect to equating accuracy implications (Moses & Oh, 2009). Moses and Oh’s (2009) equating research suggests that estimation bias can be considerably reduced relative to the use of the uniform distribution in the Raw Data approach by using a weighted average with the estimates from a loglinear model like Equation (11) that fits the major features of the observed data. The Weighted Average approach is a relatively simple approach to improving scaling accuracy of the Raw Data approach.

Assessing the Four Smoothing and Scaling Approaches for the Three Weighted Composites

The approaches for X-to-composite scaling described in the previous section vary in terms of their simplicity and plausibility. Although the XY smoothing approach would seem to be a more accurate reflection of the population model that produced the subscore data, it can also be more complex to implement and perhaps not appreciably better than the other three approaches in terms of accuracy, simplicity, and efficiency. This study was intended to compare the approaches’ accuracy implications for the three weighted composite situations described in Figures 1 to 3.

Method

The smoothing and scaling approaches of interest were compared in a series of simulations. A loglinear model of the form in Equation (9) was fit to the von Davier et al. (2004)XY data (Tables 1 -3) and treated as the population model for the XY distribution. From the XY population model, three composite score distributions were obtained from the three sets of weights for X and Y (Tables 1 -3, Figures 1 -3), and equipercentile X-to-composite scaling functions based on Equation (1) were calculated for the three composites and treated as three population scaling functions. Finally, samples of 1,000 and 10,000 examinees’XY score combinations were randomly drawn from the XY population model and the four smoothing and scaling approaches were used to estimate the population X-to-composite scaling functions in the sample data. The accuracies of the smoothing and scaling approaches were summarized by averaging the squared deviations of these methods’ sample estimates from the population values.

Population Distributions and Scaling Functions

The population model used to study the smoothing and scaling approaches was chosen based on the suggested loglinear model from von Davier et al.’s (2004) investigations, using I and H values of 3 in Equation (9) to fit the mean, variance, and skewness of X and Y, and also fitting the XY, covariance. Fitting the mean, variance, and skewness of the three unrounded weighted composite score distributions was also desired because it was consistent with how the X and Y distributions were modeled, and so as suggested in Equation (9) the results of Equations (5) to (8) were incorporated by including and fitting the XY² and X²Y cross-moments. Population X-to-composite scaling functions were obtained by first computing the three weighted composite score distributions from the XY population model using one of the three sets of weights for X and Y and then calculating the three equipercentile X-to-composite scaling functions (Equation 1).

Smoothing and Scaling Approaches

The Raw Data, XComp smoothing, XY smoothing, and Weighted Average smoothing approaches described in the “Introduction” section were assessed in the simulations of this study. To be consistent with the population model, the smoothing models used with the XComp smoothing, XY smoothing, and weighted average approaches were implemented by fitting the means, variances, and skewnesses of the marginal distributions these approaches modeled. The XY smoothing was implemented by fitting Equation (9) with I = H = 3. The XComp smoothing was implemented by fitting Equation (3) with I = H = 3. The two estimated models used with the Weighted Average approach for the marginal distributions of X and the composite were obtained as loglinear models like Equation (11) that fit the means, variances, and skewnesses of the marginal distributions. For the Weighted Average approach several weights were considered and $w_{Raw}$ values of .99 (i.e., 99% raw data) and .01 (i.e., 1% raw data) were used because initial simulation results showed that these weights produced particularly accurate results at the two considered sample sizes.

Simulations and Evaluations

The simulations involved drawing 500 independent and random samples of 1,000 scores and 500 other independent and random samples of 10,000 scores from the XY population model and computing the samples’X-to-composite scaling with the considered smoothing and scaling approaches. The values of 1,000 and 10,000 were selected because they produced results which are illustrative of the approaches’ performance tendencies for data sets with smaller and larger sample sizes. The simulation process was repeated for each of the three X and Y weights and their corresponding X-to-composite scaling functions. Comparisons of the smoothing and scaling approaches’ estimates with the dth random sample of size N, ${\overset{⌢}{e}}_{c, s m o o t h i n g, N, d} (x_{j})$ , and the population X-to-composite scaling function value, $e_{c} (x_{j})$ , were summarized in terms of root mean squared errors (RMSE),

R M S E_{s m o o t h i n g, N} = (\frac{100}{σ_{c}}) \sqrt{\sum_{j} p_{j} \frac{1}{500} \sum_{d = 1}^{500} {[{\overset{⌢}{e}}_{c, s m o o t h i n g, N, d} (x_{j}) - e_{c} (x_{j})]}^{2}},

where $p_{j}$ is the population probability at $x_{j}$ and $σ_{c}$ is the population standard deviation of the weighted composite involved in the X-to-composite scaling (Tables 1 -3). The multiplication of Equation (12) by $(100 / σ_{c})$ converts the RMSE values to percentages of the composite score standard deviation, facilitating the comparison of the RMSE values for the three weighted composites that differ with respect to score ranges and standard deviations (Tables 1 -3).

The smoothing and scaling approaches were evaluated in greater detail for the three X-to-composite scaling situations by considering the score-level biases of each approach,

\begin{array}{l} B i a s_{s m o o t h i n g, N} (x_{j}) = (\frac{100}{σ_{c}}) [\frac{1}{500} \sum_{d = 1}^{500} {\overset{⌢}{e}}_{c, s m o o t h i n g, N, d} (x_{j}) - e_{c} (x_{j})] \\ = (\frac{100}{σ_{c}}) [{\bar{e}}_{c, s m o o t h i n g, N} (x_{j}) - e_{c} (x_{j})] . \end{array}

The score-level standard errors of each approach were also considered,

Standard {Error}_{s m o o t h i n g, N} (x_{j}) = (\frac{100}{σ_{c}}) \sqrt{\frac{1}{500} {\sum_{d = 1}^{500} [{\overset{⌢}{e}}_{c, s m o o t h i n g, N, d} (x_{j}) - {\bar{e}}_{c, s m o o t h i n g, N} (x_{j})]}^{2}} .

Results

The smoothing and scaling approaches’RMSE results for the three scaling situations and two sample sizes are summarized in Table 4. The results show that scaling functions produced with XY smoothing were the most accurate (i.e., had the smallest RMSE values) for all three scaling situations and the two sample sizes. The superior accuracy of the XY smoothing was to some extent an artifact of the simulations, in that the XY smoothing was based on the actual population model used to generate the XY sample data. The second most accurate smoothing and scaling approach was usually the XComp smoothing, except for the X-to-composite scaling where the weighted composite was obtained with the $w_{X} = 1.2$ and $w_{Y} = 0.8$ set of weights with sample sizes of 10,000 where the Raw Data and Weighted Averaging with a $w_{Raw}$ value of 0.99 were the second most accurate methods. Scalings based on Weighted Averaging with a $w_{Raw}$ value of 0.99 were more accurate than those based on Raw Data for sample sizes of 1,000, and had accuracies nearly identical to those based on Raw Data for sample sizes of 10,000. Finally, Table 4 shows that scalings based on Weighted Averaging with a $w_{Raw}$ value of 0.01 were the third most accurate for sample sizes of 1,000 and had the worst accuracies of all considered approaches for sample sizes of 10,000.

Table 4.

Root Mean Squared Error (RMSE) Results.

Weighting and scaling situation	Sample size	Raw	XY smoothing	XComp smoothing	Weighted averages (0.01)	Weighted averages (0.99)
$w_{X} = w_{Y} = 1$	1,000	3.177	1.722	1.773	2.147	3.140
	10,000	0.989	0.539	0.705	1.345	0.989
$w_{X} = 1$ and $w_{Y} = 1.5$	1,000	3.628	2.085	2.116	2.350	3.586
	10,000	1.142	0.658	0.777	1.212	1.142
$w_{X} = 1.2$ and $w_{Y} = 0.8$	1,000	2.824	1.403	1.707	2.060	2.782
	10,000	0.878	0.440	1.060	1.514	0.877

More detailed comparisons of the performances of the scaling and smoothing approaches are presented in terms of the methods’ score-level biases (Figures 4 -6) and score-level standard errors (Figures 7 -9). Figures 4 to 6 show that score-level biases do not vary much with respect to sample size, but are relatively large for X-to-composite scalings involving the more complicated weighted composite distributed computed with the $w_{X} = 1.2$ and $w_{Y} = 0.8$ set of weights (Figures 3 and 6). Figures 4 and 5 show that the smoothing and scaling approaches’ biases were relatively small except at the largest and smallest X scores. Figure 6 showed larger differences in approaches’ biases than Figures 4 and 5, where the XComp smoothing and the Weighted Averaging with a $w_{Raw}$ value of 0.01 had relatively large biases for several X scores. The standard errors in Figures 7 to 9 show that the smoothing and scaling results based on Raw Data and Weighted Averaging with a $w_{Raw}$ value of 0.99 were large and nearly identical, whereas those of the XY smoothing, XComp smoothing, and Weighted Averaging with a $w_{Raw}$ value of 0.01 had smaller and very similar values for most X scores. All methods’ standard errors were larger when based on sample sizes of 1,000 than when based on sample sizes of 10,000.

Discussion

Like the distribution of a composite formed as a weighted combination of subscores, the scaling of one of the subscores to the weighted composite has a level of complexity that is largely determined by the weights of the subscores. As described in this study, there are several approaches that could be used to estimate subscore-to-composite scalings. Although somewhat unfamiliar, the XY smoothing approach based on smoothing the subscore distributions and estimating the weighted composite distribution and subscore-to-composite scaling from the smoothed subscore distributions is a direct reflection of the subscore distributions. The XY smoothing approach might also be considered a more plausible data generation model than the other approaches because of its treatment of the weighted composite distribution as a by-product of the subscore data and distributions. Simulations based on treating the smoothed subscore distributions and subscore-to-composite scalings from the XY smoothing approach as population quantities show the unsurprising result that using the XY smoothing approach in simulations results in greater estimation accuracy than other approaches. Other subscore-to-composite scaling approaches that deal with the composite distribution more simply and more directly may be less accurate, though these approaches can be of practical interest due to being more familiar and, possibly, more easily and efficiently implemented.

The subscore-to-composite scaling approach based on smoothing the joint distribution of the subscore and composite is likely to be implemented in practice, as it is the single group scaling method commonly described in equating texts (von Davier et al., 2004, pp. 113-130). This XComp approach produced the second most accurate scaling results for most of the conditions of this study. The XComp approach does especially well when sample sizes are smaller and when the weighted composite is relatively simple. For scaling situations where sample sizes are large and the weighted composite distribution is complex, the XComp approach can be relatively inaccurate because of relatively large estimation bias (Figure 6B). Additional simulations not reported here show that the increased estimation bias of the XComp approach can be partially addressed using indicator functions to model a complex weighted composite distribution (Holland & Thayer, 2000; von Davier et al., 2004), though bias never improves to the level of the XY smoothing approach.

Weighted Averaging was considered in this study as an approach that could potentially improve the estimation accuracy resulting from the Raw Data approach. Because the Weighted Average approach considered in this study deals only with the marginal distributions of the subscore and composite, it is an efficient strategy that avoids bivariate problems such as the identification of score combinations with “structural zeros” (Holland & Thayer, 2000, p. 144) that can be difficult with very complex weighted composite distributions. Although usually less accurate than the XY and XComp approaches, the Weighted Average approach did improve on the estimates obtained from the Raw Data approach, where averages based on 99% smoothed data improved estimation for smaller sample sizes (due to reduced variability, Figures 7 -9), and averages based on 1% smoothed data improved estimation for larger sample sizes (due to smaller biases, Figures 4 -6).

Most of the smoothing and scaling approaches were shown to produce scaling estimates with improved accuracy relative to scaling with raw data. However, the approaches’ accuracy improvements varied based on sample size, the complexity of the weighted composite score distribution, and the complexity of the subscore-to-composite scaling. These issues suggest that the findings of this study may be more illustrative than generalizable with respect to scaling situations not considered in this study. Going beyond the current study, weighted composites could be created using weights selected for maximizing composite reliability (Woodbury & Lord, 1955), maximizing validity (Woodbury & Novick, 1967), or from more than two subtests that exhibit higher or lower intercorrelations and different standard deviations than those considered in this study. Future studies may find that the accuracies of the subscore-to-composite smoothing and scaling approaches might vary for weighted composites created in other ways. The current study demonstrates a simulation approach that can be useful for assessing smoothing and scaling approaches’ performances for other scaling situations involving weighted composites.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Agresti

(1990). Categorical data analysis. New York, NY: Wiley.

Bishop

Y. M. M.

Fienberg

S. E.

Holland

P. W.

(1975). Discrete multivariate analysis. Cambridge: MIT Press.

Fienberg

S. E.

Holland

P. W.

(1973). Simultaneous estimation of multinomial cell probabilities. Journal of the American Statistical Association, 68, 683-691.

Holland

P. W.

Thayer

D. T.

(2000). Univariate and bivariate loglinear models for discrete test score distributions. Journal of Educational and Behavioral Statistics, 25, 133-183.

Kolen

M. J.

Brennan

R. L.

(2004). Test equating, scaling, and linking: Methods and practices (2nd ed.). New York, NY: Springer-Verlag.

Moses

Holland

(2007). Kernel and traditional equipercentile equating with degrees of presmoothing (ETS RR-07-15). Princeton, NJ: Educational Testing Service.

Moses

(2009). Pseudo Bayes estimates for test score distributions and chained equipercentile equating (ETS RR-09-47). Princeton, NJ: Educational Testing Service.

von Davier

A. A.

Holland

P. W.

Thayer

D. T.

(2004). The kernel method of test equating. New York, NY: Springer-Verlag.

Woodbury

M. A.

Lord

F. M.

(1955). The most reliable composite score (ETS Research Bulletin No. RB-55-02). Princeton, NJ: Educational Testing Service.

10.

Woodbury

M. A.

Novick

M. R.

(1967). Maximizing the validity of a test battery as a function of relative test lengths for a fixed total testing time (ETS Research Bulletin No. RB-67-03). Princeton, NJ: Educational Testing Service.