Principal Component Analysis of Smoothed Tetrachoric Correlation Matrices as a Measure of Dimensionality

Abstract

The application of principal component analysis and parallel analysis to smoothed tetrachoric correlation matrices was investigated in a simulation study. To evaluate the effect of several smoothing algorithms, 360 different types of data sets were simulated. Under each simulated condition, two item sets, each fitting a unidimensional two-parameter logistic model, were combined with each other. The simulations differed in the size of the simulated item sets, the size of the person samples, the distribution of the difficulty and discrimination parameters, and the correlation between the person parameters. In general, the application of a smoothing algorithm led to an improved performance in the assessment of dimensionality, but minor differences between the three investigated smoothing algorithms were found. Procedures to apply two of the three investigated smoothing algorithms via R software packages are presented.

Keywords

factor analysis binary variables parallel analysis two-parameter logistic model

Introduction

The assessment of the dimensionality of an item set is a central issue in test theory, and many approaches to dimensionality assessment have been proposed and discussed (e.g., Hattie, 1985; Reckase, 2009; Stout, 1987, 1990). Many writers have already emphasized the impact of a correct dimensionality assessment on practical psychological measurements (Green, 1983; Hattie, 1985) and the development of psychological theories (Weng & Cheng, 2005). Some of the standard methods to assess the dimensionality of an item set are based on exploratory factor analysis (EFA), which also includes principal axis factoring, and the related principal components analysis (PCA). The differences between these methods have been discussed, among others, by Crawford et al. (2010); Fabrigar, Wegener, MacCallum, and Strahan (1999); and Glorfeld (1995). These methods are typically based on the analysis of the matrix of Pearson product–moment correlations. In the case of binary items, this approach has been considered as critical. The factoring of Pearson product–moment correlations (or phi coefficients) may lead to spurious factors. The magnitude of the product–moment correlation of two binary items is limited by their difficulties (Carroll, 1945; Lord & Novick, 1968). Alternatively stated, their bivariate relation is not linear but nonlinear (McDonald & Ahlawat, 1974). Nonlinearity gives rise to extra factors.

As an alternative, the use of tetrachoric correlations instead of phi coefficients has been proposed. In contrast to phi coefficients, tetrachoric correlations are invariant to the item difficulties as long as the assumption of bivariate normality holds (Carroll, 1945; Lord & Novick, 1968).

To determine the correct number of components underlying a data set, a common suggestion is the application of parallel analysis (PA), a method originally proposed by Horn (1965). Several studies found evidence that PA is an accurate method to determine the number of underlying components (e.g., Humphreys & Montanelli, 1975; Zwick & Velicer, 1986). Since the original presentation of PA, several modifications to the original method have been proposed, like using the 95th percentile of the eigenvalue distribution from the simulated data as criterion for assessing the number of underlying components (e.g., Glorfeld, 1995). Recently, Crawford et al. (2010) compared multiple approaches for PA. Their study suggested that no single approach is clearly superior to the others, but that the results depended on the underlying factor structure. A recent study by Green, Levy, Thompson, Lu, and Lo (2012) also suggested a revised version of PA, which takes into account of the existence of prior factors in the determination of the critical eigenvalues of subsequent factors. Their results suggest that the suggested method leads to improved results in the presence of certain factor structures when compared with traditional methods.

Recent studies have already investigated the performance of the PA in retrieving unidimensionality in binary data (Tran & Formann, 2009; Weng & Cheng, 2005). One of the main problems in the application of PA and PCA of tetrachoric correlations to binary data identified by these studies is the presence of indefinite correlation matrices, which makes the application of PCA impossible.

The purpose of the present study was to investigate the performance of PCA and PA of smoothed tetrachoric correlations as an assessment of the dimensionality of sets of binary items in uni- and multidimensional item sets. We show that the investigated procedures perform well as assessments of dimensionality.

Two of the three investigated smoothing methods can be carried out with freely available software packages written in R (R Development Core Team, 2011). We present code that allows performing necessary computations.

This article is organized as follows. In the next section, we discuss several smoothing algorithms for indefinite correlation matrices described in the literature. Then previous studies on the application of PA to binary data are summarized. Following this the methods and results of a simulation study on the application of the improved PCA to matrices of tetrachoric correlations are presented. Finally, results are summarized and discussed.

Several Smoothing Procedures for Indefinite Matrices

One objection to the application of principal component analysis and factor analytic methods to tetrachoric correlation matrices is that these matrices might be indefinite (Lord, 1980). To apply factor analytical methods in these cases, several smoothing procedures have been suggested, of which some selected algorithms will be described below. For a general overview of earlier methods, see Devlin, Gnanadesikan, and Kettenring (1975); for an overview of current methods, see Yuan, Wu, and Bentler (2010).

The algorithm of Higham (2002) searches for the symmetric positive semidefinite matrix X with unit diagonal that is nearest to a given matrix A. The distance between two matrices is defined by the Frobenius norm, which is defined by $‖ A - X ‖_{F} = \sum_{i, j} (a - x)_{i, j}^{2} .$ In his original work, Higham (2002) also investigated a more general approach that allowed the specific weighting of rows, columns, or even specific entries of $(A - X)$ in order to control changes to the correlation matrix during the smoothing process. The smoothing procedure of Higham (2002) has been implemented in the R package Matrix (Bates & Maechler, 2011).

Knol and Berger (1991) used the following smoothing procedure in their simulation studies: Let R be a possibly indefinite matrix of correlations, and let $R = VD V^{- 1}$ be its eigen decomposition. A positive-definite correlation matrix $R^{+} (δ)$ can be obtained by

R^{+} (δ) = [Diag (V D^{+} V^{- 1})]^{- 0.5} V D^{+} V^{- 1} [Diag (V D^{+} V^{- 1})]^{- 0.5} .

(1)

In this formula, $D^{+}$ denotes a modified diagonal matrix of eigenvalues obtained from D by replacing each eigenvalue below a predefined nonnegative threshold δ with δ. $Diag (V D^{+} V^{- 1})$ denotes a diagonal matrix that contains the diagonal elements of the matrix $V D^{+} V^{- 1} .$ Although this algorithm was shown to provide acceptable results in the simulation studies reported by Knol and Berger (1991), to our knowledge it is not available in a publicly available software package.

Recently, Bentler and Yuan (2011) described another approach for obtaining a positive definite correlation matrix from an indefinite one. Given a symmetric indefinite matrix R, Bentler and Yuan (2011) show that a positive definite matrix R* can be obtained by

R^{*} = Δ R_{0} Δ + D_{R} .

(2)

In Equation (2), R₀ results from R by setting its diagonal entries to 0, D_R is the diagonal matrix containing the diagonal elements of R. Δ is a positive definite matrix that meets the condition that $D_{R} - Δ^{2} (R - D)$ is positive definite, where D is a diagonal matrix with nonzero elements such that $(R - D)$ is positive semidefinite. Like the smoothing approach of Knol and Berger (1991), this method has not been explicitly implemented in a widely available software package yet. However, a variation of this method uses minimum trace factor analysis to determine R*, as is demonstrated by Bentler and Yuan (2011). Functions contained in the R packages psych (Revelle, 2012) and Rcsdp (Bravo, 2010) may be used to carry out this approach in R.

The algorithm of Higham (2002) has been fully implemented in a software package written in R and can be easily applied to tetrachoric correlation matrices, which may also be calculated using R software (e.g., with the package psych; Revelle, 2012). In the next section, we investigate the results of several studies on the application of PCA on tetrachoric correlation matrices without applying a smoothing algorithm.

The Use of Parallel Analysis With Binary Variables

Determining the number of factors underlying a data set is important in the application of factor analytical methods, and a number of studies have focused on this issue. One of the most commonly used methods lies in the application of the Kaiser–Guttman rule, according to which all factors with eigenvalue greater than 1 (Guttman, 1954; Kaiser, 1960) are to be retained. This approach was evaluated by Bernstein and Teng (1989) with PCA of phi coefficients. They concluded that the Kaiser–Guttman rule leads to the overextraction of factors when applied to dichotomized variables.

Horn (1965) criticized this rule because it does not take sampling errors into account. He proposed to calculate a correlation matrix of random data of the same sample size and variable set size to determine the critical eigenvalues. Horn’s approach became known as parallel analysis and was recently advocated by several writers for its use with PCA and EFA (Reckase, 2009; Weng & Cheng, 2005).

Recent studies also discussed the application of PA in detecting the number of factors in data sets of binary variables. Turner (1998) analyzed the application of PA to PCA and EFA and concluded that the common approach to determine the critical eigenvalues based on the size of the item set and the sample size may lead to the underextraction of factors or components because the presence of real factors in the data may influence the size of critical eigenvalues of random factors.

Green (1983) investigated the application of PA to EFA of phi coefficients of uni- and multidimensional data in which guessing was present. He found that PA of phi coefficients led to the extraction of spurious factors but argued that these factors could be identified in well-designed tests.

Weng and Cheng (2005) found that the application of PA performed well if the method was applied using the 95th or 99th percentile eigenvalues as the criterion for comparison, an approach that was also suggested by Glorfeld (1995). In their simulations, PA performed better with increasing sample size. Weng and Cheng (2005) found no difference in the performance of PA, using phi coefficients and tetrachoric correlations.

Tran and Formann (2009) further studied the performance of PA in PCA based on tetrachoric correlations. Using simulation studies, they showed that the latent structure of an item set that conforms to a unidimensional normal ogive model is not reliably uncovered by PA and PCA based on Pearson correlations, but the performance of this approach depends on the item discrimination and difficulty parameters. They also found that PA based on tetrachoric correlations performs better than PA based on Pearson correlations when applied to PCA of tetrachoric correlations. Nevertheless, they concluded that the usefulness of PCA is diminished in the presence of binary data. One reason for their verdict was the problem of indefinite correlation matrices, which makes the application of PCA impossible. The frequent presence of indefinite tetrachoric correlations matrices was also reported in other studies (e.g., Knol & Berger, 1991; Weng & Cheng, 2005). However, to our knowledge, a systematic investigation on prevalence rates under various conditions was never carried out.

More recently, Timmerman and Lorenzo-Seva (2011) investigated the application of PA to polytomous items. In their study, they found that a PA based on polychoric correlations performed best in determining the dimensionality in PCA. However, as in previous studies, nonconvergence of PCA due to indefinite matrices again posed a serious problem, with the total convergence rate reaching only 37.01% across their 10,400 simulated data matrices.

Method

We carried out a simulation study to compare results of PCA and PA with and without applying a smoothing algorithm and to investigate the prevalence of indefinite tetrachoric correlation matrices under different conditions in a more systematic way than previous research. In each simulation, the responses of a person sample to an item set consisting of two separate scales each fitting the two-parameter logistic model (Birnbaum, 1968) were simulated. The simulations varied in the following aspects: (a) the distributions of the item difficulty and item discrimination parameters in the simulated item sets (10 distribution combinations), (b) the size of the person sample (3 sizes), (c) the size of the item set (3 sizes), (d) the correlation between the person parameters in the simulated multidimensional data sets (4 correlations), and (e) the applied smoothing algorithm. Under each condition of the 10 × 3 × 3 × 4 × 3 design, 1,000 data sets were simulated. Similar study designs were used by Tran and Formann (2009) and van Abswoude, van der Ark, and Sijtsma (2004).

Distributions of the Item Difficulty and Item Discrimination Parameters

Four types of scales were defined that differed in the standard deviation of the item difficulty parameters and in the mean of the distribution of the item discrimination parameters. The item difficulty parameters of two scale types (named A and B) followed a uniform distribution in the interval $[\frac{- \sqrt{3}}{2}; \frac{\sqrt{3}}{2}]$ (resulting in a standard deviation of 0.5), whereas the item difficulty parameters in the other two scale types (named C and D) followed an uniform distribution in the interval $[\frac{- 3 \sqrt{3}}{2}; \frac{3 \sqrt{3}}{2}]$ (resulting in a standard deviation of 1.5). The distribution of the item discrimination parameters was log normal in all scale types. In scale types A and C, the item discrimination parameters were distributed LN(ln(0.5), 1), whereas in scale types B and D, the item discrimination parameters were distributed LN(ln(1.5), 1). In all scales types, the person parameter followed a standard normal distribution.

Size of the Item Sets

The simulated item sets consisted of 10, 30, and 50 items, respectively.

Size of the Person Samples

Three person sample sizes were used, containing 250, 500, and 750 simulated respondents, respectively.

Correlations Between the Person Parameters

The correlations between the person parameters was set 0, 0.4, 0.8, and 1.0, with the first three values resulting effectively in two-dimensional item sets, while the last correlation value resulted in unidimensional item sets. Given the person parameter $θ_{1}$ in the first scale, the person parameters $θ_{2}$ in the second scale were calculated using the formula

θ_{2} = Rand * \sqrt[2]{1 - r^{2}} + θ_{1} * r .

(3)

In this formula, r is the correlation between the person parameters, and Rand is a random variable that is normally distributed with standard deviation 1.

After the parameters of every simulated item and every simulated person were set, the probability of a positive reaction and a random number between 0 and 1 were calculated for each person–item pair. If the random number was smaller than the calculated probability of a positive reaction (“1”), the reaction was set as positive for the person–item pair; otherwise, it was set as negative (“0”).

In our study, tetrachoric correlation coefficients and PCA were calculated and carried out using functions of the R package psych (Revelle, 2012). The critical eigenvalues for the PCA were determined using a PA based on tetrachoric correlations, which were also carried out by using functions of the psych package. Horn (1965) advocated in his original approach comparing whether observed eigenvalues were greater than expected eigenvalues, that is, the mean of the respective eigenvalue distribution. We used the median (= 50th percentile) of the respective eigenvalue distribution in our study, calculated from 1,000 random data matrices. Means and medians differed by less than 0.01 across all eigenvalue distributions in our study, which is in line with Glorfeld (1995).

If an indefinite correlation matrix was obtained in the simulations, a smoothing algorithm was applied. In these cases, PCA was applied to the smoothed tetrachoric correlation matrices. All simulations were replicated using each of the smoothing algorithms described in the section “Several Smoothing Procedures for Indefinite Matrices.” The smoothing algorithm of Higham (2002) was applied with the R package Matrix (Bates & Maechler, 2011). The smoothing algorithm of Bentler and Yuan (2011) was applied in R using functions of the packages psych (Revelle, 2012) and Rcsdp (Bravo, 2010). See the appendix for R code to use these two algorithms. The smoothing algorithm of Knol and Berger (1991) was applied using software which was developed specifically for this study in Java. In this software, the constant δ was set to 0.

Results of the Simulation Study

Our simulations confirmed the results of previous studies, which indicated the frequent presence of indefinite tetrachoric correlation matrices in the application of PCA to binary variables (Timmerman & Lorenzo-Seva, 2011; Tran & Formann, 2009; Weng & Cheng, 2005). Table 1 displays the relative frequencies of indefinite correlation matrices in all simulated data sets in which the correlation between the person parameters was 0.

Table 1.

Rate (in Percent) of Indefinite Correlation Matrices in Simulated Data Sets With a Correlation of 0.0 Between the Person Parameters for Given Sizes of the Samples (n) and the Item Sets (i) for Combination of the Scale Types A, B, C, and D

n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
250	10	0	1.9	0.4	6.5	8	3.3	17.5	0.6	8.8	25.2
	30	43.4	98.1	65.2	99.6	100	98.4	100	76.2	99.8	100
	50	100	100	100	100	100	100	100	100	100	100
500	10	0	0.1	0	1.2	0.9	0.1	3.3	0	1.8	8.1
	30	3.5	61.5	10.7	87.5	92.9	69.7	98.2	20.2	91.1	99.8
	50	47.1	100	72.7	100	100	99.9	100	87.8	100	100
750	10	0	0	0	0.7	0.4	0.1	1.7	0	0.7	4.3
	30	0.4	31.7	3.3	68.3	71.4	39	89.6	5.7	73.9	96.7
	50	13.7	93.4	34.5	99	99.8	97	99.9	51.8	99.8	100

To assess the practical usefulness of PCA of smoothed tetrachoric correlations, we report the principal results of our simulation study in several tables. Table 2 shows the results without application of a smoothing algorithm to the data. In unidimensional data sets, similar results were obtained. Table 3 shows the results of the analysis of multidimensional data sets when the correlation matrices were smoothed according to the algorithm of Higham (2002) before applying PCA. The smoothing algorithm of Knol and Berger (1991) led to results mostly comparable to those of the Higham algorithm; therefore, they will not be reported in detail here. The results of the smoothing algorithm by Bentler and Yuan (2011) are reported in Table 4.

Table 2.

Rate (in Percent) of Correctly Detected Dimensionality for Each Combination of the Scale Types A, B, C, and D, With Given Size of the Person Sample (n), Item Set (i), and Correlation Between the Person Parameters (R) When Smoothing Was Not Applied

R	n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
0.0	250	10	40.9	64.1	37.6	55.9	84.4	58.3	72.2	35.3	51.9	61.2
		30	23.6	1.7	11.6	0.4	0	1.2	0	5.9	0.1	0
		50	0	0	0	0	0	0	0	0	0	0
	500	10	47.8	75.9	45.1	70.4	95	73.9	89.5	43.7	66.8	84.6
		30	61.8	38.7	49.4	11.7	7.7	30.4	2.2	36.8	8.5	0.3
		50	38.8	0.2	15.8	0	0	0.1	0	6	0	0
	750	10	52.8	80.6	50.4	76.8	96.9	77.9	94.2	50.8	74.7	90.1
		30	73.8	68	63.3	30.6	29.5	60.7	11.6	53.4	24.9	4.1
		50	73.6	6.9	47.9	0.7	0.2	3.8	0	29.8	0.4	0
0.4	250	10	38.2	58.2	37.2	51.9	81.7	56.7	71	36.5	47.5	62
		30	25.5	2	13.8	0.3	0	0.9	0	7.3	0.1	0
		50	0	0	0	0	0	0	0	0	0	0
	500	10	46.4	70.2	44.5	67.6	95.3	69.3	89.6	43.1	65.9	85.6
		30	63.4	37.8	49	11.9	8	27.5	1.9	37.5	8.9	0.7
		50	37.7	0.4	17.5	0	0	0.2	0	6.3	0	0
	750	10	52.8	77	52.4	74.2	97.6	74.6	94.3	50.3	72.9	91.3
		30	72.9	70.7	63.6	31	30.7	60.6	11.7	53.2	25.4	3.1
		50	73.3	7	48.6	0.5	0.1	4.4	0	29.7	0.4	0
0.8	250	10	25.9	16.2	26.8	18.7	6.2	18.6	8.7	26.9	21	10.9
		30	14.2	0.4	10.3	0	0	0.2	0	5.8	0	0
		50	0	0	0	0	0	0	0	0	0	0
	500	10	22.4	11.4	24.1	14	4.1	12.5	5.2	24.5	15.3	7.8
		30	25.8	11.9	26	4.9	7.6	8.8	1.8	24.9	3.7	0.3
		50	22.6	0.4	9.5	0	0	0.1	0	5.3	0	0
	750	10	20.5	8.8	21.4	10.3	2.3	10.3	3.9	24.5	13	5.4
		30	30.7	30.9	30.5	14	33.4	27.7	10.6	30.9	12.4	3.5
		50	54.9	8	39.4	0.4	0.3	3.6	0	24.8	0.6	0

Table 3.

R	n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
0.0	250	10	41.4	66.4	38.3	59.6	92.2	62	87.4	34.9	57.4	83.1
		30	49.8	89.8	38.5	66.7	99.4	86.4	89.7	30.9	58.1	75.5
		50	63.2	96.6	49.9	60.5	99.9	94	84.1	34.9	48.5	54.7
	500	10	49.4	75.4	48	72.2	95.5	75.4	93.8	43	70.4	92.9
		30	65.5	95.7	57	81.2	100	92.9	96.3	48.5	77.3	88.6
		50	80.1	99.1	67.3	70.1	100	98.2	87.8	53.8	68.7	69.5
	750	10	53.9	81.5	49.1	78.9	98	78.6	97.8	51.4	76.5	96.5
		30	76.5	98.4	68	90.5	100	96.5	98.3	57.5	85.9	95.4
		50	85.1	99.6	76.8	79.7	100	98.8	92.8	66.1	74.2	76.2
0.4	250	10	38.7	58.9	37.3	54.5	90.1	57.5	85.8	36.8	54.7	83.2
		30	49.6	88.9	43.8	64.6	99.7	85.8	91.5	32.9	58.9	77.3
		50	64.7	97.7	51.6	60.9	100	93.8	85	39.7	52.2	58.7
	500	10	47.3	69.6	44.8	69.1	96.1	70.6	94	44.9	67.1	93.3
		30	65.9	95.1	57.8	83.1	100	93.9	97.4	51.2	76.6	89.1
		50	76	99.5	67.2	73.3	100	97.7	90.4	53.9	68.8	72.6
	750	10	54.5	79.1	52.8	74.2	98	75.3	97.3	50.6	74.1	96.8
		30	73.2	98	67.5	89.7	100	96.8	98.7	58.2	86.5	95.8
		50	86	99.6	77.9	79.6	100	99.7	91	67.7	75.5	75.6
0.8	250	10	24.9	16.4	27.4	20.8	7.4	18.2	10.6	24.6	23.2	16.2
		30	24.2	17.8	29.5	28.2	57.3	21.9	57.4	24.4	31.6	59.6
		50	33.2	40.9	34	50.3	95.5	44.8	85	33.4	48.6	54
	500	10	22.5	11.2	24.4	14.8	3.5	14.7	5.1	23.2	14.6	10
		30	25.5	34.4	31	40.7	91.8	36.6	88	32.8	39.8	82.4
		50	50.5	78.4	49.8	70.5	100	79.8	93	47.2	68.6	61.6
	750	10	19	7.7	20.4	10.6	2.5	9.4	3.3	24.4	11	4.6
		30	29.6	42.7	31.2	48.5	98.4	50.7	96.7	31.8	52.4	91.6
		50	65.1	94.2	65.6	80.8	100	91.5	92.5	62.4	77.4	65

Table 4.

R	n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
0.0	250	10	40.8	63.7	35.5	59.8	92.1	60.2	86.5	36.2	58.4	83.3
		30	52.3	91.9	46	80.5	99.7	88.1	97.6	40	77.9	91.6
		50	85.5	99	83.7	95.8	100	99.1	99.4	81.1	94.1	97.2
	500	10	46.3	75.8	43.2	71	96.4	73	93.2	45.5	68.4	93.2
		30	61.9	95.3	56.1	87.3	99.9	94.5	98.2	48.3	83.8	96.4
		50	77.3	99.4	73.2	92	100	98.7	98.9	66.5	88.2	93.8
	750	10	51.9	78.2	52.9	78	97.7	78.6	97.2	51	76.3	96.3
		30	72.5	97.1	66.2	91.4	100	96.6	99.3	60.1	91.5	96.9
		50	85.6	99.7	76.6	91.4	100	99.2	98	69.4	89.9	92.9
0.4	250	10	36.5	59.5	37.1	56.8	90.6	59.2	88	38.8	51.9	82.2
		30	53.2	92	47.3	80.5	99.4	88.6	97.4	42.6	77.5	92.6
		50	88.6	99.5	85.2	96.1	100	99.2	99.4	84.6	94.7	98
	500	10	45.5	68.7	46.3	69.5	97.1	68.1	94.3	43.2	68	93.6
		30	66	95.8	57.3	87.4	99.9	94.1	99.1	49.1	84.6	96.8
		50	80.6	99.7	74.8	89.5	100	98.6	98.7	67	89.1	96.7
	750	10	51.9	76.1	51.7	77.4	97.7	75.2	96.1	49	75.6	97.2
		30	72.9	97.9	66.7	91.9	100	96.6	99.2	59.4	89.9	97.6
		50	86	99.7	80.6	92	100	99.3	98.3	71.5	90.4	95.2
0.8	250	10	26.6	16.9	27.8	18.5	6.9	17.9	10	27	21.8	12.4
		30	23.1	13.5	25.7	16.6	44.7	13.7	37.2	26.9	19.9	36.5
		50	10.3	16.4	11	12.6	80.6	15.2	63.3	12.2	11.3	47.9
	500	10	21	11.1	23.5	13.1	4.1	11.2	4.7	24.6	15.7	7.9
		30	27.6	29.8	28.9	33.5	88.6	31	82.5	33.9	37.4	79.5
		50	46.1	72.4	42	69.7	100	68.1	98.9	42	64.9	93.3
	750	10	20.1	8.9	22.2	10.1	2.6	10.2	3.8	25.2	13.8	6.1
		30	31.3	44	30.8	42.5	98.4	45.8	96.7	34.8	51	94.1
		50	66	91.4	66.1	87.2	100	91	99	59.8	83	91.2

As can be seen from Tables 3 and 4, the algorithm of Bentler and Yuan (2011) led to more correct solutions compared with the algorithm of Higham (2002) when the correlation between the person parameters was low, but to fewer correct solutions when it was high. As was to be expected, the number of correct results in general increased with person sample size and decreased with the number items. In general, the number of correct results did not decrease when the correlation between the person parameters of the two item sets increased.

The results of the simulations without application of smoothing in unidimensional data sets are not reported here in detail, since the results were comparable to those observed in bidimensional data sets. For the unidimensional data sets, the results of the application of the smoothing algorithm by Higham (2002) are presented in Table 5. Table 6 shows the results of the application of the smoothing algorithm of Bentler and Yuan (2011).

Table 5.

Rate (in Percent) of Correctly Detected Dimensionality for Each Combination of the Scale Types A, B, C, and D, With Given Size of the Person Sample (n) and Item Set (i) for Each Data Set Combination When the Smoothing Algorithm of Higham Was Applied

n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
250	10	65.7	88.2	62.1	80.5	97.4	84.7	95.9	58.4	78	91.7
	30	79.7	97.2	71	84	100	96.7	94.2	61.9	74.8	66.6
	50	87.6	99.8	82.3	73.9	100	98.3	84	68	58.8	29.5
500	10	75.9	93.1	70.4	90.5	99.5	92	99.1	67.8	89.5	96.8
	30	87.3	99.3	81.9	93.5	100	98.7	98.1	78.3	86.9	74.5
	50	95.1	99.9	90.2	83.5	100	99.6	88.5	79.4	69.6	33.2
750	10	77.6	95.3	76.5	94.5	99.6	94.3	99.8	73	92.1	98.4
	30	92.9	99.5	88.5	95.7	100	99.1	98.9	85	93	81.7
	50	97.6	99.9	93.6	78.8	100	99.8	91.4	91	72.8	38.8

Table 6.

n	i	A-A	A-B	A-C	A-D	B-B	B-C	B-D	C-C	C-D	D-D
250	10	63.1	87.6	59.3	80	97.8	85.2	96.3	57.6	77.1	91.6
	30	80.1	98.9	77.7	96.1	100	97.7	98.7	76	92.6	88.3
	50	98.4	100	97.4	99.8	100	100	99.9	97	98.6	90
500	10	74.4	93.6	71.2	90.3	99.4	93.1	99.3	70	86.4	97.8
	30	87.4	99.1	83.7	95.2	100	99.1	99.3	78	93.5	90.4
	50	94.8	100	93.2	97.9	100	99.8	99	89.5	94.3	81
750	10	76.5	94.7	75	93.6	99.6	94.1	99.7	74.5	92.5	98.7
	30	92.3	99.9	90.7	98.5	100	99.8	99.3	87.2	96.5	91.5
	50	97.1	99.9	94.4	97.8	100	99.8	99	91	93.6	75.1

In unidimensional item sets, the number of correct results increased with person sample size but decreased with the number of items. Both in bidimensional and in unidimensional item sets, the size of the change depended on the distributions of the item parameters. Similar results have been previously reported by Tran and Formann (2009) and Weng and Cheng (2005). The smoothing algorithm by Bentler and Yuan (2011) led to improved results when compared with the smoothing algorithm of Higham (2002), especially in the analysis of large item sets.

Across all 90 simulation conditions of unidimensional data sets, the algorithm of Bentler and Yuan led 54 times to better results than the other two algorithms. In two-dimensional data sets, this number was 52 out of 90. However, when the correlation was 0.8, the Bentler and Yuan algorithm achieved better results only in 18 out of 90 conditions. Under these conditions, the algorithm of Knol and Berger performed best.

We also evaluated how often each smoothing algorithm achieved an accuracy percentage of 95% or higher for all simulated conditions. In this evaluation, the Bentler–Yuan algorithm generally achieved again the best results. In unidimensional data sets, an accuracy percentage of more than 95% was achieved for 44 out of 90 conditions. The Higham and the Knol–Berger algorithms achieved this accuracy only in 32 and 21 conditions, respectively. In bidimensional data sets, high accuracy was achieved less often when the correlation between the person parameters in both scales increased. When this correlation was 0, the Bentler–Yuan algorithm achieved an accuracy percentage of more than 95% in 29 simulated conditions, compared with 13 for the Knol–Berger algorithm and 21 for the Higham algorithm. When the correlation between the person parameters was high, all smoothing algorithms achieved accuracy percentages of more than 95% only in four to six conditions.

Discussion

The results obtained in the simulation study are in line with the results of past studies on the PA of tetrachoric correlations (Tran & Formann, 2009; Weng & Cheng, 2005). The presence of indefinite correlation matrices often precludes the application of PCA and PA and underlines the need for a smoothing algorithm. In our simulation study, we observed that indefinite matrices of tetrachoric correlation matrices tended to occur if the analyzed item set is large, the analyzed person sample is small, and the discrimination parameters of the item sets are large. We emphasize that the calculation of the tetrachoric correlation coefficients is not advisable if the assumption of a bivariate normal distribution of the latent variables is likely to be violated in the data (e.g., Lord, 1980). This may be the case if guessing is present. It should be noted that this assumption was not violated in our simulations.

Application of smoothing algorithms generally improved correct identification of dimensionality when the correlation between the latent dimensions was 0.0 or 0.4 in our simulations. When the correlation between the person parameters of the two scales was 0.8, the results were less reliable and more dependent on the size of the person and item sample. In general, the results improved when the size of the analyzed multidimensional item set and person sample increased and when the discrimination parameter increased. We also observed minor differences in the performance of the three smoothing algorithms used in our study. In data sets with a clear dimensional structure, that is, in unidimensional data sets and multidimensional data sets with a low correlation between the underlying dimensions, the algorithm of Bentler and Yuan (2011) performed best, especially when large item sets were analyzed.

In summary, our results seem to indicate that the application of PCA and PA to binary data seems to assess the dimensionality of a multidimensional item set correctly if the correlations between the dimensions are low to medium and when a smoothing procedure is applied.

When applied to unidimensional item sets, PCA and PA of smoothed correlation matrices led to correct results in most cases that were investigated. However, this was not the case with unidimensional data sets in which a large item pool with high discrimination parameters and a wide range of difficulty parameters was combined with a small person sample. Our interpretation of this observation is that in these cases each simulated data set contained at least some items with a very high or very low difficulty parameter that were positively answered by almost all (or very few) persons in the person sample. The tetrachoric correlation coefficients between items with extreme difficulty parameters were generally very low, which led to the extraction of multiple components in PCA.

Although our study was based on the original approach of Horn (1965), future studies should investigate the performance of smoothing algorithms when used in more recent variations of PA (i.e., using the 95th percentile of the eigenvalue distribution as criterion). The evidence collected so far in this and similar studies (Crawford et al., 2010; Green et al., 2012) seems to suggest that no single algorithm leads to optimal results, but the results of each algorithm depend on the underlying factor structure.

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Bates

Maechler

(2011). Matrix: Sparse and dense matrix classes and methods (R Package Version 1.0-6). Retrieved from http://CRAN.R-project.org/package=Matrix

Bentler

Yuan

K.-H.

(2011). Positive definiteness via off-diagonal scaling of a symmetric indefinite matrix. Psychometrika, 76, 119-123.

Bernstein

I. H.

Teng

(1989). Factoring items and factoring scales are different: Spurious evidence for multidimensionality due to item categorization. Psychological Bulletin, 105, 467-477.

Birnbaum

(1968). Some latent trait models. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 395-479). Reading, MA: Addison & Wesley.

Bonett

D. G.

Price

R. M.

(2005). Inferential methods for the tetrachoric correlation coefficient. Journal of Educational and Behavioral Statistics, 30, 213-225.

Bravo

H. C.

(2010). Rcsdp: R interface to the CSDP semidefinite programming library (R Package Version 0.1-41). Retrieved from http://CRAN.R-project.org/package=Rcsdp

Carroll

J. B.

(1945). The effect of difficulty and chance success on correlations between items or between tests. Psychometrika, 10, 1-19.

Crawford

Green

S. B.

Levy

W.-J.

Scott

Svetina

D. S.

Thompson

M. S.

(2010). Evaluation of parallel analysis methods for determining the number of factors. Educational and Psychological Measurement, 70, 885-901.

Devlin

S. J.

Gnanadesikan

Kettenring

J. R.

(1975). Robust estimation and outlier detection with correlation coefficients. Biometrika, 62, 531-545.

10.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.

11.

Glorfeld

L. W.

(1995). An improvement on Horn’s parallel analysis methodology for selecting the correct number of factors to retain. Educational and Psychological Measurement, 55, 377-393.

12.

Green

S. B.

(1983). Identifiability of spurious factors using linear factor analysis with binary variables. Applied Psychological Measurement, 7, 139-147.

13.

Green

S. B.

Levy

Thompson

M. S.

W.-J.

(2012). A proposed solution to the problem with using completely random data to assess the number of factors with parallel analysis. Educational and Psychological Measurement, 72, 357-374.

14.

Guttman

(1954). Some necessary conditions for common factor analysis. Psychometrika, 19, 149-161.

15.

Hattie

(1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9, 139-164.

16.

Higham

(2002). Computing the nearest correlation matrix: A problem from finance. IMA Journal of Numerical Analysis, 22, 329-343.

17.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.

18.

Humphreys

L. G.

Montanelli

R. G.

(1975). An investigation of the parallel analysis criterion for determining the number of common factors. Multivariate Behavioral Research, 10, 193-206.

19.

Kaiser

H. F.

(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141-151.

20.

Knol

Berger

(1991). Empirical comparison between factor analysis and multidimensional item response models. Multivariate Behavioral Research, 26, 457-477.

21.

Lord

F. M.

(1980). Application of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.

22.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison & Wesley.

23.

McDonald

R. P.

Ahlawat

K. S.

(1974). Difficulty factors in binary data. British Journal of Mathematical and Statistical Psychology, 27, 82-99.

24.

R Development Core Team. (2011). R: A language and environment for statistical computing (ISBN 3-900051-07-0). Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/

25.

Reckase

M. D.

(2009). Multidimensional item response theory. Dordrecht, Netherlands: Springer.

26.

Revelle

(2012). psych: Procedures for personality and psychological research (R Package Version 1.2.4). Retrieved from http://CRAN.R-project.org/package=psych

27.

Stout

(1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.

28.

Stout

(1990). A new item response theory modeling approach with applications to unidimensional assessment and ability estimation. Psychometrika, 55, 293-326.

29.

Timmerman

M. E.

Lorenzo-Seva

(2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16, 209-220.

30.

Tran

U. S.

Formann

A. K.

(2009). Performance of parallel analysis in retrieving unidimensionality in the presence of binary data. Educational and Psychological Measurement, 60, 50-61.

31.

Turner

N. E.

(1998). The effect of common variance and structure pattern on random data eigenvalues: Implications for the accuracy of parallel analysis. Educational and Psychological Measurement, 58, 541-568.

32.

van Abswoude

A. A. H.

van der Ark

L. A.

Sijtsma

(2004). A comparative study of test dimensionality assessment procedures under nonparametric IRT models. Applied Psychological Measurement, 28, 3-24.

33.

Weng

L.-J.

Cheng

C.-P.

(2005). Parallel analysis with unidimensional binary data. Educational and Psychological Measurement, 65, 697-716.

34.

Yuan

K.-H.

Bentler

P. M.

(2010). Ridge structural equation modeling with correlation matrices for ordinal and continuous data. British Journal of Mathematical and Statistical Psychology, 64, 107-133.

35.

Zwick

W. R.

Velicer

W. F.

(1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99, 432-442.