Procedures for Analyzing Multidimensional Mixture Data

Abstract

The multidimensional mixture data structure exists in many test (or inventory) conditions. Heterogeneity also relatively exists in populations. Still, some researchers are interested in deciding to which subpopulation a participant belongs according to the participant’s factor pattern. Thus, in this study, we proposed three analysis procedures based on the factor mixture model to analyze data in the multidimensional mixture context. Simulations were manipulated with different levels of factor numbers, factor correlations, numbers of latent classes, and class separation. Issues with regard to model selection were discussed at first. The results showed that in the two-class situations the procedures of “factor structure first then class number” (Procedure 1) and “factor structure and class number considered simultaneously” (Procedure 3) performed better than the “class number first then factor structure” (Procedure 2) and yielded precise parameter estimation and classification accuracy. It would be appropriate to choose Procedures 1 and 3 when strong measurement invariance is assumed while using an information criterion, but Procedure 1 saved more time than Procedure 3. In the three-class situations, the performance of all three procedures was limited. Implementations and suggestions have been addressed in this research.

Keywords

factor mixture model factor structure latent class multidimensional mixture data

Introduction

Measurement data are composed of tests (or inventories) and participants. Multi-factor structures have long existed in many classical test conditions such as the Primary Mental Abilities Test (Thurstone, 1947) and the NEO Personality Inventory (Costa & McCrae, 1978). When a psychological construct is measured by more than one factor, the construct is multidimensional. On the contrary, populations may consist of unknown subpopulations, and the research goal can be to decide which of the subpopulations a participant most likely belongs. Mixture in statistics means that a population contains two or more subpopulations. It is this context of the multidimensional mixture that this study examines.

Multidimensional mixtures are a common type of data that are often seen in research. A factor structure can be decided in advance, after which different classes can be investigated with empirical data (Bacci et al., 2014; Lin & Wilson, 2013; Litson et al., 2017; Lubke & Muthén, 2005). In some studies, the factor structure was discovered, and different factor structures existed between classes in real data (De Boeck et al., 2011).

In the model-based clustering approach, the factor mixture model (FMM; Arminger et al., 1999; Dolan & van der Maas, 1998; B. Muthén & Shedden, 1999; Yung, 1997), a synthesis of the common factor model and the latent class model, seems to satisfactorily deal with multidimensional mixture data, which comprise categorical and continuous latent variables.

Previous studies (Lubke & Muthén, 2007) often assumed that the items that corresponded to the factor were known, and investigated the performance of the FMM in different situations. Lubke and Muthén (2005) illustrated the analysis step via empirical data. However, a factor structure that is set in advance can be doubtful and needs to be proven as existing in different populations. Heterogeneity in populations can cause inaccurate structural equation model estimation results (B. Muthén, 1989). Sometimes, researchers can be ambitious about building a new theory construct and distinguishing possible subpopulations at the same time. Thus, the assumption of the factor structure cannot always be made in advance. In the research field, researchers have to face scenarios with an unknown factor structure and unknown populations simultaneously.

Lubke and Neale (2006) investigated the issue of distinguishing the latent variable between continuous factors and categorical classes. In their research, most of the situations fixed on zero factor with two classes, three classes, and four classes, or on one class with one factor, two factors, three factors, and so on; that is, the number of classes and factors were kept low in turn. Lubke and Neale also suggested extending higher dimensions and class situations.

Once the numbers of factors and classes increase simultaneously, in the combination of an unknown factor structure and unknown classes, how to choose the factor structure (including the number of factors and the correspondence between the items and the measured factors) in unknown class situations systematically becomes an issue.

Starting from either the “multi-factor” side or the “latent class” side, when fitting the FMM embedded in a confirmatory framework perspective, which side is better? If the factor structure is explored first while making a single-class assumption, the unknown classes and the different factor structures between them can influence the model estimation result. Still, there is probably more than one class. Making a class assumption via manifest variables such as the participants’ backgrounds can also cause an assumption error. The number of latent classes can be unknown, and a rational assumption must be made.

On the contrary, when exploring the numbers of classes (the heterogeneity of the population) at first, while making the same factor structure assumption between classes, the unknown variant factor structure hidden in the classes can interfere with the classification accuracy (CA) of the previous step. Besides, a rational factor structure assumption, with one factor or two factors, and others, is required. However, one factor alone cannot reflect a multi-factor appearance (Finch & Finch, 2013). If we want to make different factor structure assumptions between classes, there must be favorable evidence that a rational factor structure assumption can be made.

In summary, the context of multidimensional mixtures is often seen in research. When researchers face scenarios with an unknown factor structure and unknown classes simultaneously, the FMM, with categorical and continuous latent variables, was appropriate to analyze multidimensional mixture data (Arminger et al., 1999; Dolan & van der Maas, 1998; McLachlan & Peel, 2000; B. Muthén & Shedden, 1999; Yung, 1997). However, starting from either the “multi-factor” side or the “latent class” side, can have inevitable assumption error and may cause inaccurate parameter estimation (Finch & Finch, 2013; B. Muthén, 1989). How to make a rational assumption of the factor structures and latent classes were not discussed yet. More factor numbers and latent classes were suggested to be investigated (Lubke & Neale, 2006). Also, the parameter estimation and classification results under different conditions composed of different factor structures and latent classes were vague. Thus, in this article, fluctuating factor structures and classes were considered. Our assumption is that an unknown factor structure and an unknown latent class both exist at the start. And we explore a better procedure for analyzing multidimensional mixture data to fill the gap in previous studies.

The purpose of this study is to propose three procedures for analyzing multidimensional mixture data based on FMM at different levels of factor numbers, factor correlations, latent class numbers, and latent class separations, and to investigate the model selection accuracy (MSA) and the parameter estimation and CA results.

Factor Mixture Models

FMMs involve factor analysis (FA) and latent class analysis. They can be used to estimate heterogeneity in populations and to investigate the appearance of latent traits at the same time. Consider the set of continuous observed variables y related to the continuous latent variables $η$ and the categorical latent variables $c$ for an individual p. A mixture model can be formulated as Equations 1 and 2:

y_{pc} = v_{c} + Λ_{yc} η_{pc} + ε_{pc},

(1)

η_{pc} = A k_{p} + ζ_{pc},

(2)

where $v_{c}$ is the intercept vector, $Λ_{yc}$ is the matrix of factor loadings, and $ε_{pc}$ is the residual vector within the class. The continuous latent variable $η_{pc}$ is the linear combination of the intercept $A$ , the weight $k_{p}$ , and the residual $ζ_{pc}$ ; $k_{p} = (k_{p 1}, \dots, k_{pc})^{'}$ is a vector of a multinomial distribution, where $k_{pc}$ = 1 if the individual $p$ falls under class $c$ , and if it does not, $k_{pc}$ = 0.

Mixture models differ according to their degree of measurement of invariance (B. Muthén, 2008). Figure 1 shows a model with strong measurement invariance, where c is the categorical latent class, $η$ is the continuous factor score, y is the observed item, and the arrow pointing to $η$ is the residual. In that model, c influences $η$ , and then $η$ influences y, which indicates that the factor means and variances may vary between classes, with no changes in slopes in the regression of y on $η$ . It means that there are equal intercepts (without the arrow from c to y) and equal factor loading parameters across the latent classes (without the arrow from c to the arrow from $η$ to y). In this study, the model with a strong measurement invariance assumption was adopted.

Some influential variables were investigated in the past. The class assignment accuracy has been improved for larger class separation and for smaller class separation with covariate effects (Lubke & Neale, 2006). Specifically, the Mahalanobis distance was used to manipulate the degree of class separation, that is, to produce different factor means between classes. It was found that an increasing distance between classes exhibits higher CA (Lubke & Neale, 2006; McDonald, 2013). And, with an increasing number of classes, the CA decreased (McDonald, 2013).

Some variables seemed less effective. Different factor numbers did not influence the CA when there were three to five observed items related to each factor, and 500 individuals seemed to have been the medium sample size (Lubke & Muthén, 2007; McDonald, 2013). Still, balanced and imbalanced mixing proportions of subgroups would not influence the model selection using information criterion (IC) indices (Lubke & Neale, 2006; McDonald, 2013).

Otherwise, there can be different choices of model decision criteria. Comparing multiple fit indices for reference was also recommended (B. Muthén, 2008). The IC indices were commonly used to select the correct model based on likelihood tests; and the Akaike information criterion (AIC; Akaike, 1987), Bayesian information criterion (BIC; Schwartz, 1978), and sample size-adjusted Bayesian information criterion (SABIC; Sclove, 1987) were often used with them. Conditions with larger class separation, which have significantly different intercept or factor loadings between classes, yielded better model selection correctness (Lubke & Neale, 2006).

Extensions to Validity Research and Categorical Data

Litson et al. (2017) applied the mixture model to the multitrait-multimethod (MTMM) framework to assess convergent and discriminant validity in the multidimensional mixture context. By considering the heterogeneity in latent classes of individuals, the framework can detect different convergent and discriminant validity values between subgroups. The steps of the framework include evaluating the degree of measurement invariance in a single class at first, and then exploring different numbers of classes that fit better, and proceeding to compare the equality across classes to see if the trait–method interaction is present. The step “factor structure first then the class number” seemed to be the choice of the study. We ask the following question, however: What happens if a different order is selected?

On the contrary, if the dependent variables are categorical data, two approaches to dealing with the multidimensional mixture data can be used. One of them is the two-stage method, and the other is the single-model method.

Two-Stage Method

Bacci et al. (2014) extended the multidimensional item response theory models of Bartolucci (2007) to polytomous-scored items and suggested a step for model selection when the data are multidimensional and contain more than one latent class. The step for model selection considers the following, in this order: (a) the number of latent classes, (b) the adopted parameterization in terms of the link function and the constraints to the item parameters, and (c) the number of latent dimensions and the corresponding allocation of items to each dimension using the likelihood ratio (LR) test and BIC.

Lin and Wilson (2013) used the mixed Rasch model (Mislevy & Verhelst, 1990; Rost, 1990; von Davier & Yamamoto, 2007) and the multidimensional random coefficients multinomial logit model (MRCMLM; Adams et al., 1997) successively. They first used the mixed Rasch model to identify latent classes, and then they used the MRCMLM in each latent class to model multidimensionality. Bacci et al. (2014) and Lin and Wilson (2013) used the two-stage method of first classifying the participants and then determining the number of latent trait dimensions. However, the second step was influenced by the first step, and the degree of the influence was unknown in previous references.

Single-Model Method

Lin and Wilson (2013) proposed a second way to solve the issue in analyzing multidimensional mixture data. They proposed the multidimensional mixture rating scale model (MMRSM), which considers the concepts of multiple dimensions and latent classes when fitting the rating scale model in each class. Under the model, the probability of person p obtaining a score j on item i can be represented as Equation 3:

P_{ij} (θ_{pdc}) = \frac{\exp [\sum_{s = 0}^{j} (θ_{pdc} - β_{idc} - δ_{sdc})]}{\sum_{h = 0}^{m} \exp [\sum_{s = 0}^{h} (θ_{pdc} - β_{idc} - δ_{sdc})]}, j = 0, 1, \dots, m_{i},

(3)

where $θ_{pdc}$ denotes person p with ability θ in dimension d who belongs to the latent class c; $β_{idc}$ , the difficulty parameter on item i, dimension d, and class c; and $δ_{sdc}$ , the threshold parameter on threshold s, dimension d, and class c.

Lin and Wilson (2013) compared the single-model method and the two-stage method using the dataset of the National Politics Study (NPS), which measures individuals’ political attitudes and behaviors, and had different parameter estimation results but similar classification consistencies of up to 98.3% in the two methods. However, they examined the simulation only in the two-stage method. Thus, the difference between the two methods was still uncertain.

In the mentioned extended studies on validity research and categorical data, the step of analyzing empirical data can only illustrate the procedure, without explanation and verification. In the mixture model estimation, the factor structure has to be set. However, the single-model method cannot analyze unknown factor structure data at the same time. As for the two-stage method, studies have pointed out the procedure for exploring latent classes prior to latent factors, but not the basis, and no simulation study has been conducted yet to prove the rationality of such procedure. This study proposed a solution to this gap.

Methodology

The objectives of this study are to propose three FMM-based procedures for analyzing multidimensional mixture data and to investigate the parameter estimation and CA of such procedures. The procedures and other manipulated variables (factor numbers, factor correlations, class numbers, class separations) adopted in the simulations herein are as follows.

Procedures

Procedure 1 is “factor structure first then class number.” In this procedure, exploratory FA (EFA) was first conducted to detect the factor structure, and then IC was used to select the suitable factor number and structure. FMM was used afterwards. Figure 2 illustrates the steps: first, EFA of one to four factors was conducted; and after two factors were selected, FMMs of 2f1c (i.e., two factors, one class), 2f2c, 2f3c, . . . were produced and compared to choose the better model, after which the parameter estimation results were computed.

Procedure 2 is “class number first then factor structure.” In this procedure, the suitable latent class number and classification result were first executed using FMM, where the factor number was fixed at 1 to explore the numbers of classes, after which the factor structure in each class was explored with EFA of one to four factors.

Finally, the parameters were estimated using the multiple group confirmatory FA (MGCFA) method. Figure 3 illustrates the steps in Procedure 2: first, the FMMs of 1f1c, 1f2c, 1f3c, . . . were compared to choose the better model, and if 1f2c was selected because it had the lowest IC, EFA was conducted to choose the possible factor structure in each of the two classes. The parameter estimation result was computed using MGCFA.

Procedure 3 is “factor structure and class number considered simultaneously.” At the start, EFA was used, different factor structures were obtained, possible combinations of factor structure and class number were assumed simultaneously, and the model with the lowest IC was selected. Figure 4 illustrates the steps: first, EFA was used and factor structures of one, two, three, and four factors were produced; and second, different combinations (1f1c, 1f2c, 1f3c, 2f1c, 2f2c, 2f3c, . . .) were simultaneously assumed. Figure 4 shows the 2f2c model, which was chosen using FMM as it had the lowest IC, and the estimation result will be computed later.

Other Manipulated Variables

The other manipulated variables were the factor numbers (one to four factors), factor correlations (low correlation, about 0.30; and high correlation, about 0.70; Allan et al., 2014; Lubke & Muthén, 2005), class numbers (one, two, and three classes), and class separation (small distance = 1, middle distance = 2, and large distance = 3), measured according to the Mahalanobis Distance (e.g., Lubke & Muthén, 2007; Lubke & Neale, 2006; McDonald, 2013). Mahalanobis Distance is computed using Equation 4:

Mahalanobis Distance (C_{1}, C_{2}) = \sqrt{{(μ_{1} - μ_{2})}^{T} \sum^{- 1} (μ_{1} - μ_{2})} .

(4)

In Equation 4, there are two classes, $C_{1}$ and $C_{2}$ , and their factor-mean matrices are $μ_{1}$ and $μ_{2}$ , respectively. $Σ^{- 1}$ is the invertible matrix of the factor variance–covariance. The parameter settings in each situation are presented in Table 1.

Table 1

Parameters Settings in Each Situation

Situation codeC1 factor meansFactor correlationTest length	A1(0)-5	A2(0,0)Low10	A3(0,0)High10	A4(0,0,0)Low15	A5(0,0,0)High15	A6(0,0,0,0)Low20	A7(0,0,0,0)High20
Situation code C1 factor means C2 factor means Factor correlation Test length	B1 (1) (0) - 5	B2 (0.82,0.82) (0,0) Low 10	B3 (0.95,0.95) (0,0) High 10	B4 (0.71,0.71,0.71) (0,0,0) Low 15	B5 (0.93,0.93,0.93) (0,0,0) High 15	B6 (0.66,0.66,0.66,0.66) (0,0,0,0) Low 20	B7 (0.92,0.92,0.92,0.92) (0,0,0,0) High 20
Situation code C1 factor means C2 factor means Factor correlation Test length	D1 (2) (0) - 5	D2 (1.58,1.58) (0,0) Low 10	D3 (1.89,1.89) (0,0) High 10	D4 (1.41,1.41,1.41) (0,0,0) Low 15	D5 (1.85,1.85,1.85) (0,0,0) High 15	D6 (1.31,1.31,1.31,1.31) (0,0,0,0) Low 20	D7 (1.83,1.83,1.83,1.83) (0,0,0,0) High 20
Situation code C1 factor means C2 factor means Factor correlation Test length	C1 (3) (0) - 5	C2 (2.37,2.37) (0,0) Low 10	C3 (2.84,2.84) (0,0) High 10	C4 (2.12,2.12,2.12) (0,0,0) Low 15	C5 (2.78,2.78,2.78) (0,0,0) High 15	C6 (1.97,1.97,1.97,1.97) (0,0,0,0) Low 20	C7 (2.75,2.75,2.75,2.75) (0,0,0,0) High 20
Situation code C1 factor means C2 factor means C3 factor means Factor correlation Test length	L1 (2) (1) (0) - 5	L2 (1.65,1.65) (0.82,0.82) (0,0) Low 10	L3 (1.90,1.90) (0.95,0.95) (0,0) High 10	L4 (1.50,1.50,1.50) (0.75,0.75,0.75) (0,0,0) Low 15	L5 (1.88,1.88,1.88) (0.94,0.94,0.94) (0,0,0) High 15	L6 (1.32,1.32,1.32,1.32) (0.66,0.66,0.66,0.66) (0,0,0,0) Low 20	L7 (1.84,1.84,1.84,1.84) (0.92,0.92,0.92,0.92) (0,0,0,0) High 20
Situation code C1 factor means C2 factor means C3 factor means Factor correlation Test length	H1 (4) (2) (0) — 5	H2 (3.24,3.24) (1.62,1.62) (0,0) Low 10	H3 (3.78,3.78) (1.89,1.89) (0,0) High 10	H4 (2.95,2.95,2.95) (1.47,1.47,1.47) (0,0,0) Low 15	H5 (3.58,3.58,3.58) (1.79,1.79,1.79) (0,0,0) High 15	H6 (2.62,2.62,2.62,2.62) (1.31,1.31,1.31,1.31) (0,0,0,0) Low 20	H7 (3.66,3.66,3.66,3.66) (1.83,1.83,1.83,1.83) (0,0,0,0) High 20
Situation code C1 factor means C2 factor means C3 factor means Factor correlation Test length	K1 (6) (3) (0) - 5	K2 (4.90,4.90) (2.45,2.45) (0,0) Low 10	K3 (5.56,5.56) (2.77,2.77) (0,0) High 10	K4 (4.40,4.40,4.40) (2.20,2.20,2.20) (0,0,0) Low 15	K5 (5.40,5.40,5.40) (2.70,2.70,2.70) (0,0,0) High 15	K6 (3.94,3.94,3.94,3.94) (1.97,1.97,1.97,1.97) (0,0,0,0) Low 20	K7 (5.5,5.5,5.5,5.5) (2.75,2.75,2.75,2.75) (0,0,0,0) High 20

Besides, the control variables were the test length and the mixing proportion. In this study, one factor was related to 5 items, two factors were related to 10 items, and so on; the mixing proportion was equal; and there were 500 individuals in each class. A total of 49 situations were implemented in this study that were replicated 100 times in each cell, with the simulated responses reproduced in each replication.

As for the IC index, AIC, BIC, and SABIC were used in this study, which were computed using Equations 5, 6, and 7, respectively:

AIC = - 2 \log L + 2 k,

(5)

BIC = - 2 \log L + k \log (n), and

(6)

SABIC = - 2 \log L + k \log (n^{*}), n^{*} = (n + 2) / 24,

(7)

where L denotes the maximum value of the likelihood function; k, the number of parameters in the model; and n, the sample size.

Theoretically, BIC can have a stable effect when the sample is close to infinity. AIC tends to select a complex model. Some studies reported BIC as superior in the mixture model context (Jedidi et al., 1997; McDonald, 2013; Nylund et al., 2007), and other studies found SABIC a fit index due to its different numbers of participants and classes (Henson et al., 2007; Yang, 2006). Lubke and Neale (2006) reported that larger class separation was conducive to the selection of correct models using AIC and SABIC, and that larger class separation with different intercept and factor loadings between groups can be helpful in detecting the correct model using AIC, BIC, and SABIC.

Because of the uncertainty of the effect of IC indices, we compared the performance of the three IC indices and discussed the selection issue in the first part of our results.

Data Analysis

For data analysis, we used the software programs Mplus 7.0 (L. K. Muthén & Muthén, 1998–2015) and R 3.2.2 (https://www.cran.r-project.org/). To calibrate the model parameters, we used maximum-likelihood estimation (MLE). For each replication, we calculated the MSA, mean difference (MD), root mean square difference (RMSD), and CA.

MSA is the ratio of the number of correct model numbers detected using IC to all the replication numbers in one situation. The correct model means the correct combination of factor structure and class number. In selecting the correct model, the following other dependent variables were calculated.

The MDs of the factor loadings, factor means, factor covariance, and residuals were computed separately as follows:

MD (\hat{ζ}) = \frac{\sum_{i = 1}^{I} \sum_{r = 1}^{R} ({\hat{ζ}}_{ir} - ζ_{ir})}{I \times R},

(8)

where i is the number of items, R is the number of simulation replications, ${\hat{ζ}}_{ir}$ is the estimation value of each item, and $ζ_{ir}$ is the true value of each item.

The RMSDs of the said values were computed separately as follows:

RMSD (\hat{ζ}) = \sqrt{\frac{{\sum_{i = 1}^{I} \sum_{r = 1}^{R} ({\hat{ζ}}_{ir} - ζ_{ir})}^{2}}{I \times R}} .

(9)

CA is the percentage of individuals being categorized accurately. When label switching occurred, we compared the true value of the parameters and the estimation ones depending on the factor means to identify the correct class.

Results

IC Indices

Figures 5 and 6 show the MSA of the IC indices. Some characteristics of the IC indices in our study are as follows.

First, overall, the three IC indices yielded better MSAs in situations where Procedures 1 and 3 were used than in situations where Procedure 2 was used. More specifically, with Procedure 2, in the situations of low covariance and more factors, the MSA decreased using the IC indices. BIC and SABIC performed similarly in Procedures 1 and 3 situations, and BIC performed better in Procedure 2 situations. AIC performed less satisfactorily than the others in most situations.

Second, all three IC indices—AIC, BIC, and SABIC—were able to detect the correct model only in few situations with three classes. However, in most of the situations with three classes, their performance was unsatisfactory.

In short, with Procedures 1 and 3, the ratio where BIC was superior to SABIC in the detection of the correct model was 27/49 (27 situations in all the 49 situations), and the ratio where SABIC was superior to BIC was 13/49. With Procedure 2, the ratio where BIC was superior to SABIC was 27/49, and where SABIC was superior to BIC, ratio was 8/49. These results show that BIC is more suitable than SABIC to be the model selection criterion. As reported in previous studies, BIC was a consistent and stable criterion in mixture model situations (Jedidi et al., 1997; McDonald, 2013; Nylund et al., 2007).

In the next part, MD, RMSD, and CA are computed after the correct model had been selected using BIC.

MSA

Table 2 summarizes the MSA results across all the situations. In Procedures 1 and 2, two values are reported to provide more information about the accuracy of the selection of classes and factors.

Table 2

MSA Result in All the Situations

Situations	ProceduresCorrect model	P1: factor first then class	P2: class first then factor	P3: factor and class simultaneously (%)
MSA in single-class situations	1f1c (A1)	100%→98%	98%→98%	98
	2f1c_L (A2)	100%→100%	100%→100%	100
	2f1c_H (A3)	100%→99%	100%→100%	99
	3f1c_L (A4)	100%→99%	100%→100%	99
	3f1c_H (A5)	100%→100%	100%→100%	100
	4f1c_L (A6)	100%→100%	99%→99%	100
	4f1c_H (A7)	100%→100%	100%→100%	100
MSA in two classes with small-separation situations	1f2c (B1)	100%→96%	96%→80%	96
	2f2c_L (B2)	100%→100%	100%→92%	100
	2f2c_H (B3)	100%→100%	100%→100%	100
	3f2c_L (B4)	100%→100%	0%→0%	100
	3f2c_H (B5)	100%→98%	100%→73%	98
	4f2c_L (B6)	100%→100%	0%→0%	100
	4f2c_H (B7)	100%→99%	100%→52%	99
MSA in two classes with medium-separation situations	1f2c (D1)	100%→98%	96%→89%	98
	2f2c_L (D2)	100%→100%	75%→75%	100
	2f2c_H (D3)	100%→100%	99%→91%	100
	3f2c_L (D4)	100%→100%	0%→0%	100
	3f2c_H (D5)	100%→100%	100%→65%	100
	4f2c_L (D6)	100%→100%	0%→0%	100
	4f2c_H (D7)	100%→100%	99%→53%	100
MSA in two classes with large-separation situations	1f2c (C1)	100%→99%	99%→99%	99
	2f2c_L (C2)	100%→100%	7%→7%	100
	2f2c_H (C3)	100%→100%	100%→100%	100
	3f2c_L (C4)	100%→100%	0%→0%	100
	3f2c_H (C5)	86%→86%	100%→100%	86
	4f2c_L (C6)	100%→100%	0%→0%	100
	4f2c_H (C7)	80%→80%	100%→90%	80
MSA in three classes with small-separation situations	1f3c (L1)	100%→0%	0%→0%	0
	2f3c_L (L2)	100%→0%	82%→80%	0
	2f3c_H (L3)	100%→0%	0%→0%	0
	3f3c_L (L4)	100%→0%	0%→0%	0
	3f3c_H (L5)	100%→0%	0%→0%	0
	4f3c_L (L6)	100%→0%	0%→0%	0
	4f3c_H (L7)	100%→0%	0%→0%	0
MSA in three classes with medium-separation situations	1f3c (H1)	100%→50%	75%→51%	50
	2f3c_L (H2)	100%→2%	52%→49%	2
	2f3c_H (H3)	100%→2%	12%→12%	2
	3f3c_L (H4)	100%→0%	0%→0%	0
	3f3c_H (H5)	63%→0%	2%→1%	0
	4f3c_L (H6)	100%→0%	0%→0%	0
	4f3c_H (H7)	50%→0%	1%→1%	0
MSA in three classes with large-separation situations	1f3c (K1)	100%→100%	100%→100%	100
	2f3c_L (K2)	100%→44%	24%→24%	44
	2f3c_H (K3)	100%→73%	99%→99%	73
	3f3c_L (K4)	100%→34%	0%→0%	34
	3f3c_H (K5)	0%→0%	98%→97%	0
	4f3c_L (K6)	100%→15%	0%→0%	15
	4f3c_H (K7)	0%→0%	99%→99%	0

Note. In A1 using P1, 100% → 98% means that if 1f was 100% detected first and then 1c was 98% detected, the MSA will eventually be 98%; and in A1 using P2, 98% → 98% means that if 1c was 98% detected first and then 1f was 98% detected using the correct data following the previous step, the MSA will eventually be 98%. MSA = model selection accuracy.

Table 2 shows that in a single class with all the three procedures, the MSA values were close to or equal to 1. Next, using Procedures 1 and 3, the MSA was good in situations with two classes but decreased in situations with three classes. Thus, increasing class numbers caused lower MSA values. Class separation did not seem to influence the situations with two classes, but MSA increased when class separation rose in three classes. More specifically, high covariance was influential in two classes with large class separation (C5 and C7) and in three classes with medium and large class separation (H5, H7, K5, and K7). In these situations, high covariance prevented the assumed multi-factor structures from showing up and substituted them with a single factor in some replications. For example, in C5, 86% of the data emerged as correctly connected between items and factors, which meant i1–i5 corresponded to the first factor; i6–i10, to the second factor; and i11–i15, to the third factor, as did the study set; and 14% of the data emerged only for a single factor. Lower accuracy of the detection of the factor structure influenced the successive MSAs.

The use of Procedure 2 was also good in single-class conditions, 1f2c conditions (B1, D1, and C1), and some high-covariance conditions (such as B3, D3, C3, C5, and C7). In high-covariance situations, MSA increased and the chance of selecting correct class numbers became higher than in low-covariance situations, such as B3, B5, B7, D3, D5, and D7. More class numbers had lower MSA. High covariance accompanied by large class separation increased the accuracy of the selection of class numbers and possibly also the accuracy of the selection of successive factor structures.

MD, RMSD, and CA

Table 3 presents the MDs, RMSDs, and CAs in single-class situations. The MD fell between −0.01 and 0.03, the RMSD was less than 0.05, and the CA was 1 in all the single-class situations with the three procedures. As stated in the previous section, the MSA, MD, RMSD, and CA were excellent in the single-class situations.

Table 3

MDs, RMSDs, and CAs in a Single Class

Correct model	P1: factor first then classP3: factor and class simultaneously				P2: class first then factor
Correct model	$Λ$	$ψ$	$ε$	CA	$Λ$	$ψ$	$ε$	CA
1f1c (A1)	.00/.00	—	−.01/.00	1.00	.00/.00	—	−.01/.00	1.00
2f1c_L (A2)	.01/.02	−.01/.02	−.01/.03	1.00	.01/.02	−.01/.02	−.01/.03	1.00
2f1c_H (A3)	.00/.02	−.02/.02	−.01/.03	1.00	.00/.02	−.02/.02	−.01/.03	1.00
3f1c_L (A4)	.00/.02	−.01/.02	−.01/.03	1.00	.00/.02	−.01/.02	−.01/.03	1.00
3f1c_H (A5)	.00/.02	−.01/.02	−.01/.02	1.00	.00/.02	−.01/.02	−.01/.02	1.00
4f1c_L (A6)	.01/.02	−.02/.04	−.01/.03	1.00	.01/.02	−.02/.04	−.01/.03	1.00
4f1c_H (A7)	−.01/.02	.03/.05	.01/.03	1.00	−.01/.02	.03/.05	.01/.03	1.00

Note. $Λ$ means factor loadings; $ψ$ , covariance; and $ε$ , residual variance. The values on the left of the slashes are the means of the MDs; for example, in A6, $Λ = (Λ_{1} + Λ_{2} + Λ_{3} + Λ_{4}) / 4$ , $ψ = (ψ_{12} + ψ_{13} + ψ_{14} + ψ_{23} + ψ_{24} + ψ_{34}) / 6$ , and $ε = (ε_{1} + ε_{2} + ε_{3} + ε_{4}) / 4$ . The values on the right of the slashes are the means of the RMSD; and “—” means not estimated. MD = mean difference; RMSD = root mean square difference; CA = classification accuracy.

Table 4 presents the MDs, RMSDs, and CAs in the two-class situations. In the two-class situations with small separation, with the use of Procedures 1 and 3, the MSAs were above 96%, the MDs were between −0.31 and 0.18, the RMSDs were between 0.01 and 0.96, and the CAs were about 55% to 63%. Although the MSAs were excellent, the CAs were only about 55% to 63%, accompanied by underestimated and variant factor means. Using Procedure 2, the MSAs were 0 in three- and four-factor situations with low covariance. The MDs in other situations were between −0.50 and 0.43, the RMSDs were between 0.01 and 0.96, and the CAs were about 51% to 55%. Except for the wrong model selection in the three- and four-factor situations with low covariance, the values were biased and variant, especially in terms of the factor means, and the CAs were unsatisfying.

Table 4

MD, RMSD, and CA in Two Classes

Two classes with small separation
Correct model		P1: factor first then classP3: factor and class simultaneously					P2: class first then factor
Correct model		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f2c (B1)	C1	−.02/.01	.18/.96	—	.03/.11	.55	−.02/.01	.18/.96	—	.03/.11	.55
1f2c (B1)	C2	−.04/.16	—	—	.04/.24		−.04/.16	—	—	.04/.24
2f2c_L (B2)	C1	.00/.02	−.06/.24	.02/.06	−.01/.04	.63	.04/.04	−.44/.45	.43/.46	−.05/.07	.51
2f2c_L (B2)	C2	.01/.05	—	−.01/.06	−.01/.08		.03/.04	—	−.20/.29	−.04/.07
2f2c_H (B3)	C1	.02/.03	−.20/.35	.01/.04	−.02/.05	.60	.03/.04	−.49/.54	.05/.07	−.05/.06	.52
2f2c_H (B3)	C2	.02/.06	—	−.04/.17	−.03/.10		−.01/.09	—	−.48/.58	.01/.15
3f2c_L (B4)	C1	.01/.02	−.23/.27	.06/.08	−.02/.04	.56
3f2c_L (B4)	C2	.01/.07	—	−.01/.26	−.03/.12
3f2c_H (B5)	C1	.01/.03	−.23/.38	−.00/.06	−.02/.05	.58	.03/.03	−.49/.49	.06/.06	−.05/.05	.52
3f2c_H (B5)	C2	.02/.02	—	.03/.03	−.03/.10		.00/.06	—	−.22/.32	−.00/.10
4f2c_L (B6)	C1	.01/.02	−.27/.79	.04/.09	−.02/.04	.55
4f2c_L (B6)	C2	.02/.07	—	.01/.25	−.03/.12
4f2c_H (B7)	C1	.02/.03	−.31/.58	.01/.12	−.03/.05	.56	.03/.03	−.50/.50	.06/.07	−.05/.05	.52
4f2c_H (B7)	C2	.02/.07	—	.01/.10	−.04/.11		.02/.03	—	−.16/.28	−.05/.06
Two classes with medium separation
		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f2c (D1)	C1	.01/.03	−.01/.05	—	−.01/.05	.79	.01/.03	−.01/.05	—	−.01/.05	.79
1f2c (D1)	C2	.01/.05	—	—	−.01/.05		.01/.05	—	—	−.01/.05
2f2c_L (D2)	C1	.00/.02	.00/.13	−.01/.07	.00/.03	.83	.05/.06	−.97/.97	.41/.48	−.10/.11	.52
2f2c_L (D2)	C2	.00/.02	—	.00/.05	.00/.03		.05/.06	—	−.11/.13	−.09/.11
2f2c_H (D3)	C1	.00/.02	−.01/.20	−.01/.06	.00/.03	.83	.01/.07	−.09/.91	−.03/.13	.00/.11	.70
2f2c_H (D3)	C2	.00/.02	—	.00/.05	.00/.03		−.02/.06	—	−.15/.29	.02/.10
3f2c_L (D4)	C1	.00/.02	−.00/.15	.00/.06	.00/.03	.82
3f2c_L (D4)	C2	.00/.02	—	.00/.05	.00/.03
3f2c_H (D5)	C1	.00/.02	−.04/.15	.00/.06	−.01/.03	.83	.05/.07	−.76/1.03	.06/.12	−.08/.12	.59
3f2c_H (D5)	C2	.00/.02	—	.00/.05	.00/.03		−.01/.07	—	−.15/.20	−.02/.11
4f2c_L (D6)	C1	.00/.02	−.02/.17	.00/.08	−.00/.03	.81
4f2c_L (D6)	C2	.00/.02	—	.01/.06	−.01/.04
4f2c_H (D7)	C1	.00/.02	−.01/.19	.00/.05	.00/.03	.83	.06/.07	−.96/1.07	.11/.12	−.10/.12	.56
4f2c_H (D7)	C2	.00/.02	—	−.00/.06	.00/.03		−.04/.06	—	−.08/.12	.06/.10
Two classes with large-separation
		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f2c (C1)	C1	.00/.02	−.01/.12	—	−.03/.02	.92	.00/.02	−.01/.12	—	−.03/.02	.92
1f2c (C1)	C2	.00/.02	—	—	−.00/.02		.00/.02	—	—	−.00/.02
2f2c_L (C2)	C1	.00/.02	.00/.10	.00/.04	.00/.03	.92	.11/.11	−1.6/1.6	.62/.62	−.19/.19	.52
2f2c_L (C2)	C2	.00/.02	—	.01/.05	.00/.02		.11/.11	—	.03/.04	−.19/.19
2f2c_H (C3)	C1	.00/.01	−.01/.12	−.01/.05	.00/.02	.92	−.03/.03	.35/.36	−.05/.07	.04/.05	.93
2f2c_H (C3)	C2	.00/.02	—	−.01/.06	.00/.02		−.03/.03	—	−.05/.08	.05/.05
3f2c_L (C4)	C1	.00/.02	.01/.09	.00/.04	.00/.02	.92
3f2c_L (C4)	C2	.00/.02	—	.00/.04	.00/.02
3f2c_H (C5)	C1	.00/.02	−.01/.11	.00/.06	.00/.02	.93	−.03/.03	.31/.32	−.04/.07	.04/.05	.93
3f2c_H (C5)	C2	.00/.02	—	.00/.05	.00/.02		−.03/.03	—	−.04/.07	.04/.05
4f2c_L (C6)	C1	.00/.02	.00/.09	.00/.05	.00/.02	.92
4f2c_L (C6)	C2	.00/.02	—	.00/.05	.00/.02
4f2c_H (C7)	C1	.00/.02	.02/.11	.00/.06	.00/.02	.93	.00/.01	.27/.41	.00/.05	.00/.02	.92
4f2c_H (C7)	C2	.00/.02	—	.01/.06	.00/.02		−.02/.03	—	−.03/.07	.04/.06

Note. $η$ is the factor mean; a blank means a wrong model was selected so parameter estimation was not applicable; and “—” means not estimated. MD = mean difference; RMSD = root mean square difference; CA = classification accuracy.

In the two-class situations with medium-class separation, using Procedures 1 and 3, the MSAs were above 98%, the MDs were between −0.04 and 0.01, the RMSDs were between 0.02 and 0.20, and the CA was around 80%. The MSAs were excellent, with less biased and more stable parameter estimation results, and the CAs were acceptable.

Using Procedure 2, the MSAs were 0 in three- and four-factor situations with low covariance. In other situations, the MDs were between −0.97 and 0.41, the RMSDs were between 0.01 and 1.07, and the CAs were about 52% to 79%. Procedure 2 led to dissatisfactory MDs, RMSDs, and CAs.

In the two-class situations with large separation, using Procedures 1 and 3, the MSAs ranged from 80% to 100%, the MDs were between −0.03 and 0.02, the RMSDs were between 0.01 and 0.12, and the CAs were about 92% to 93%. Thus, the MSAs, MDs, RMSDs, and CAs were satisfactory. Using Procedure 2, the MSAs were 0 in three- and four-factor situations with low covariance, and the MSAs were between 90% and 100% in single-factor and high-covariance situations. There were also better MDs, RMSDs, and CAs in high-covariance situations. In the single-factor and high-covariance situations, the MDs were between −0.05 and 0.35, the RMSDs were between 0.01 and 0.36, and the CAs could reach about 92% to 93%. Thus, there were better results in the single-factor and high-covariance situations.

Table 5 presents the parameter estimation results in three classes. In the three-class situations with small separation, the MSAs were all 0 using procedures 1 and 3. Using Procedure 2, the MSA in the low-covariance situation reached 80%, the MDs were between −0.89 and 0.28, the RMSDs were between 0.06 and 0.90, and the CA was merely 35%. Thus, the MSAs, MDs, RMSDs, and CA were unsatisfactory.

Table 5

MDs, RMSDs, and CAs in Three Classes

Three classes with small separation
Correct model		P1 : factor first then classP3 : factor and class simultaneously					P2 : class first then factor
		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f3c(L1)
2f3c_L(L2)	C1						.04/.06	−.89/.90	.01/.37	−.07/.09	.35
	C2						.05/.06	−.17/.20	.28/.48	−.09/.10
	C3						.04/.09	—	−.08/.36	−.07/.15
2f3c_H(L3)
3f3c_L(L4)
3f3c_H(L5)
4f3c_L(L6)
4f3c_H(L7)
Three classes with medium separation
		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f3c(H1)	C1	.00/.03	−.11/.70	—	.00/.01	.73	.00/.03	−.11/.70	—	.00/.01	.73
	C2	.00/.03	−.14/.56	—	.00/.05		.00/.03	−.14/.56	—	.00/.05
	C3	.00/.05	—	—	.00/.08		.00/.05	—	—	.00/.08
2f3c_L(H2)	C1	.03/.03	−.06/.61	.11/.12	−.04/.05	.43	.11/.12	−2.23/2.23	.14/.31	−.19/.20	.35
	C2	.02/.03	−1.09/1.09	.13/.15	−.06/.06		.11/.12	−.68/.68	.50/.56	−.19/.20
	C3	.02/.03	—	.26/.56	−.02/.14		.11/.12	—	.02/.18	−.19/.20
2f3c_H(H3)	C1	.05/.05	−.94/.95	.04/.05	−.08/.09	.53	−.02/.05	.13/.83	−.04/.09	.03/.08	.60
	C2	.05/.05	−1.34/1.35	.07/.09	−.08/.09		−.02/.05	−.86/1.23	.01/.06	.03/.07
	C3	.08/.13	—	.11/.11	−.15/.24		−.05/.10	—	−.04/.14	.06/.15
3f3c_L(H4)
3f3c_L(H4)
3f3c_H(H5)	C1						.01/.01	−.41/.41	.02/.03	−.01/.02	.53
	C2						.01/.02	−1.41/1.41	.10/.10	−.01/.03
	C3						−.03/.08	—	−.27/.27	−.04/.18
4f3c_L(H6)
4f3c_L(H6)
4f3c_H(H7)	C1						.01/.02	−.49/.49	.01/.02	−.02/.03	.53
	C2						.01/.02	−1.37/1.37	.08/.08	−.02/.03
	C3						.00/.06	—	−.17/.20	−.01/.10
Three classes with large separation
		$Λ$	$η$	$ψ$	$ε$	CA	$Λ$	$η$	$ψ$	$ε$	CA
1f3c(K1)	C1	.00/.01	.04/.21	—	.00/.02	.89	.00/.01	.04/.21	—	.00/.02	.89
	C2	.00/.02	.02/.13	—	.00/.02		.00/.02	.02/.13	—	.00/.02
	C3	.00/.02	—	—	.00/.02		.00/.02	—	—	.00/.02
2f3c_L(K2)	C1	.02/.05	−.56/1.05	.05/.13	−.04/.07	.78	.15/.15	−3.63/3.63	.31/.35	−.25/.25	.36
	C2	.02/.04	−.39/1.12	.04/.12	−.03/.07		.15/.15	−1.33/1.34	.65/.66	−.26/.26
	C3	.02/.06	—	.02/.14	−.03/.10		.14/.14	—	.20/.23	−.25/.26
2f3c_H(K3)	C1	.05/.07	−1.60/1.96	.05/.10	−.08/.11	.67	−.04/.04	.86/.87	−.07/.09	.06/.06	.90
	C2	.05/.07	−.99/1.68	.07/.11	−.08/.12		−.04/.04	.40/.41	−.07/.10	.06/.06
	C3	.04/.08	—	.03/.14	−.08/.14		−.04/.04	—	−.07/.09	.06/.07
3f3c_L(K4)	C1	.00/.01	.03/.16	−.02/.04	.00/.05	.91
	C2	.00/.02	.02/.08	.00/.06	.00/.02
	C3	.00/.01	—	.00/.04	.00/.02
3f3c_H(K5)	C1						−.03/.04	.75/.75	−.06/.09	.05/.06	.91
	C2						−.03/.04	.34/.35	−.07/.09	.05/.05
	C3						−.03/.04	—	−.05/.07	.05/.06
4f3c_L(K6)	C1	.01/.02	−.01/.14	.00/.09	−.01/.03	.90
	C2	−.01/.04	−.03/.09	.02/.08	.01/.06
	C3	.01/.02	—	.02/.07	−.02/.04
4f3c_H(K7)	C1						−.03/.04	.67/.77	−.06/.08	.05/.06	.90
	C2						−.03/.04	.28/.47	−.07/.08	.05/.06
	C3						−.03/.04	—	−.05/.08	.05/.06

Note. MD = mean difference; RMSD = root mean square difference; CA = classification accuracy.

In the three-class situations with medium separation, the MSAs were only 50% in the single-factor situation and only 2% in the two-factor situations using Procedures 1 and 3. The MSAs were 0 in the three- and four-factor situations. In the single- and two-class situations, the MDs were between −1.34 and 0.26, the RMSDs were between 0.01 and 1.35, and the CAs were about 43% to 73%. Using Procedure 2, the MSAs were 0 in the three- and four-factor situations with low covariance. In other situations, the MSAs ranged between 1% and 51%. The MDs were between −2.23 and 0.50, the RMSDs were between 0.01 and 2.23, and the CAs were about 35% to 73%. The MSAs and CAs were low, and the parameter estimation results were still unsatisfactory.

In the three-class situations with large separation, using Procedures 1 and 3, the MSA was 0 in the three- and four-factor situations with high covariance, and the MSAs were between 15% and 100% in the other situations. The MDs ranged from −1.60 to 0.07, the RMSDs were between 0.01 and 1.96, and the CAs were about 67% to 91%. In the three- and four-factor situations with low covariance, if the correct model was selected, the CA and parameter estimation results were satisfactory. Using Procedure 2, the MSAs were 0 in the three- and four-factor situations with low covariance, and between 24% and 100% in the other situations. The MDs ranged from −3.63 to 0.86, the RMSDs were between 0.01 and 3.63, and the CAs were around 90% and dropped to 36% in the 2f3c situation with low covariance. The single-factor and high-covariance situations led to better MSAs, MDs, RMSDs, and CAs. Despite the poor performance in the three-class situations, when the distance increased to large separation, the parameter results in 1f3c were good; and in the 3f3c and 4f3c situations, the three different procedures were complementary.

Overall, the three procedures were good in the single-class situations. In the two-class situations, the performance of Procedures 1 and 3 was better than that of Procedure 2. Without considering the parameter estimation results, Procedure 2 could have similar CA results as Procedures 1 and 3 in large-class-separation and high-covariance situations. When the data were composed of three classes, the three procedures were unsatisfactory. Procedures 1 and 3 did not have the same performance as in the two-class situations. Procedure 2 seemed to have some advantages over Procedures 1 and 3 in three-class situations with large separation and high covariance.

Discussion

Most past studies that analyzed data with unknown class characteristics used the confirmatory approach (Bacci et al., 2014; Lin & Wilson, 2013; Litson et al., 2017). Such studies, whether using the two-stage or single-model method or applying the method to empirical data, often set the factor structure first and then required latent-class-category results. The implementation was limited by the suspected factor structure, because adoption of the wrongly assumed factor structure resulted in the wrong category. Besides, few studies have been conducted with more multi-factors and classes. So we increase the factor numbers to three and four factors, and also increase the class numbers to three classes. Therefore, in this study, we used three different procedures and found that different combinations of such procedures, factor numbers, factor correlations, class numbers, and class separation could influence the dependent variables, that is, MSA, MD, RMSD, and CA. The effects of the independent variables are as follows.

Three Procedures

When the data had a continuous appearance, the use of Procedures 1 and 3 with BIC enabled detection of the true model in single and two classes, whereas the MSA decreased in three classes. Procedure 2 with BIC enabled detection of the correct model in single-class, single-factor, and two-factor situations with high covariance, and sometimes had unsatisfactory parameter estimation results. Procedures 1 and 3 showed stable performance, whereas Procedure 2 could have performed better in some situations. However, Procedure 3 was still restricted to the EFA structure, besides which it took much more time to estimate all the assumed models and its performance was the same as that of Procedure 1.

Our data imply that the same factor structure between classes contributed to the better results with the use of Procedures 1 and 3. When Procedure 2 was used, because the main difference in the latent classes came from the factor means, it seemed difficult to distinguish the latent class. To confirm the conjecture, future research can investigate situations with different factor loadings and factor structures, to find out to what extent the mixture model can be applied with different procedures. Past studies (Litson et al., 2017) suggested deciding on the factor structure first and then deciding different class numbers without discussing the impact of different procedures. Lin and Wilson (2013) failed to compare the two methods in both their simulation and empirical data. This study is a start in defining the different procedures with separate steps and implementing them with simulation data. A complete research design and comprehensive understanding are important to obtain rewarding conclusions.

Factor Numbers

The use of Procedures 1 and 3 did not seem to have influenced the MSA in the single- and two-class situations as the factor numbers increased. However, in three-class situations, more factors led to a slightly lower MSA. This is because when dealing with multidimensional mixture data, EFA could not reflect the correct factor structure. The results echoed the issue that Armstrong mentioned in 1967 on the inability of EFA to uncover the expected dimensions in theory. In this study, it was found that more factors and more classes with large class separation are among the reasons for the failure of EFA to identify the underlying factor structure.

Otherwise, EFA could use MLE and consider the IC index result in choosing the better-fit model (Fabrigar et al., 1999). However, there are still other methods that are commonly used to decide on the better factor numbers and structures, such as the scree plot and the eigenvalue method. In this study, we found that BIC can be used to decide on the correct factor structure in single- and two-class situations with Procedures 1 and 3, and that BIC can also contribute in single-class situations and in some high-covariance situations with two and three classes using Procedure 2.

Factor Correlations

When Procedures 1 and 3 were used, the factor correlations did not seem to influence the MSA in single-factor situations and in most of the two-factor situations. However, in two classes with large separation and in three classes with medium separation and large separation, the correct factor structure in high-covariance situations may not be detected. This study showed that factor covariance affects the MSA in situations with more classes and high covariance. Some studies (Allan et al., 2014; Lubke & Muthén, 2005) have found that the factor covariance varies in empirical data, and our study showed the mixture model appearances at different degrees of factor covariance.

When Procedure 2 was used, higher MSAs and CAs were seen in factors with high covariance. Thus, high covariance with large class separation favored Procedure 2. When the factors were with high covariance, the situations were close to the single-factor assumption of Procedure 2. Thus, it may be possible to start from testing 1f1c, 1f2c, 1f3c, . . . and then choosing a suitable class. Thus, in the low-covariance situations, the MSA decreased. In the future, we can try to relax the factor restrictions and use another classifying method such as latent profile analysis (LPA) in Procedure 2 to find a better way of finding suitable class numbers.

Class Numbers

In Procedures 1 and 3, as the class numbers increased to three classes, the MSA and CA decreased. In Procedure 2, the MSAs were good in single-class situations but decreased as the class numbers increased. More class numbers led to lower MSAs and CAs. The results are identical to those of McDonald (2013), in which three- and four-class numbers led to lower CAs.

Class Separation

In Procedures 1 and 3, when class separation increased, the MSAs and CAs increased. Procedure 2 had similar results. In short, when class separation increased, the CA increased, as they did in past studies (Lubke & Neale, 2006; McDonald, 2013).

Conclusion

The purpose of our research focused on choosing suitable procedures to analyze multidimensional mixture data. When the factor structure and participants’ classification were both unknown, the three procedures performed well in single-class situations. Procedures 1 and 3 performed better than Procedure 2 in two-class situations. In three-class situations, the performance of all three procedures was limited.

Thus, we prioritize the use of Procedures 1 and 3 over Procedure 2 in fewer classes. Procedure 1 will be an appropriate choice to save time. If we apply the procedure to real data in one- and two-class situations with different factor means, we can expect good MSA, MD, RMSD, and CA results.

The settings of our simulation data assumed the same factor structure except for the difference of the factor means between classes, while the step of Procedures 1 and 3 started from the assumption of the same factor structure within each subpopulation. It might be probably the consistence between simulation data and Procedures 1 and 3, which lead to better results. Or perhaps, large amount of participants is beneficial to identify the connection between factors and items. But it would be a challenge to Procedure 2, because the classification is identified at first, and the connection between factors and items is explored in each class. Fewer participants in each class supply less information to distinguish correct factor structure. The explanations are supposed and needed further inspection.

The settings of our research mainly considered differences in the factor means. However, the degree of factor invariance varies in reality, sometimes it can be with different factor loadings between classes, or the factor structure differs from class to class. More situations should be manipulated in succession. Still, there are different types of mixture models, and different multidimensional type, for example, the item measuring two factors or more. Those circumstances are not yet investigated.

Future research is also needed to examine more complex models or different response styles (e.g., problem-solving strategies, speed tests, different levels of motivation, and positive and negative word effects) and categorical response types (binary and polytomous items) for the sake of understanding the mixture model more thoroughly.

As for the estimation method, the study used MLE, thus giving the assumption of the linear link function between factors and items. In some previous studies, Bayesian estimation was also adopted. In some situations that lacked the ideal MSA, parameter estimation results, and CA, such as three-class situations, Bayesian estimation may be a substitute choice.

As for the IC, in the aspect of selecting indices for model selection, the BIC index had advantages over the other IC indices in the selection of the correct model. In the future, we can take BIC as a suitable reference for selecting the correct mixture model when the data have continuous indicators.

The BIC index was inspected and recommended in our study, which is based on the concept of maximizing the likelihood function. Some other indices can be investigated, such as the Lo–Mendell–Rubin test (LMR; Lo et al., 2001) and the bootstrapped likelihood ratio test (BLRT; Nylund et al., 2007).

There are other relative latent-class-analysis methods for variables not included in this study. Different classification approaches can be applied in the future. Moreover, the findings from this study were limited to the simulation conditions, so the conclusions and suggestions are based on the context of multidimensional mixture data from a confirmatory perspective and using measurement invariance models. Considering those assumptions can help with the selection of the most suitable procedure backed by the most reasonable explanation.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Hsu-Lin Su

Po-Hsi Chen

References

Adams

Wilson

Wang

(1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. https://doi.org/10.1177/0146621697211001

Akaike

(1987). Factor analysis and AIC. Psychometrika, 52, 317–332. https://doi.org/10.1007/BF02294359

Allan

N. P.

Raines

A. M.

Capron

D. W.

Norr

A. M.

Zvolensky

M. J.

Schmidt

N. B.

(2014). Identification of anxiety sensitivity classes and clinical cut-scores in a sample of adult smokers: Results from a factor mixture model. Journal of Anxiety Disorders, 28, 696–703. https://doi.org/10.1016/j.janxdis.2014.07.006

Arminger

Stein

Wittenberg

(1999). Mixtures of conditional mean- and covariance-structure models. Psychometrika, 64(4), 475–494. https://doi.org/10.1007/BF02294568

Bacci

Bartolucci

Gnaldi

(2014). A class of multidimensional latent class IRT models for ordinal polytomous item responses. Communications in Statistics—Theory and Methods, 43(4), 787–800. https://doi.org/10.1080/03610926.2013.827718

Bartolucci

(2007). A class of multidimensional IRT models for testing unidimensionality and clustering items. Psychometrika, 72, 141–157. https://doi.org/10.1007/s11336-005-1376-9

Costa

P. T.

Jr. McCrae

R. R.

(1978). Objective personality assessment. In Storandt

Siegler

I. C.

Elias

M. F.

(Eds.), The clinical psychology of aging (pp. 119–143). Plenum.

De Boeck

Cho

S. J.

Wilson

. (2011). Explanatory secondary dimension modeling of latent differential item functioning. Applied Psychological Measurement, 35(8), 583–603. https://doi.org/10.1177/0146621611428446

Dolan

C. V.

van der Maas

H. L. J.

(1998). Fitting multivariate normal finite mixtures subject to structural equation modeling. Psychometrika, 63(3), 227–253. https://doi.org/10.1007/BF02294853

10.

Fabrigar

L. R.

Wegener

D. T.

MacCallum

R. C.

Strahan

E. J.

(1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299. https://doi.org/10.1037/1082-989X.4.3.272

11.

Finch

W. H.

Finch

M. E. H.

(2013). Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educational and Psychological Measurement, 73(6), 973–993. https://doi.org/10.1177/0013164413494776

12.

Henson

J. M.

Reise

S. P.

Kim

K. H.

(2007). Detecting mixtures from structural model differences using latent variable mixture modeling: A comparison of relative model fit statistics. Structural Equation Modeling: A Multidisciplinary Journal, 14(2), 202–226. https://doi.org/10.1080/10705510709336744

13.

Jedidi

Jagpal

H. S.

DeSarbo

W. S.

(1997). Finite-mixture structural equation models for response-based segmentation and unobserved heterogeneity. Marketing Science, 16(1), 39–59. https://doi.org/10.1287/mksc.16.1.39

14.

Lin

Y. H.

Wilson

(2013). Two IRT methods for multidimensional latent class parameter estimation. Tzu-Chi University Journal of the Educational Research, 9, 223–254.

15.

Litson

Geiser

Burns

G. L.

Servera

(2017). Examining trait×method interactions using mixture distribution multitrait-multimethod models. Structural Equation Modeling: A Multidisciplinary Journal, 24(1), 31–51. https://doi.org/10.1080/10705511.2016.1238307

16.

Mendell

Rubin

(2001). Testing the number of components in a normal mixture. Biometrika, 88, 767–778. https://doi.org/10.1093/biomet/88.3.767

17.

Lubke

G. H.

Muthén

(2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10, 21–39. https://doi.org/10.1037/1082-989X.10.1.21

18.

Lubke

G. H.

Muthén

(2007). Performance of factor mixture models as a function of model size, covariate effects, and class-specific parameters. Structural Equation Modeling, 14(1), 26–47. https://doi.org/10.1080/10705510709336735

19.

Lubke

G. H.

Neale

M. C.

(2006). Distinguishing between latent classes and continuous factors: Resolution by maximum likelihood. Multivariate Behavioral Research, 41, 499–532. https://doi.org/10.1207/s15327906mbr4104_4

20.

McDonald

S. R.

(2013). Identification of latent classes in mixture models: A Monte Carlo simulation study [Doctoral dissertation, University of California, Riverside].

21.

Mislevy

R. J.

Verhelst

(1990). Modeling item responses when different subjects employ different solution strategies. Psychometrika, 55, 195–215. https://doi.org/10.1007/BF02295283

22.

Muthén

(1989). Latent variable modeling in heterogeneous populations. Psychometrika, 54, 557–585. https://doi.org/10.1007/BF02296397

23.

Muthén

(2008). Latent variable hybrids: Overview of old and new models. In Hancock

G. R.

Samuelsen

K. M.

(Eds.), Advances in latent variable mixture models (pp. 1–24). Information Age Publishing.

24.

Muthén

Shedden

(1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 55(2), 463–469. https://doi.org/10.1111/j.0006-341x.1999.00463.x

25.

Muthén

L. K.

Muthén

B. O.

(1998–2015). Mplus user’s guide (7th ed.).

26.

Nylund

K. L.

Asparouhov

Muthén

(2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569. https://doi.org/10.1080/10705510701575396

27.

Rost

(1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14, 271–282. https://doi.org/10.1177/014662169001400305

28.

Schwarz

G. E

. (1978). Estimating the dimension of a model. Annals of Statistics, 6 (2), 461–464.https://doi.org/10.1214/aos/1176344136

29.

Sclove

(1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333–343. https://doi.org/10.1007/BF02294360

30.

Thurstone

L. L.

(1947). Multiple factor analysis: A development and expansion of vectors of the mind. University of Chicago Press.

31.

von Davier

Rost

Carstensen

C. H

. (2007). Introduction: Extending the Rasch model. In von Davier

Carstensen

C. H.

(Eds.), Multivariate and mixture distribution Rasch models: Extensions and applications (pp. 1–12). Springer.

32.

Yang

C. C.

(2006). Evaluating latent class analysis models in qualitative phenotype identification. Computational Statistics and Data Analysis, 50(4), 1090–1104. https://doi.org/10.1016/j.csda.2004.11.004

33.

Yung

(1997). Finite mixtures in confirmatory factor analysis models. Psychometrika, 62(3), 297–330. https://doi.org/10.1007/BF02294554