A Commentary on the Relationship Between Model Fit and Saturated Path Models in Structural Equation Modeling Applications

Abstract

The relationship between saturated path-analysis models and their fit to data is revisited. It is demonstrated that a saturated model need not fit perfectly or even well a given data set when fit to the raw data is examined, a criterion currently frequently overlooked by researchers utilizing path analysis modeling techniques. The potential of individual case residuals for saturated model fit assessment is revealed by showing how they can be used to examine local fit, as opposed to overall fit, to sense possible model deficiencies or misspecifications, and to suggest model improvements when needed. The discussion is illustrated with several numerical examples.

Keywords

individual case residual local fit overall fit path-analysis model saturated model

Structural equation modeling (SEM) has enjoyed much popularity among educational, behavioral, social, biomedical, business, and marketing scientists for more than four decades. A prominent role in its applications in these and cognate disciplines has been played by the so-called “saturated model” (SM). This model has been routinely used as a benchmark when evaluating the fit of structural equation models with positive degrees of freedom. An SM is also very useful when one is interested in estimating certain model parameters, with associated measures of instability (standard errors), and no existing theory or a priori hypothesis can be used to develop further parameter constraints rendering a model under consideration with positive degrees of freedom (e.g., Raykov, Marcoulides, & Patelis, 2013).

A widely appreciated feature of an SM in much of the SEM literature seems to be the fact that an SM “fits a given data set perfectly” (see next section for a needed qualification). Typically, references to this feature of an SM have been made in contexts where only overall model fit has been meant, although explicit mentioning of this has been rare. These references, however, miss another main criterion when assessing quality of statistical models, namely the degree to which a model under consideration actually fits the analyzed raw data. The extant SEM literature also seems not to have attended sufficiently to this criterion, which has been routinely followed for instance in widespread applications of the general linear model (e.g., Draper & Smith, 1998). In fact, use of this criterion with saturated path-analysis models, as we demonstrate later, can have consequential implications for its quality as a means of describing and explaining an available data set.

This article revisits the relationship between model fit and the concept of a saturated path-analysis model (referred to as “saturated path model” in this discussion) in SEM applications. Contrary to the large majority of SEM utilizations, the following discussion is characterized by the view that evaluation of the degree to which a given saturated path model fits analyzed data cannot be complete without use of the above criterion of raw data fit, that is, without assessment of the extent to which the model fits the raw data on the subjects/cases in the sample. Following this view, we demonstrate that a saturated model need not necessarily fit an available data set well and hence cannot be generally considered “fitting perfectly.” We also show how applying routinely this criterion in empirical research with saturated path models can lead to improved model building, a major analytic activity that educational, behavioral, and social scientists frequently conduct.

Notation and Assumptions

To accomplish the goals of this article, we suppose that a set of observed variables are given, denoted as y₁, y₂, . . ., y_k (k > 1). For its aims, the following discussion will assume for a model in question multinormality of the response variables given the explanatory variables. The latter will also be assumed measured without error (or only negligible such), and we will be concerned in the remainder of this discussion with a path-analytic framework and identified saturated models considered within it.¹ A saturated model, or SM, is defined here as a model with d = 0 degrees of freedom (df), that is, with as many parameters as there are variances, covariances, and means of the k observed variables (cf. Bollen, 1989). Collecting the observed variables in the vector y of size k×1, a saturated path model can be represented by the following set of equations:

y = m + B y + e .

In Equation (1), m denotes a k×1 vector of intercepts, B is the k×k matrix of regression coefficients for response variables on predictors, with I_k−B assumed invertible (I_k denoting the identity matrix of size k×k), and e is a k×1 vector of error terms with unrestricted covariance matrix and assumed uncorrelated with associated predictors, entailing d = 0 (including all path coefficients of outcomes on predictors as model parameters as well as their variances and covariances; e.g., Bentler, 2004).

Following Bollen and Arminger (1991), individual case residuals (ICRs) for the ith subject and jth response variable can be directly estimated for Model (1) as

{\hat{r}}_{i j} = y_{i j} - {\hat{m}}_{j} - \sum_{s = 1}^{p_{j}} {\hat{γ}}_{j s} x_{i s},

where for simplicity the xs denote the p_j explanatory variables posited for the jth outcome y_j within the model, whereas ${\hat{m}}_{j}$ and ${\hat{γ}}_{j s}$ are the maximum likelihood (ML) estimates of the intercept and pertinent path coefficients (i = 1, . . ., n, j = 1, . . ., k, s = 1, . . ., p_j; when the same predictors are used for all response variables, d = 0 ensues, which is the case of interest in this article; see also Conclusion section). In contrast to overall fit, usually indexed by the popular chi-square goodness-of-fit value or the root mean square error of approximation (RMSEA; e.g., Bollen, 1989, and references therein), the ICRs in Equation (2) can be viewed as indexes of local fit of a given model to an available data set.²

How Could a Saturated Model Be Associated With “Perfect” Fit?

In the extant SEM literature, particularly that dealing with path-analysis type SEM applications, one can find multiple statements of a saturated model fitting a data set “perfectly,” or even that it will “fit perfectly” any other data set for the same number of response and explanatory variables. From a statistical viewpoint, we believe that such general statements misinterpret the notion of “perfect fit.” They are in addition potentially misleading as suggestive of an SM being an adequate means of data description and explanation, which need not be the case in general. In our opinion, such statements of perfect fit and the resulting view among some empirical researchers that a saturated model cannot be disconfirmed by data, may have possibly stalled progress or even misdirected some areas in the educational, behavioral, and social disciplines where saturated models may have been used in theoretical developments.

What in effect such references to “perfect fit” of saturated models mean, is just the overall goodness of fit of the model to the data, that is, the fact that its chi-square goodness-of-fit value or RMSEA say attain their minimal possible value of 0. However, from the perspective of the main statistical modeling criterion mentioned earlier, which requires examination of fit at the raw (subject-level) data, it is not sufficient to consider only overall fit of an SM before its parameter estimates and standard errors may be interpreted and/or the model potentially recommended for further use. In fact, as demonstrated in the next section it is possible that an SM, although fitting perfectly overall, may not even be a good means of data description and explanation.

The reason for this apparent “paradox” (which, in fact, is not a paradox, as we argue here) is that overall fit is merely an omnibus statement that in the setting of interest in this article amounts to a statement of the model being capable of perfectly reproducing only the analyzed variable means, variances, and covariances. This fact per se has no implications however with regard to the model being (or not being) capable of reproducing the raw data to a satisfactory degree (e.g., Cook & Weisberg, 1982). More specifically, although an SM has perfect fit as far as the covariance/means matrix is concerned, it is still possible that (a) for some (group of) individuals the model does not reproduce the raw data well and/or that (b) the model has some important deficiencies or misspecifications, such as excluded relevant predictors or higher powers or interactions of predictors in it.

With these possibilities in mind—none of which are addressed by simply examining the overall fit of an SM under consideration—it is clear that such a model need not in general fit a given data set well or be an adequate means of its description (see next section for illustrative examples). In fact, while an SM fits data perfectly overall, that model may still be misspecified to an intolerable degree. Although the last statement may seem at first controversial, it is not as contentious as it may appear. Indeed, from a statistical viewpoint there is nothing in an SM that prevents it from not fitting well the raw data, or from being misspecified in the sense of having for instance omitted important (a) predictors, or (b) higher powers of explanatory variables in the model, and/or (c) interactions between (some of) these variables. The reason is that in the circumstances considered, the model is effectively fitted to the covariance/means matrix only, yet the latter is potentially a rather severe summarization of the raw data where specific aspects of subject-level data can get lost, in particular those related to (a), (b), or (c) (e.g., Rogosa & Willett, 1985).

Having established that a considered SM need not fit the analyzed raw data well in general, the next question is how such a situation can be diagnosed and possibly remedied. To this end, the ICRs provided by Equation (2) can be particularly helpful for addressing the possibility of a saturated path model being misspecified or not fitting well the available data, that is, the ICRs in Equation (2) can serve as model diagnostics. Specifically, these ICRs can be examined, after fitting the model to an available data set, with the view of searching for possible model misspecifications. This can be accomplished by assessing for instance the plots of the ICRs against (a) included or excluded explanatory variables in the model and (b) higher powers of included (excluded) predictors and their interactions. Similar to how these plots are utilized in the general linear model framework (e.g., Draper & Smith, 1998), in applications of path analysis in particular they can be especially helpful when evaluating the quality of a saturated path model as a means of data description and explanation. These ICR plots can provide suggestive information enabling an empirical researcher to sense possible model deficiencies or misspecifications, in particular omitted important predictors or functions thereof, as well as ways of resolving such misspecifications. We demonstrate in the next section the potential of the ICRs as such diagnostics of saturated path models.

Individual Case Residuals Can Sense Saturated Path Model Misspecifications and Contribute to Model Improvement

In the context of the preceding discussion emphasizing the relevance of also examining raw data fit, in particular for saturated path models, we employ in this section several numerical examples to demonstrate how the ICRs in Equation (2) can be used to sense possible model misspecifications and also ways of dealing with them and improving a considered saturated path model. To accomplish this aim, we use an initial simulated data set for n = 500 cases generated according to the following model (cf. Equations 1):

\begin{matrix} y_{1} = . 5 + x_{1} + 2 x_{2} + . 5 x_{3} + x_{4} + ε_{1}, \\ y_{2} = 1 + 2 x_{1} + x_{2} + 1.5 x_{3} + 1.5 x_{4} + ε_{2}, \end{matrix}

where the xs were zero-mean normal each with variance 1 and correlations of .3 among themselves, whereas ε₁ and ε₂ were normal zero-mean variates with variances 1 and 1.5, respectively, and covariance .4. (Details on the simulation process can be obtained from the authors on request.)

Fitting first to the resulting data set the misspecified saturated model ensuing from that defined in Equations (3) by omitting the predictor x₄ (and allowing for error covariance, as in the data generating model 3), we obtain “perfect” overall goodness-of-fit indexes: chi-square (χ²) = 0, d = 0, RMSEA = 0 (0, 0). (See first line of source code in the appendix; we employ the popular software Stata for this purpose because of the ease of subsequent estimation of the ICRs; StataCorp, 2011). We emphasize that the fitted path model, although saturated and with “perfect” overall fit, is in actual fact misspecified.

With the resulting parameter estimates, we readily furnish the ICRs (2) for each response, y₁ and y₂ (see following source code lines in the appendix). A plot of the ICRs for either outcome then against a predictor included in the model is found to exhibit the “no-pattern” appearance displayed in Figure 1.

As can be seen from Figure 1, this ICR scatterplot shows no discernible pattern, as could be expected since the predictor used in it was included in the fitted model. (For simplicity, we refer to such a point scatter as a “no-pattern” appearance in the remainder of the article.) However, a plot of the ICRs for either outcome, y₁ or y₂, against the omitted predictor, x₄, exhibits an easily discernible pattern (see Figure 2).

Figure 2 shows a clear linear relationship between the ICRs (for y₁, in this case) and the omitted predictor, x₄, suggesting consideration of linearly including this variable in a subsequently improved model. When this inclusion is carried out and the so-modified model fitted (which is not misspecified anymore, for this example), a plot of the ICRs for either outcome, y₁ or y₂, against any of the four predictors in that model exhibits the general “no-pattern” appearance displayed in Figure 3 (cf. Figure 1).

As a next example, we generate data following the same model (3) but with the last predictor now included also quadratically in the first equation:

\begin{matrix} y_{1} = . 5 + x_{1} + 2 x_{2} + . 5 x_{3} + x_{4} + . 7 x_{4}^{2} + ε_{1}, \\ y_{2} = 1 + 2 x_{1} + x_{2} + 1.5 x_{3} + 1.5 x_{4} + ε_{2}, \end{matrix}

Proceeding as above, we render the ICRs for the saturated model obtained from Equations (4) by omitting the quadratic term in the equation for its first outcome, y₁ (cf. the appendix; we dispense with presenting here its “perfect” overall goodness-of-fit indexes and, to avoid tautology, also for any fitted saturated model in the remainder of the article). As with the first example, we stress that the last fitted model is saturated and with “perfect” overall fit but in actual fact misspecified.

Plotting then the ICRs for y₂ against any of the four predictors, x₁ through x₄, yields the same generic “no-pattern” appearance like that evident in Figure 1, which is consistent with the fact that the equation for this outcome was not misspecified in the last fitted model. (For the sake of space, because of this scatterplot similarity, we dispense with presenting any of these four plots on a separate figure.)

Plotting next the ICRs for y₁ against x₁, x₂, or x₃, however, yields a different general pattern appearance, as in Figure 4.

Figure 4 suggests some violations of the fitted model (equation) for this outcome, y₁ (with its ICRs plotted against x₁). In particular, there are some large residuals in absolute value that are associated with moderately large predictor values. While there is no clear pattern in this deviation per se from the generic “no-pattern” appearance like that in Figure 1 say, given the ‘no-pattern’ finding in any plot of the ICRs for y₁ against x₁, x₂, or x₃ one may hypothesize some model misspecification with regard to a predictor not used in any of those three plots, that is, with respect to x₄. To probe this hypothesis, we plot now the ICRs for y₁ against x₄, which is presented in Figure 5.

As can be readily seen in Figure 5, the clear quadratic pattern in this plot suggests large residuals for extreme values on x₄, which is consistent with omission of the square of this predictor, as in the last fitted model. We include in the next modified saturated model also the square of the last predictor, x₄. Estimation then of the ICRs for either response as well as plotting them against any of the four predictors in the model, yields a scatter with the general “no-pattern” appearance like that in Figure 1. (For the sake of space, and again because of the scatterplot similarities, we dispense with presenting any of these four plots on a separate figure.)

As a final example, we generate data using the same initial model (3) but including now also the interaction of the last pair of predictors, specifically according to the following model:

\begin{matrix} y_{1} = . 5 + x_{1} + 2 x_{2} + . 5 x_{3} + x_{4} + . 8 x_{3} x_{4} + ε_{1}, \\ y_{2} = 1 + 2 x_{1} + x_{2} + 1.5 x_{3} + 1.5 x_{4} + ε_{2} . \end{matrix}

Proceeding as above, after fitting the saturated model (5) with omitted interaction term and furnishing the ICRs for its two responses, we plot these ICRs against each of the four predictors. (As with the above two examples, we underscore that the last fitted model is saturated and with “perfect” overall fit, while in fact being misspecified.) When plotting the ICRs of y₂ against any predictor, and the ICRs of y₁ against x₁ or x₂, we obtain a “no-pattern” appearance like that in Figure 1 say. (Because of this similarity, we dispense with presenting any of these plots on a separate figure.) When plotting the ICRs of y₁ against x₃ or x₄, however, the resulting scatter is presented in Figure 6.

Figure 6 for the plot of these ICRs against x₄ suggests some deviation from the “no-pattern” scatter like that in Figure 1 say. Specifically, there are quite a few comparatively large residuals in absolute value at such large values of x₄ and mostly considerably smaller residuals at average values for this predictor. Given that there was no pattern found in the scatter of the ICRs for y₁ against x₁ and x₂, as well as the result of no clear pattern in the plot of Figure 6 in relation to x₄—in terms of approximate functional relationship between residuals and this predictor (see above)—it would seem plausible to hypothesize that this finding of comparatively large ICRs could have to do with x₃. A simple way of probing this possibility is to include the interaction of x₃ and x₄ in a corresponding modification of the last fitted model. When this is done, and the resulting modified saturated model fitted to the data, any plot of an outcome’s ICRs against a predictor from the four used in the model will exhibit a general “no-pattern” appearance like that in Figure 1 say. (For the sake of space, because of this similarity we dispense with presenting any of these plots.)

These three examples demonstrate the potential of the advocated ICRs in the present article as useful saturated model diagnostics. The examples also show clearly that an SM need not even fit a data set well (let alone “perfectly”) when using the essential criterion of raw data fit frequently overlooked in current SEM and particularly path analysis applications, while it should not be so. In addition, the plots demonstrate how use of the ICRs can suggest possible ways of improving if need be a saturated path model under consideration.

Conclusion

This article was concerned with the relationship between saturated models and their data fit. In contrast to the large majority of literature on SEM applications, an alternative main criterion of model fit was emphasized throughout, namely fit to the raw data (e.g., Bollen & Arminger, 1991; see also Coffman & Millsap, 2006; Raykov & Penev, 2001, 2002, in press; Raykov & Zajacova, 2012). Using this main principle of statistical model fit assessment, which is widely followed in applications of the general linear model (e.g., Belsley, Kuh, & Welsch, 1980), it was also demonstrated for SEM applications that saturated models need not even fit well a given data set. Specifically, employing the individual case residuals it was shown how saturated path-analysis model misspecifications can be detected and ways possibly found in which these models can be improved so as to fit the raw data better (satisfactorily). This note also contributed to discrediting the myth that a saturated model cannot be improved any further, a myth that to date seems to have found a considerable segment of followers among empirical educational, behavioral, and social scientists.

Our intended message contrasts with a multitude of statements found in the SEM and particularly path-analysis based literature, implying that a saturated model fits always perfectly the data in an available sample and even that it will fit so any other data set for the same outcomes and explanatory variables. While these statements (presumably) only mean to refer to overall fit, we voiced serious concerns about the current lack of attention in empirical educational, social, biomedical, business, and marketing research to the extent to which used structural equation models, and especially saturated path-analysis models, actually fit the raw data. Adhering to a major and essential statistical modeling principle that proclaims adequate fit only for models satisfactorily fitting the raw data (e.g., Cook & Weisberg, 1982), we also discussed a readily and widely applicable means with which one could assess raw data fit of saturated path-analysis models. This means is provided by the individual case residuals that hold substantial promise to aid empirical scientists in their search for potential model misspecifications as well as avenues of possibly fixing them.

Several limitations of the fit assessment method used in this article are worth noting here. First, our note was concerned with saturated models having zero degrees of freedom and assumed error-free predictors, which represent the typical path-analysis models used in most empirical research areas. However, we submit that the potential of the ICRs remains strong also for path analysis models having positive degrees of freedom (and error-free predictors), as well as for more general structural equation models including fallible predictors/construct indicators (e.g., Bollen & Arminger, 1991; see also Raykov & Penev, 2001, 2002; Raykov & Zajacova, 2012). The examination of the former possibility is beyond the scope of this note, whereas the last cited three sources provide evidence for the potential of appropriately extended ICRs with more general SEM models containing latent variables. We would like to mention, however, that in our view the latter ICRs might turn out to be less informative in cases where latent response variables are associated with low communalities.

Second, we effectively assumed the availability of a large sample as well as (conditional) normality throughout this commentary, since for our purposes we used instrumentally the ML as a model fitting method that as well known obtains its optimality features with large samples (e.g., Casella & Berger, 2002). We encourage future research into possible guidelines for determining sample size to ensure trustworthy use of the ICRs as model diagnostics, as well as into the degree of possible robustness of the ICRs as such means in cases of violations of normality. In the interim, we would conjecture that use of robust ML would likely lead to largely trustworthy ICRs for these purposes, especially with large samples and factor communalities and when violations of normality are not substantial (such as, for instance, piling at scale end for response variables, highly discrete outcome measures, and/or considerable clustering effect on some of them).

Third, there seems to be an inevitable subjective element involved in the interpretation of ICR plots, like the ones in this note (Figures 1 -6). This element is more likely to be present when searching for a “pattern” in them and also when assessing if they show a “no-pattern” appearance. We would however submit that this subjective element need not have a significant impact on pattern interpretation (or lack-of-pattern interpretation) when that interpretation is (a) identical across several researchers applying the saturated model diagnostics advocated in the article and (b) consistent with the available theory and prior research in a given substantive area. Similarly, in cases where no close to unambiguous ICR plot interpretation appears possible, prior theory may be used or a hypothesis advanced with regard to a predictor or model term in question (higher power of or interaction of predictors), which is consistent with a possible interpretation and to be examined in the given as well as in a replication study (see illustration section).

Fourth, we should like to emphasize that no part of this article should be interpreted as implying that (a) the ICRs will always find any saturated path (or more general structural equation) model misspecification(s) if present in a given empirical setting for a model under consideration; or (b) when a misspecification(s) is found, the ICRs will always suggest a way of improving the model so as to fix/remove the misspecification(s); or (c) all possibly informative ICR plots were used in this note. Rather, as mentioned above, our intention was only to demonstrate the potential of the ICRs for finding considerable saturated (path analysis) model misspecifications and possibly suggesting ways of improving models by resolving these misspecification issues.

Last but not least, given that capitalization on chance fluctuations in a given sample may occur when using the ICRs in this note, a replication study would be recommended to carry out before more trust could be placed in saturated path model modifications arrived at in this way.

In conclusion, this commentary aimed to enhance the awareness among empirical educational, behavioral, and social scientists that contrary to widespread belief in its perfect fit, a saturated model—and in particular a saturated path-analysis model—need not be free from important and potentially consequential misspecifications. The article also discussed a readily and widely applicable means in the ICRs, which can help scholars detect saturated model misspecifications and possibly improve such models that are often used in SEM and path analysis applications.

Footnotes

Appendix

Acknowledgements

We are grateful to S. Penev and K. MacDonald for valuable discussions on model fitting, case residuals, and software implementation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Belsley

Kuh

Welsch

(1980). Regression diagnostics. New York, NY: Wiley.

Bentler

P. M.

(2004). EQS structural equation program manual. Encino, CA: Multivariate Software.

Bollen

K. A.

(1989). Structural equations with latent variables. New York, NY: Wiley.

Bollen

K. A.

Aminger

(1991). Observational residuals in factor analysis and structural equation models. Sociological Methodology, 21, 235-262.

Casella

Berger

R. L.

(2002). Statistical inference. Monterey, CA: Wadsworth.

Coffman

Millsap

R. E.

(2006). Evaluating latent growth curve models using individual fit statistics. Structural Equation Modeling, 13, 1-27.

Cook

D. R.

Weisberg

(1982). Residuals and influence in regression. New York, NY: Chapman & Hall.

Draper

Smith

(1998). Applied regression analysis. New York, NY: Wiley.

Raykov

Marcoulides

G. A.

Patelis

(2013). Saturated versus identified models: A note on their distinction. Educational and Psychological Measurement, 73, 162-168.

10.

Raykov

Penev

(2001). The problem of equivalent structural equation models: An individual residual perspective. In Marcoulides

G. A.

Schumacker

R. E.

(Eds.), New developments and techniques in structural equation modeling (pp. 297-322). Mahwah, NJ: Lawrence Erlbaum.

11.

Raykov

Penev

(2002). Exploring structural equation model misspecifications via latent individual residuals. In Marcoulides

G. A.

Moustaki

(Eds.), Latent variable and latent structural models (pp. 121-134). Mahwah, NJ: Lawrence Erlbaum.

12.

Raykov

Penev

(in press). Latent growth curve model selection: The potential of individual case residuals. Structural Equation Modeling.

13.

Raykov

Zajacova

(2012). On latent change model choice in longitudinal studies. Structural Equation Modeling, 19, 580-592.

14.

Rogosa

D. R.

Willett

J. B.

(1985). Satisfying a simplex structure is simpler than it should be. Journal of Educational Statistics, 10, 99-107.

15.

StataCorp. (2011). Stata: Release 12. Statistical software. College Station, TX: Author.