Mediating Analysis Approaches

What Is Mediation Research?

General Concept of Mediation Research

Mediation and mediating approaches refer to those techniques examining three steps of relations between the independent variable(s) (predictor) and dependent variable(s) (criterion), between the independent variable(s) and mediating variable(s), and between mediating variable(s) and dependent variable(s) to identify how the mediating variable(s) are intervening in the relationships between independent variable(s) and dependent variable(s) (Frazier, Tix, & Barron, 2004). Mediation analysis provides more complicated information regarding the influential relation between independent and dependent variables using the mediating variable within the equation.

In this context, the mediating variable (also known as an intervening variable) is defined as the variable that explains how the effects could occur by accounting for the relations between the designated independent and dependent variables (James & Brett, 1984; Judd & Kenny, 1981). In its statistical formula, the total effect of the mediating model is the sum of the direct and mediated effects and the statistical significance of the mediation model could be examined by the Sobel test and bootstrapping approach, which are introduced in the results section of this research (Kenny, Kashy, & Bolger, 1998; MacKinnon, Warsi, & Dwyer, 1995).

In Figure 1, the comparison between direct and indirect effects was visualized to clarify the relations among the independent, dependent, and mediating variables. As shown in Figure 1, the total path coefficient from the independent variables to the dependent variables (Path C) could be explained by the combination of the direct effect from the independent variable (Path C) and indirect effect of the mediating variable (Paths A and B via mediating variable M) to the dependent variable (Frazier et al., 2004).

OPEN IN VIEWER

Figure 1.

Example of mediating variable and mediating research design.

Mediation Research Versus Moderation Research

In social science research, understanding the differences between mediation and moderation research approaches is critical for designing a research study appropriately as well as analyzing and interpreting study results (Frazier et al., 2004). In mediation modeling, the path coefficient estimate of the relation between independent and dependent variables could be determined by the role of the mediating variable(s). Different from the mediation model, in moderation modeling, the magnitude of the relation between independent and dependent variables could be determined by the types of sample characteristics within the moderating variable (e.g., gender, type of education, level of education, etc.; Frazier et al., 2004; Mason, Tu, & Cauce, 1996). In addition, the moderating effect is an interpretation of the interaction (or contingent) effect of any one moderating variable that explains the relationship between the independent and dependent variables (James & Brett, 1984; Mason et al., 1996). The differences between mediation and moderation can be illustrated using specific variable examples, as described in Figure 2.

OPEN IN VIEWER

Figure 2.

Examples of mediation and moderation research variables.

As shown in the first example in Figure 2A, the level of self-efficacy of the employees could mediate the relation between on-line training and the task performance level in the workplace, which means the higher the self-efficacy level of the employees, the better their performance level. The second example in Figure 2B illustrates the moderating effect of gender in the relation between on-line training and task performance. This design could be developed based on the assumption that the on-line training program would interact with gender when predicting the levels of employee task performance.

Research Design and Approaches: Context for the Field

For a survey of mediation approaches in HRD publications, we employed a critical literature review technique following Fraenkel and Wallen’s (2006) six-step model: define problems, look at secondary sources, select general references, formulate search terms, search the primary sources, and summarize the key points. In addition, the within-study literature approach was used for finding research sources and analyzing selected research sources (Onwuegbuzie, Leech, & Collins, 2012). The rationale to use this critical literature review technique was twofold. First, one of the primary intentions of this study was to test and establish a methodological approach to conduct a meta-analytic study on mediation. We believe our approach will identify global findings contributing to theory building in mediation studies. Second, we expected that our critical review of existing literature on mediation studies would reveal a meaningful analysis about the different types and cases of mediation variables tested within the HRD discipline. Findings such as these can provide useful guidelines for HRD practitioners to understand mission-critical organizational and task-related variables and factors influencing their employees’ work and performance.

The literature review targeted the four existing AHRD journals: ADHR, HRDI, HRDQ, and the Human Resource Development Review (HRDR). However, to focus mainly on theory-building approaches, HRDR was excluded from the literature review due to the nature and aims of the journal.

As per our research purpose, the following search keywords were used for finding relevant literatures: mediation and mediating. Furthermore, all articles were reviewed individually by the two authors of the current research to make sure all mediating-related articles were included in the sample pool.

A total of 1,066 refereed manuscripts published in the aforementioned three journals between 2000 and 2014 were reviewed and cross-checked using the keyword searching and hands-on reviewing process. In this process, book reviews, forums, invited reactions, and perspective articles were excluded. As a result, the researchers found that 84 articles out of a total of 1,066 published articles (approximately 8% of total research articles) in the three journals used some type of mediation approach for analyzing data. All these articles were used for the analyses in the current research. More detailed information is presented in Table 1.

OPEN IN VIEWER

Table 1.

Yearly Counts of Mediation Studies in HRDQ, HRDI, and ADHR.

	HRDQ							HRDI							ADHR
			Method							Method							Method
Year	All	MS a	SEM	REG	BTS	HLM	SOB	All	MS a	SEM	REG	BTS	HLM	SOB	All	MS a	SEM	All/yearly	MS a /yearly
2000	14	3	1	2				34							22			70	3
2001	19	4	1	3				29							19			67	4
2002	22	1	1					23							29			74	1
2003	20	2	1	1				29	2	1	1				29			78	4
2004	19	1	1					25	2		2				33			77	3
2005	20	2	1	1				23							34			77	2
2006	16	5	2	1				22	1		1				34			72	6
2007	20	5	2	1				20							29			69	5
2008	13	3	1	1		1		33	1		1				45			91	4
2009	18	6	1	5				24	2	1	1				49			91	8
2010	16	8	4	4				25	2		2				41			82	10
2011	18	9	5	3	1			27	2		2				28			73	11
2012	15	6	3	2	1			25							26	1	1	66	7
2013	16	8	5	1	2		1 b	26	4	4 c		2		1	23			65	12
2014	4	3	1	2 d	2 e			4	1	1					6			14	4
Total	250	66	30	27	6	1	1	369	17	7	10	2		1	447	1	1	1,066	84

Note.HRDQ = Human Resource Development Quarterly; HRDI = Human Resource Development International; ADHR = Advances in Developing Human Resource; SEM = structural equation modeling; REG = regression; BTS = bootstrapping; HLM = hierarchical linear modeling; SOB = Sobel test.

a

Frequency of mediation studies.

b

One study used SEM and Sobel test.

c

One study used SEM/Sobeltest/bootstrapping and one study used SEM/bootstrapping.

d

One study used bootstrapping/regression.

e

One study used SEM/bootstrapping.

Results

From our review of the 84 articles on mediation studies identified from the total 1,066 published articles in the three journals, HRDQ included 66 articles, HRDI included 17 articles, and ADHR included 1 article. More detailed information about the frequencies of those mediation studies by journal is provided in Table 1. From our review, the regression analysis approach (n = 41; including multiple regression, stepwise regression, and so on) was found to be the most frequently used technique, followed by structural equation modeling (SEM; n = 38), bootstrapping (n = 7), hierarchical linear modeling (HLM; n = 2), and the Sobel test (n = 2). Although any approach could be better than the other, we should recognize the strengths and weaknesses of each approach for choosing the most suitable method based on the research goal and rationale.

In addition, Figure 3 describes the research trends of mediation research in the HRD discipline. In this figure, the articles published in 2014 were excluded as only the issues for the years 2000-2013 were printed out for HRDQ and HRDI at the time of the research, limiting the comparison between the years. As presented in Figure 3, the number of mediation studies in the HRD discipline has gradually and continually increased since 2000. Based on the comparison of the number of mediation studies in 2000 with that in 2013, we found a 400% increase (from 3 articles in 2000 to 12 articles in 2013). The results indicate that mediation research is gaining more attention in the HRD discipline, and mediation research is becoming more popular among HRD researchers.

OPEN IN VIEWER

Figure 3.

Trend of mediation studies in the HRDQ, HRDI, and ADHR journals.

Based on the results, we examined each approach for mediation model analysis. In addition, we analyzed the strengths and weaknesses of each technique and in the following section suggest appropriate techniques for future HRD research.

First, through the in-depth review of each article in which regression analysis was employed as the primary technique for the mediation analysis, we found that the r² comparison and r² pattern change were the most commonly used approaches for examining mediating effect. Although the regression approach is recognized as one of the most common approaches (Baron & Kenny, 1986; Kenny et al., 1998), we found that many of the sample studies in the current research have failed to examine the significance and magnitude of the mediating variable effect estimate. Most of the studies using regression analysis generally focused on the mediating relations but not on the magnitude of the mediating variable effect size. In addition, although the four-step approach (Kenny et al., 1998) should be used to explain most of the indirect and direct effect estimates among the variables (MacKinnon et al., 2002; Shrout & Bolger, 2002), many studies simply compared the r² change pattern. This approach is not considered sufficient in explaining the relation between an independent variable and a dependent variable within the context of a mediating relation as the interactive relations among the other variables should be considered to measure the mediating effect (Frazier et al., 2004).

Beyond the r² change comparison approach, to identify the significance of the mediated effect, the standard error of each path and its z score should be considered based on the p value of .05 to better examine the significance of the mediated effect (Frazier et al., 2004; Kenny et al., 1998). In this procedure, the significance between the effect size of Path C and Path C* (in Figure 1) must be assessed using the path coefficient estimates of Paths A and B (in Figure 1). Statistically, the z score of the mediated effect of the mediating variable can be easily calculated when the path coefficient estimates of Paths A and B are divided by the standard error term. Furthermore, the z score (1.96) should be checked with p value levels (p< .05) to determine the significance of any potentially mediating effect (Frazier et al., 2004; MacKinnon & Dwyer, 1993).

The SEM approach was the second most frequently used technique as it has several advantages including controlling measurement error, suggesting overall model-data fit, and flexibility of number of variables (Hoyle & Smith, 1994; Judd & Kenny, 1981). Although SEM analysis could be considered a more informative and preferred approach for examining the mediating effect, SEM requires a greater sample size than the regression approach does, which could severely limit the broader use of SEM analysis in HRD studies (Hair et al., 2010; Kline, 2010).

Nonetheless, in using SEM, model comparison (full research model and controlled model) and chi-square change comparison were the most popular approaches to exploring mediation paths. We note, however, that several prior requirement assumptions must be met before proceeding with the SEM model comparison: (a) individual relations among the variables need to be significantly correlated (X/Y, X/M, and M/Y in Figure 2B), (b) model fit indices need to be examined between the full model and the controlled model (the direct Path C is fixed to zero), and (c) chi-square differences need to be compared between the full model and the controlled model (Hair et al., 2010). In interpretation, when the chi-square difference between the two models is less than |3.84|, meaning that the t value is less than |1.96| and the direct Path C (in Figure 1) is non-significant, the mediating variable M (in Figure 1) is playing a full mediating role (Kaplan, 2000; Kline, 2010).

More importantly, several basic assumptions should be assessed before applying and interpreting SEM analysis in mediational testing. One of the basic but essential statistical assumptions for running SEM for mediational modeling is the examination of multivariate normality. This assumption is receiving increased attention in more advanced journals to retain the quality, reliability, and validity of the research (Byrne, 1998; Kaplan, 2000). In exploring multivariate normality, the maximum likelihood (ML) approach should be used for running SEM when the observed variables are multivariate and normally distributed (Hair et al., 2010). Several statistical packages (e.g., LISREL, AMOS, and Mplus) currently provide the univariate normality information using the skewness and kurtosis distribution, but not the information of multivariate normality (Hair et al., 2010). Theoretically, we can assume that univariate normality of each variable could result in multivariate normality, but this is not true in realistic situations due to the multi-dimensional interactions among the observed variables (Jobson, 1992; Kline, 2010). One of the most common methods to detect multivariate normality is the Mahalanobis distance approach through detecting multivariate outliers from the multivariate sample. In this approach, the values of the chi-square plot and beta probability plot of squared Mahalanobis distance can be used (Schinka, Velicer, & Weiner, 2003). In more general terms, if the skewness values are greater than |1| and/or the kurtosis value is greater than |3|, then it can be concluded that the data set is a non-normally distributed sample. More recently, Kline (2010) suggested that a more rigorous standard to detect outlying variables, such as skewness values, needs to be less than |3| and kurtosis values need to be less than |10|.

Regarding other approaches, the Sobel test, bootstrapping, and HLM were used 11 times in nine articles. The Sobel test is one of the more common approaches to identifying a mediation effect. The Sobel test is a specialized type of t test based on the determination of whether the effect-size reduction of independent variables, when the mediating variable is included in the model, is significant and therefore whether the mediation effect is statistically significant (Sobel, 1982, 2006). Essentially, the Sobel test uses the t value table and z score to determine the significance of the mediating effect based on the comparison of the estimated standard error of each variable in the model (Sobel, 1982). Similar to SEM, one of the limitations of using the Sobel test is sample size. To utilize the Sobel method, a large sample size is required (i.e., more than 200 cases). Otherwise, the true p value of the t distribution cannot be established with accuracy (Hair et al., 2010).

As one of the alternative methods, bootstrapping is becoming increasingly popular in the social science literature (Shrout & Bolger, 2002). Principally, bootstrapping can be used with a non-normal sample distribution based on the resampling method (Politis & Romano, 1994). Based on the resampling method, the bootstrapping approach can typically produce a sample data set from 1,000 to 10,000 larger than the actual collected data set (Politis & Romano, 1994). Statistically, the bootstrapping method is a straightforward approach to examine a mediating effect using the standard errors and confidence interval estimates (Shrout & Bolger, 2002). One of the advantages, as well as limiting disadvantages, is the fact that the computed larger sample size is created based on the original sample. This is due to the fact that the bootstrapped data do not guarantee a general finite-sample, although more reliable information could be produced based on a larger sample set (Shrout & Bolger, 2002).

Finally, two articles used the HLM approach based on the examinations of the model comparison between two equated models. In these cases, the two studies focused on the order of modeling grounded in theory similar to the approach utilizing multiple regression. However, the order of the modeling is critically important in HLM as different orders of the model can result in different results. In developing such a model, theoretical rationales and statistical justifications should be stressed to keep the unit of analysis on the variable of interest. For more critical examination of the mediating effect in this analysis, Monte Carlo simulation should be considered based on the over-estimation and under-estimation of the model and variable relations.

Recently, researchers have recognized that common method variance (CMV) is a potential and critical problem in social science research (Podsakoff, MacKenzie, Lee, & Podsakoff, 2003; Reio, 2010). The critical issues around CMV are related to actual collected data and the resulting measurement error, which threatens the structural validity of any research model as well as the reliability of results (Podsakoff et al., 2003). Among all of the studies reviewed in our research, only 23 articles mentioned CMV, and among them, 19 articles handled CMV statistically, and 4 articles mentioned CMV in the limitations section. A mere 22% of the reviewed articles from our sample addressed the CMV issue both statistically and methodologically.

The CMV challenge should be handled through two approaches: (a) pre-treatment (e.g., sampling time interval, use of multiple data sources, use of objective data set, etc.) and (b) post-treatment (e.g., single-factor analysis method, unmeasured latent factor analysis method, etc.). Among the articles that addressed CMV, only 5 used a pre-treatment approach (sampling time interval = 3, use of multiple data sources = 1 and use of objective data = 1) and 18 articles used post-treatment approaches including single-factor analysis method (n = 14, 74%), unmeasured latent factor analysis method (n = 2), and another correlation coefficient analysis approach (n = 2).

In addition, although the post-treatment approach could be acceptable, ultimately pre-treatment approaches should be considered to prevent, or at least minimize, the CMV issue during the beginning stages of design. As a final note to our article sample, Harman’s single-factor method was used frequently (n = 14); this approach has been recognized as one of the most generous approaches, as opposed to the unmeasured latent factor analysis approach.

Implications for Stakeholders

The primary purpose of our research was to identify the general trends of mediation research in the HRD discipline and to identify what analytical approaches have been used in HRD research—especially in those articles published in three AHRD journals between 2000 and 2014. We believe our study findings provide valuable implications that inform existing researchers of HRD in addition to those who will utilize mediation analysis in future theory-building efforts. Among several implications, first, we identified the encouraging phenomenon that more researchers have tried to adopt mediation analytical approaches in their published work, which has anecdotally resulted in enhanced research quality. This is considered a meaningful trend that contributes to solid theory building in HRD through a rigorous application of methodologically sound research design approaches. For practical purposes and future theory-building efforts, it is important to use more advanced mediation analysis methods such as SEM and bootstrapping as compared with simple regression-based analysis.

Utilizing such advanced methods may allow researchers better opportunities to identify the strength in relationships among study variables more accurately resulting in sound theory building (Preacher & Hayes, 2008). Also, we believe that practicing advanced methods will enhance the overall quality of HRD research studies by raising the bar in the quality of analytical approaches and put the AHRD community in a competitive position compared with other fields such as management, psychology, and organizational studies. Eventually, this will facilitate more collaborative research efforts and activities across disciplines and contribute to a wider community of research composed of the HRD field and other related disciplines.

With continued use of more rudimentary approaches, research results could continue to hinder theory-building efforts. For example, while it is highly advised to use multiple analytical procedures in the analysis of mediating relations among research variables, few researchers adopted this approach in their work (e.g., only 4 out of the 84 studies used multiple analytical procedures such as combination of SEM with Sobel test or SEM with bootstrapping). It is a best practice to use multiple analytical procedures if a research study is trying to address multiple or complicated research frameworks (e.g., multilevel mediation models or inclusion of multiple mediators). Moreover, it is common in HRD theory-building studies that many researchers attempt to measure research data clustered at several levels such as departments, companies, and intercultural settings. In this case, SEM and HLM are appropriate multi-analytical procedures permitting a more reliable analysis of the study data (Hofmann & Gavin, 1998), but as our analysis has indicated, few researchers have actually utilized such methods. In the case of using multiple mediators, the multiple-mediator analytical model allows for a more accurate assessment of the mediating effects occurring between independent, multiple mediators, and dependent variables (MacKinnon, 2008).

Another important implication for consideration is the overarching CMV issue. The problem of not fully addressing CMV is that it can skew results, and thus a researcher cannot infer true causality because relations may be produced by inflated or deflated correlations among the research variables. The resulting variance may be attributed to errors in measurement method rather than the actual constructs of interest (Podsakoff & Organ, 1986). Within the HRD literature, the CMV issue has been considered serious as it may hamper research intention pursuing reliable and valid HRD theory-building efforts (Reio, 2010). As a discipline-wide solution for this issue, we would recommend that the editorial boards and reviewers of AHRD journals emphasize the importance of addressing CMV through the manuscript review processes, through symposiums for publishing in AHRD journals at conferences, and perhaps through recommending graduate HRD programs that include this matter in their graduate curricula of research methodology. Preferably, if possible, they should advice future contributing researchers to use pre-treatment methods to minimize the CMV issue rather than addressing it through post-treatment methods.

From a practical viewpoint, the value and utility of mediation studies are high as the results of such studies provide an empirical stepping stone to improving existing practices of HRD research. Whereas most HRD research studies are not single faceted and it is hard for researchers to control multiple variables in a single study, the use of mediation analysis may turn out to be extremely helpful. This approach allows researchers not only to identify the overall relations among research variables while managing multiple variables but also to reveal the causal relations among them. This can help pinpoint key influential factors of HRD issues within organizational settings. The use of mediation analysis then helps researchers investigate various important relational maps (direct and indirect) among work environment, people, and performance-related variables. The findings from this type of multi-faceted research will become important sources for conducting future research looking to investigate complicated inter-relational frameworks. For HRD practitioners, utilizing this kind of advanced research design can help devise appropriate interventions and solutions that address various key learning and performance issues in the workplace (e.g., use of research findings for training needs assessment, improving training transfer effectiveness through controlling influencing factors and causal relations, etc.).

In pursuing quality mediation studies that contribute to sound theory building in HRD, temporal issues have become an important factor as a prerequisite to planning a good mediation research study (Preacher & Hayes, 2008). In our review of the journal articles involving mediation analysis, few examples were found to indicate this consideration in explaining the research design. In designing a thorough mediation study, hypothesized temporal relationships between the independent, mediation, and dependent variables must be clearly defined to specify the temporal precedence among them. When this consideration is taken care of, the researcher(s) can decide the appropriate time lag between intended measures, and the research design may allow enough time for the intended full effect between the independent, mediation, and dependent variables to occur. We believe the research findings obtained through this rigorous research design may deserve future studies or provide a meaningful and significant contribution toward sound theory building within the HRD discipline.

In a nod to our own limitations, this study is not without inherent limitations. First, we included only those articles published in AHRD journals. This may have created a sampling bias that does not allow us to generalize the research findings beyond our field. Second, our analytical approach is purely based on a literature review (although we followed stringent procedures). For more meaningful findings, empirical studies employing meta-analytical approaches might be recommended for inclusion in future research.