Abstract
How to use this document
This Campbell methods policy note on synthesizing partial effect sizes sets out current Campbell Collaboration policy on the synthesis of partial effect sizes (i.e. adjusted effect sizes) in Campbell systematic reviews of intervention effects. It is intended as a point of reference for Campbell authors and editors in the design and production phases of Campbell reviews of intervention effects, to ensure that protocols and full reviews are prepared in line with policy. To provide context for the current policy, the note is prefaced with a brief introduction to the topic and appended with a list of additional resources. It is not intended to provide comprehensive guidance or a tutorial on how to apply methods in Campbell or other systematic reviews (see below).
Campbell authors and editors should consult
Campbell authors and editors should also refer to
Campbell also publishes Campbell Methods Guides, that provide detailed guidance for authors on how to apply specific methods in the production of Campbell systematic reviews. Campbell Methods Guides supplement general guidelines for producing Campbell systematic reviews, as described in Campbell Collaboration systematic reviews: policies and guidelines. Campbell Methods Guides do not currently cover the topic of synthesizing partial effect sizes.
Introduction
Bivariate effect sizes are effect sizes that represent an association between two variables, such as the standardized difference in means between two groups that has not been adjusted for other extraneous variables. In contrast, partial effect sizes refer to those effect sizes that have been statistically adjusted for one or more variables (e.g., ANCOVA models). Perhaps most relevant to Campbell Collaboration reviews are partial effect sizes derived from regression coefficients indexing the effect of an “intervention” (or treatment, policy, program, or practice). Partial effect sizes are often taken from regression models that include one or more control variables, especially in quasi-experiments. This is a particularly important issue for research syntheses, given that many primary studies attempting to make causal statements regarding the effect of an intervention rely on regression techniques. Indeed, both experimental and non-experimental primary studies can report partial effects. Studies with experimental designs often use regression adjustments to control for imbalance between groups after randomization. Non-experimental studies, on the other hand, often use regression adjustments to try to reduce selection bias by controlling for a range of possible confounding variables.
The main goal of a research design is to control variance by maximizing systematic explainable variance, minimizing error variance, and controlling for extraneous systematic variance (e.g., Kerlinger & Lee, 2000). Controlling for extraneous variables allows researchers to isolate the influence of those independent extraneous variables. Researchers control for the effects of these variables using both design and statistical controls. For instance, a researcher might control extraneous variables in a design by blocking or use of range restriction. In contrast, a researcher may decide to adjust for extraneous variables by building models with those variables as predictors or covariates. When the influence of extraneous variables is controlled by design in such a manner, those extraneous variables may not be included in the statistical analysis used to estimate intervention effects. Indeed, in such cases, even a bivariate effect size is adjusted for the variables that have been controlled for in the design. On the other hand, when the influence of extraneous variables is adjusted statistically, the fitted model partials out the effects of those variables. In these cases, the partial effect size is used to reflect the association between the variables of interest, given the adjusted variables.
Many research syntheses and meta-analyses use bivariate effect sizes to index the effects of interventions. Such syntheses of bivariate effects do not incorporate within-study statistical adjustments from other confounding variables. In contrast, partial effect sizes represent the association between two variables, adjusting for the effects of other extraneous variables. Thus, partial effect sizes and bivariate effect sizes estimate different parameters and should not be treated as if they are the same.
Given that partial and bivariate effect sizes estimate different parameters, the critical issue is how to handle them in a meta-analysis where some studies only provide partial effects and others only provide bivariate effects. There is currently no clear consensus among statisticians and meta-analysts about how best to approach this issue.
The most conservative approach would be to plan to conduct and report two separate meta-analyses: one that synthesizes the bivariate effect sizes and a second that synthesizes the partial effect sizes. The most liberal approach (and one that is not recommended) would be to plan to incorporate both bivariate and partial effects into the same meta-analysis, ignoring the difference between the bivariate and partial effects. Either approach should be pre-specified in the protocol.
A third option is to incorporate both bivariate and partial effects into the same meta-analysis, but use pre-specified sensitivity analyses and/or moderator analyses to model the differences between the bivariate and partial effects. Taking into account differences in primary studies is one of the reasons for coding for moderators in any meta-analysis. When synthesizing partial effect sizes moderator analysis may be more informative than analysis of overall effects, given that partial effects may arise from models with different number and/or type of predictors (Aloe, 2015).
Campbell policy
This policy on the synthesis of partial effect sizes in Campbell systematic reviews of intervention effects was adopted by the Campbell Collaboration at a Steering Group meeting. Campbell methods policy on this issue is that: Protocol inclusion-exclusion criteria should state explicitly that partial and/or bivariate effect sizes are being included in the review. Formulas used to compute partial and/or bivariate effect sizes and their standard errors should be provided in an appendix of the full review. A rationale should be provided in the protocol for the inclusion of partial effect sizes in any meta-analysis. The overall approach to be adopted for synthesizing bivariate and partial sizes (see 5, 6, 7 and 9) should be fully described in the protocol. Bivariate and partial effect sizes may be synthesized in separate meta-analyses. Bivariate and partial effect sizes may be synthesized in one meta-analysis if the synthesis includes mainly bivariate effects and demonstrates (via a pre-specified sensitivity analysis) that results are not altered by the inclusion of a small number of partial effect sizes. Bivariate and partial effect sizes may be synthesized in one meta-analysis if the synthesis demonstrates that bivariate effects are comparable to partial effects conceptually and statistically (e.g., overall weighted mean and amount of between-studies variability). Bivariate and partial effect sizes may never be synthesized in the same meta-analysis if the difference between the bivariate and partial effects is ignored.
Given that partial effect sizes often arise from different regression models across studies, when analyzing a collection of partial effect sizes, the reviewer must conduct a meta-regression including predictor variables (covariates) that reflect differences in model complexity and structure. The goal of such analysis is to capture differences among partial effect sizes.