Network meta-analysis

Introduction

This document provides an overview of the Campbell Collaboration policy regarding the use of network meta-analysis methods in systematic reviews of intervention effects. In addition, this policy briefing note aims to provide the reader with an understanding of what network meta-analysis is, when network meta-analysis might be useful, and the core concepts of the method. References to useful resources, including software, are also provided. This policy-briefing note is not a tutorial on how to conduct a network meta-analysis and does not provide an exhaustive treatment of all aspects of the method.

Network meta-analysis

Network meta-analysis is an extension of standard meta-analysis methods to the synthesis of two or more interventions (defined broadly here to include interventions, treatments, policies, programs, or practices). The goal of network meta-analysis is to take advantage of studies that compare these interventions with a standard comparator condition (such as a placebo or other control condition) as well as studies that compare these interventions with each other. Thus, this approach makes use of all available comparisons within a network of related studies addressing a common condition on a common outcome.

Network meta-analysis is well suited for comparing the effectiveness of multiple drugs for a common condition. For example, Cipriani, et al. (2009) compared the effectiveness of 12 new generation antidepressants for treating depression. The goal was to estimate which drug was most effective. This review included 117 randomized controlled trials. These trials were a mix of studies that compared one or more of the 12 drugs to a placebo or compared two or more drugs with each other. Thus, these 117 studies provided a network of comparisons among the drugs. Network meta-analysis can be applied to social interventions as well but it is important that the interventions are for a common problem (i.e., addressing a common population) and the studies are examining the same outcome construct.

To illustrate the concepts of network meta-analysis, we will use a simple example that only includes three interventions for adolescent drug use: a group based cognitive-behavioral program (labeled A), a psychoeducational program (B), and random drug testing (C). Suppose that a systematic literature search yielded studies comparing all three interventions to a no treatment control condition (labeled D). Additionally, the search identified studies that compared the cognitive-behavioral program (A) with drug testing (C). Note that in this example, there are no studies comparing A with B or B with C. These comparisons are shown graphically in Figure 1 below. The goal of network meta-analysis is to assess the relative effectiveness of these three interventions using all available information. In this figure, the nodes represent the treatments, including the control condition D. The lines (called edges in network meta-analysis) connecting the nodes reflect the presence of studies comparing the two nodes.

OPEN IN VIEWER

1

Example of a simple network

An important concept in network meta-analysis is the distinction between direct and indirect effects. Any two nodes that are connected by a line can be estimated directly, that is, there are studies that provide effect sizes comparing the two nodes. As the name implies, indirect effects are not estimated directly but rather based on the relative effects of two nodes compared to a common third node. In a network plot, these are typically represented as dashed lines. Figure 2 illustrates both the direct and indirect effects for our example.

OPEN IN VIEWER

2

Example with both direct and indirect effects

Indirect effects are estimated by comparing the difference in the effectiveness of related nodes. In our example, we can estimate the indirect effect between A and B by comparing the relative effects of A versus D and B versus D. This is illustrated in Figure 3. In this example, the direct evidence produced a mean effect size of 0.40 between nodes A and D. Similarly, the direct evidence produced a mean effect size of 0.20 between nodes B and D. Based on this information, we estimate the effect size between A and B as the difference between these two direct estimates, or 0.20. Thus, A appears to be more effective than B despite the absence of any studies that directly compared these two interventions.

OPEN IN VIEWER

3

Example showing how an indirect effect is estimated

The estimate of the indirect effect makes the assumption that the effects are transitive. In its simplest form, the transitive property assumes that if A = B and B = C, then A = C. In the context of network meta-analysis, the assumption is that if the A vs. D effect size is greater than the B vs. D effect size, then A is greater than B. Thus, we are determining that A is more effective than B because the effect sizes for A compared to no treatment (D) are bigger than the effect sizes for B compared to no treatment (D). It is important to note that the anchor intervention may be another intervention and need not be a control or placebo type condition.

Transitivity requires that the anchor intervention is the same for both sets of comparisons. In our example, if the comparator condition (D) is different between the A-D studies and the B-D studies, then the estimated indirect effect will be biased. Similarly, other differences between the A-D and B-D studies, such as different sample characteristics (e.g., a higher percentage of younger participants in the A-D studies relative to the B-D studies) or differences in the way the outcome is measured, will undermine the credibility of the transitivity assumption. In a nutshell, the transitivity assumption states that the distribution of effect modifiers is the same across treatment comparisons (see Jansen & Naci, 2013). It is an untestable assumption but it can be evaluated empirically by comparing the direct and indirect evidence. The consistency assumption states that agreement between direct and indirect effects is evidence of transitivity.

The credibility of the consistency assumption can be tested when both indirect and direct effects are available between two nodes. Both direct and indirect effects are available for closed loops only. In our example, we have one closed loop (A-C-D) and two open loops, (A-B-D) and (B-C-D). A closed loop has estimates between all pairs within a network of three nodes. An open loop has no direct effects for at least one of the pairs. These are illustrated in Figure 4.

OPEN IN VIEWER

4

Example illustrating open and closed networks

Comparing the direct and indirect estimates for an effect between two nodes assesses consistency. In a closed loop, all three possible pairings have both a direct effect and an indirect effect estimate, as shown in Figure 4. Network meta-analysis provides a statistical assessment of the consistency between direct and indirect effects. A lack of consistency indicates that the transitivity assumption is untenable.

The benefit of network meta-analysis over conventional meta-analysis with moderator analysis comparing different intervention types is the combining of both direct and indirect evidence and assessing the consistency of that evidence. This requires studies that not only compare the interventions of interest to a control condition but also studies that compare interventions with each other (i.e., at least some closed loops). In the absence of any closed loops, network meta-analysis and an analog to the ANOVA type moderator analysis produce comparable findings and share a common underlying statistical model.

Another benefit of network meta-analysis over conventional meta-analysis with moderator analysis is the ability to estimate the probability that a particular intervention is the best, the second best, etc., in a network. All of the interventions can be ranked in terms of effectiveness on the assessed outcome and we can produce rankograms and cumulative ranking plots that depict visually which intervention is the most effective (Salanti et al., 2011). This is particularly useful for a condition with multiple viable intervention options.

An important limitation of network meta-analysis is the observational nature of the indirect comparisons. This is a common concern for all moderator type analyses in meta-analysis. There is always a concern that there are differences in the distribution of effect modifiers between the two sets of studies producing the indirect effect estimate. That is, there may be a difference between the A-D and B-D studies other than intervention type, A or B. This reflects the observational nature of between study comparisons. Meta-analysts using this method need to ensure that the studies that make up a network are sufficiently similar to justify the method.

For more detailed information on network meta-analysis, see the following references: Cipriani et al. (2009), Lumley (2002), Salanti et al. (2008), and White et al. (2012).

Campbell policy

The following methods policy was proposed for consideration by the Campbell Collaboration at its Steering Group meeting in Dublin on 24 May 2015.

Network meta-analysis is an acceptable method for Campbell reviews. Reviewers using this method are expected to attend to the following issues in the review. 1.

Discuss the appropriateness of network meta-analysis for the literature being reviewed.

Further resources

A YouTube video of a workshop on network meta-analysis by Dimitris Mavridis given at the Campbell Collaboration Colloquium at Queen's University Belfast (16-19 June 2014) can be found at: https://www.youtube.com/watch?v=SWfi9v-TaV4