Abstract
Decision makers in different health care settings need to weigh the benefits and harms of alternative treatment strategies. Such health care decisions include marketing authorization by regulatory agencies, practice guideline formulation by clinical groups, and treatment selection by prescribers and patients in clinical practice. Multiple criteria decision analysis (MCDA) is a family of formal methods that help make explicit the tradeoffs that decision makers accept between the benefit and risk outcomes of different treatment options. Despite the recent interest in MCDA, certain methodological aspects are poorly understood. This paper presents 7 guidelines for applying MCDA in benefit-risk assessment and illustrates their use in the selection of a statin drug for the primary prevention of cardiovascular disease. We provide guidance on the key methodological issues of how to define the decision problem, how to select a set of nonoverlapping evaluation criteria, how to synthesize and summarize the evidence, how to translate relative measures to absolute ones that permit comparisons between the criteria, how to define suitable scale ranges, how to elicit partial preference information from the decision makers, and how to incorporate uncertainty in the analysis. Our example on statins indicates that fluvastatin is likely to be the most preferred drug by our decision maker and that this result is insensitive to the amount of preference information incorporated in the analysis.
Many decisions in health care involve assessing the balance of favorable and unfavorable effects of alternative treatment regimens, taking into account the associated uncertainties. For example, to choose among alternative treatment options, doctors and patients need comparative evidence to assess whether a new compound is expected to have a more favorable benefit-risk profile than existing alternatives. While subjectivity in the assessment of the benefit-risk balance of alternative treatments cannot be avoided, the decision-making process itself can be made more transparent by describing the underlying value judgments in a formal and consistent manner.
A recent systematic review 1 identified multicriteria decision analysis (MCDA) and stochastic multicriteria acceptability analysis (SMAA) to be among the most promising methods for conducting a quantitative benefit-risk assessment. MCDA provides a framework for systematic and replicable analyses of complex decision problems involving value tradeoffs. 2 MCDA based on multiattribute value or utility models has been proposed for use in benefit-risk assessments.3–5 SMAA allows application of these models in cases where exact information on the decision maker’s preferences is not available. Although previous studies have demonstrated applicability of these approaches in relative treatment effect assessment,2,6 certain methodological aspects are still poorly understood in the health care research community. The structured process of arriving at a multicriteria benefit-risk decision regarding a particular medication is not trivial, and various potential pitfalls lie on the analyst’s path.
To enable wider application of MCDA in drug benefit-risk assessment, this paper provides guidance on 7 important phases of the assessment process: how to define the decision problem, how to select a set of nonoverlapping evaluation criteria, how to synthesize and summarize the available data, how to translate relative measures obtained through evidence synthesis to absolute scales that permit comparisons between the criteria, how to define suitable scale ranges, how to elicit preference information, and how to incorporate uncertainty into the analysis. Using a running example on a widely used class of cholesterol-lowering drugs, statins, we make recommendations about potential ways to address these key methodological challenges.
Data
Our running example considers a set of 6 statins for which there was evidence available from 3 recently conducted systematic reviews7–9: atorvastatin, fluvastatin, lovastatin, pravastatin, rosuvastatin, and simvastatin. The focus of the first study was on determining the comparative tolerability and harms of the individual statins, and it included data on the number of participants who experienced myalgia, elevations in hepatic transaminases, elevations in creatine kinase (CK), and discontinuations because of adverse events. The second and third studies focused on assessing the comparative benefits of statins and included data on all-cause mortality and major coronary events and on major cerebrovascular events, respectively. All open-label and double-blind randomized, controlled trials comparing one statin with another at any dose or with control that had more than 50 participants per trial arm, lasted longer than 4 weeks, and reported any of the outcomes of interest were eligible for inclusion. In total, this resulted in the inclusion of 184 randomized controlled trials with 260,630 individuals with or without cardiovascular disease at baseline.
Guidelines for MCDA of Comparative Benefit-Risk Assessment
MCDA can potentially be useful for supporting different decisions. These include regulatory decisions at the market entry level, development of clinical practice guidelines when it is imperative to recommend a specific treatment option to initiate prescription drug therapy, and prescribing decisions in clinical practice. Although different in nature, all of these problems concern choosing among multiple treatment alternatives, and therefore the evidence concerning the beneficial and harmful effects of the available drug options needs to be evaluated. The general guidelines we describe are therefore relevant for a broad spectrum of decision problems. An overview of the guidelines is presented in Table 1.
Summary of the Guidelines
Define the Decision Problem
The first step in the benefit-risk assessment of a prescription drug is to explicitly define the decision problem. This involves specifying, among others, the indication for which the assessment is conducted, the alternative treatments under consideration, the criteria on which the different treatments are to be evaluated, and the decision maker(s). The indication considered for our case study was the use of statins to reduce cardiovascular disease risk in patients who have one or more elevated cardiovascular risk factors, who are free of diabetes, and who do not have a history of cardiovascular disease (i.e., primary prevention). The alternatives under consideration were the 6 different statins included in our dataset, and the criteria were defined based on the clinical endpoints contained in this dataset. Our decision maker is a clinical expert from the domain of cardiovascular diseases.
We specifically focus on the comparative benefit-risk assessment of statins and the difficulty in choosing a first-line treatment out of the 6 currently available options. Our case study on statins is important in a number of ways. First, statins are among the most widely prescribed and used medications around the world. The recent clinical practice guidelines in the United States and the United Kingdom considerably expanded the scope and intensity of statin therapy for a broader population of individuals with or at risk of developing coronary heart disease. Second, the 6 statins currently on the market differ in terms of their benefit and harm profiles. So far, clinical practice guidelines have not considered the important differences among the 6 statins. Third, statins constitute a case in which the selection among alternative treatments is primarily a clinical one: By 2016, all 6 statins will be available in generic formulations, making cost considerations largely irrelevant.
Choose a Set of Nonoverlapping Evaluation Criteria
When selecting the evaluation criteria, it is important to avoid overlaps as much as possible. In general, one can therefore not simply conduct the benefit-risk assessment based on all the available study endpoints as this is prone to result in an overrepresentation of certain health effects. For example, while the change in HbA1c and the change in fasting plasma glucose are 2 well-established endpoints in clinical trials related to treatment with glucose-lowering products, they both serve as surrogate endpoints that measure how well the patients have, on average, responded to the investigated treatments. Only one of these endpoints should therefore be included in a multicriteria decision model, especially if this model is of an additive structure. Similarly, it may happen that the same clinical events get counted multiple times. In such situations, the data set is ideally restructured such that overlap in the definition of the endpoints is avoided. Where this is not possible, the decision maker should select a nonoverlapping subset of these endpoints.
When there are multiple decision makers, they might disagree on which set of nonoverlapping endpoints is most relevant for the benefit-risk assessment. In such cases, either a consensus on the model structure should be reached through discussions, or if no consensus can be reached, multiple analyses with different endpoints must be performed. Yet another option is to keep all (overlapping) endpoints, but this would require using a more complex, nonadditive multicriteria decision model, which includes additional preference parameters whose elicitation is out of scope of this paper.
Most of the discontinuation events in our data set are likely to be due to myalgia or transaminase elevation. Therefore, when applied to our statins case study, the principle of removing overlapping criteria implies that either discontinuation or myalgia and transaminase should be excluded from the set of decision criteria. Our decision maker chose to include myalgia and transaminase elevation and to exclude discontinuation. Additionally, if we were to simultaneously consider all-cause mortality, risk of stroke, and risk of myocardial infarction, the occurrence of a fatal stroke or a fatal myocardial infarction would be counted toward 2 of these criteria. The main reason for initiating statin treatment is to reduce an individual’s risk of experiencing a cardiovascular event. One way to resolve this problem would therefore be to include the risks of stroke and myocardial infarction and exclude all-cause mortality. A downside of this approach is that any beneficial or harmful effects that statin treatment may have on non-cardiovascular-related mortality would then no longer be captured. Alternatively, one could further refine the available data by decomposing stroke and myocardial infarction into fatal and nonfatal and then including all-cause mortality, nonfatal stroke, and nonfatal myocardial infarction as the beneficial effects. The advantage of this approach is that it captures all the relevant clinical endpoints. A potential drawback, however, is that those clinical trials in which fatal and nonfatal events are not clearly differentiated can no longer be included. Because we had sufficient data available, we chose this latter approach in our statin selection example. The following 6 criteria were therefore included in our analysis: all-cause mortality, nonfatal stroke, nonfatal myocardial infarction, myalgia, transaminase, and CK elevation.
Criteria that represent the same health effects must necessarily be highly correlated. However, this does not mean that including 2 correlated criteria always results in overlap. For example, the effect of a blood-thinning agent on the prevention of thrombosis is strongly correlated to the risk that it will cause bleeding events. However, despite this correlation, both are separate events and should be represented as distinct criteria in the MCDA model. Data permitting, this correlation can be taken into account in the analysis (see Lahdelma and others10,11).
Synthesize and Summarize the Available Data
The next step is to numerically assess the performance of the treatments on the selected benefit and risk criteria. For some criteria, there could be only one clinical study available from which the estimates of absolute treatment effects (e.g., incidence rates) can be obtained directly. 2 For other criteria, there may be multiple studies available, meaning that some form of evidence synthesis is required before one is able to express a treatment’s performance on these criteria in terms of a single numerical value (with associated credible intervals). For criteria measurements that are derived from randomized controlled trials, the use of network meta-analysis is now commonplace.12–15 As this approach uses both direct and indirect comparisons when estimating differences in the performance between the considered treatments, it is not required to restrict the analysis to only those studies that have a chosen common comparator, which would be required when applying traditional pairwise meta-analyses.
The clinical trials included in our dataset cover multiple indications, including primary prevention, secondary prevention, diabetes management (among individuals with or at risk of developing coronary heart disease), and treatment of acute coronary syndrome. As both the relative and absolute reduction in cardiovascular disease risk associated with the use of statins can vary depending on the patient population considered, it is important that all the studies included in the benefit-risk assessment fit the indication for which a decision has to be taken. Previous network meta-analyses indicated no significant differences in the estimated relative effects with different subgroups.7–9,16 For this reason, and to increase accuracy of the estimates, we chose to estimate relative effects for the 4 chosen decision criteria using data from all 184 available studies. Figure 1 presents the network structure for studies that measured all-cause mortality events. Odds ratios estimated for the 6 chosen decision criteria are presented in Figure 2.
Network of trials used for the analysis. Edge thickness indicates the amount of studies comparing the treatments on the outcome “all-cause mortality”(1–22).
Odds ratios of the 6 treatments against control on the 6 chosen benefit-risk criteria.
Translate Relative Measures Obtained in Evidence Synthesis to Absolute Scales That Permit Comparisons between the Criteria
Network meta-analyses produce relative effect estimates, which are unsuitable for specifying value tradeoffs, as they do not contain information on the baseline effect. To illustrate why, consider 2 alternatives that are evaluated in terms of 2 criteria. Suppose that the relative risk of alternative 1 against alternative 2 is 1.5 for criterion A and 0.8 for criterion B. While on a relative scale, the difference between the 2 treatments is much larger for criterion A than for criterion B, these differences are impossible to interpret without considering the baseline effect of treatment 2. For example, suppose that the baseline effect of alternative 2 is 2% on criterion A and 50% on criterion B. For alternative 1, the previously reported relative risks then translate into an absolute risk of 3% on criterion A and an absolute risk of 40% on criterion B. Depending on what these criteria entail, a 10% difference in absolute risk on criterion B may be far more important than a 1% difference in absolute risk on criterion A, showing that the results from a network meta-analysis first need to be translated to values measured on absolute scales before one is able to make value tradeoffs in a meaningful way.
The results of our previously conducted network meta-analyses on statins were presented on the odds ratio scale, which suffers from the same problem illustrated above for the risk ratio. In both cases, the solution is the same: Translate the relative effects to an absolute scale using an estimate of the absolute effect for a suitably selected baseline treatment.6,17 The absolute effect estimates can be from randomized trials or observational studies, and the baseline treatment can be either a placebo or an active treatment. What matters is that the studies included in the estimation of the baseline effects are representative of the target population and that the mean follow-up of each of these studies is similar so that the event rates observed in these studies are comparable. If no suitable data are available, estimations of the baseline effects may need to be elicited from expert clinicians.
We estimated baseline effects in our case study for both the hard clinical outcomes (nonfatal strokes, nonfatal MIs, and all-cause mortality) and side effects (myalgia, transaminase, CK elevation). For each side effect, we estimated the baseline effect using a Bayesian random-effects pooling of the event rates in the placebo arms across all trials. We specified an informative prior (a half-normal: |N(0, 0.25)|) for the heterogeneity standard deviation to ensure it could be estimated. The absolute effect of the placebo intervention was then defined as the resulting predictive distribution. The predictive distribution incorporates both the uncertainty around the mean and the between-study heterogeneity and thereby fully accounts for heterogeneity in the observed effects. Baseline risks of the hard clinical outcomes were estimated using a single large study representative of the target population. The study results were then used to model the Beta distributed baseline effects using a Bayesian approach with a flat Beta(1,1) prior, following Tervonen and others. 2 We chose ALLHAT-LLT as the largest non-industry-sponsored study corresponding to the primary prevention population. 18 According to the American College of Cardiology risk calculator, the trial population of ALLHAT-LLT has on average a 21.40% 10-year risk for atherosclerotic cardiovascular disease, defined as coronary death or nonfatal myocardial infarction or fatal or nonfatal stroke (http://tools.cardiosource.org/ASCVD-Risk-Estimator/).
Estimates of the absolute effects for the 6 statins were subsequently obtained by combining the absolute effect of placebo with the relative effects obtained from the network meta-analyses. This is achieved by sampling from both the estimated distribution for the baseline risk and the distribution for the log odds ratios. For a given baseline risk pA and a log-odds ratio for treatment B of dAB, the absolute risk for treatment B is given by pB = logit−1(logit(pA) + dAB). The absolute effects are illustrated for the 6 chosen criteria in Figure 3. When compared with Figure 2, the different rate of side effects appears to be much less pronounced because all absolute effect estimates incorporate uncertainty around the highly uncertain baseline effect. However, the occurrence of side effects is strongly correlated through this common baseline, and this is taken into account in the decision analysis, so that the true treatment differences are preserved.
Absolute scale measurement ranges for the 6 treatments and control (used as baseline) on the 6 chosen benefit-risk criteria.
Define Suitable Scale Ranges
In practical applications of MCDA, the constructed multicriteria model is often assumed to be of an additive structure, which is illustrated in Figure 4. Then the problem of formally representing the decision maker’s preferences reduces to the problem of specifying a set of partial value functions that reflect the relative desirability of decision criteria levels (e.g., increase of a side effect from 0% to 5% v. from 5% to 10%) and a set of weights that reflect the relative importance of worst-best scale increases across the criteria ranges. To contextualize the decision and make subsequent weight elicitation meaningful, the criteria scale ranges should be defined with respect to plausible outcome ranges. That is, if mortality over the set of considered treatments varies only within 2%–6%, the partial value functions should be defined for this range instead of, for example, 0%–100%. Although the partial values can always be interpolated within the range, using ranges irrelevant for the decision context causes the preferences to be captured with a lower accuracy.
Example partial value functions (u1, u2) for a 2-criterion additive value model, for a hypothetical choice of an obesity treatment. The first criterion measures the incidence rate of serious side effects (treatment risks), and the second criterion measures the observed weight loss in percentages (treatment benefit). Treatment a1 has a 0.5 incidence rate of side effects, which translates to a value of 0.66 on criterion 1. Treatment a2 has a 0.6 incidence rate of side effects, which translates to a value of 0.33 on criterion 1. The weights 0.6 and 0.4 express that the scale swing of the first criterion [0.7, 0.4] is 50% more important than the scale swing of the second criterion [0%, 15%]. In this example, given exact measurements and preference information (shapes of partial value functions and weights), the second alternative (a2) is preferred over the first alternative (a1) because it has a higher value with the given value function (0.56 v. 0.44).
The challenge in defining suitable scale ranges relates to uncertainty of the measurements. We have previously 2 suggested defining the worst-best scaling based on interval hulls of the per-criterion 95% credible intervals from the absolute scale joint distribution. For example, in case of all-cause mortality, this is 0.05–0.15, as defined by the distributions of fluvastatin and lovastatin (Figure 3, mid-left panel). Although such ranges capture most of the variance, they might be inappropriate with long-tailed measurement distributions. Table 2 illustrates this with respect to the statins case; the span of 95% empirical credible intervals for transaminase (0.01–0.31) is considerably smaller than the full sample range (0.00–0.86). However, because using scale ranges that are too large causes imprecision for the preference elicitation, we do not currently have a better recommendation than using the 95% ranges.
Sample Quantiles (100,000 draws) on Absolute Measurement Scales
Note: The 0% and 100% (minimum and maximum) are highly unstable and are shown only to illustrate that by setting the scale ranges to the 95% credible interval, we must accept a small error in the final analysis due to sampled values falling outside the defined scale range.
Elicit Preference Information
After the scale ranges have been defined, additive value models incorporate decision maker preferences by eliciting partial value functions and their scaling factors (weights).
Partial value functions
Partial value functions reflect the desirability of scale values within individual criteria. Eliciting preferences over scale values on continuous criteria can be done with the bisection method. 19 For example, if an outcome of interest is the risk of stroke and its worst and best levels are 6% and 2%, respectively, then the first step in the bisection method would be to ask for the value of x such that a decrease from 6% to x% is as important as a decrease from x% to 2%. If the decision maker replies by stating that x equals 4, the partial value function for stroke is likely to be linear. Linear partial value functions are often appropriate for criteria that measure event rates in a defined patient population, as in these cases equal size ranges in percentages (e.g., 5%–7% or 90%–92%) reflect the same number of affected patients. Nonlinear partial value functions, in contrast, reflect situations where the value associated with a fixed performance increment depends on the level of achievement on a criterion. For example, suppose that the amount of toxicity that a decision maker is willing to accept for a 6-month increase in mean survival time is larger for an increase from 6 months to 1 year (e.g., patients who underwent prior treatment) than for an increase from 2.5 years to 3 years (e.g., treatment-naïve patients). The partial value function for overall survival would then be concave (i.e., a function whose slope decreases as the level of performance increases). Partial value functions that are convex or S-shaped are also possible.
The lower value function in Figure 4 illustrates a hypothetical case of bisection elicitation where the criterion measures weight loss and the observed losses vary from 0% to 15%. Let us suppose that the first answer of the clinical expert is that half of the benefits are obtained at 10% weight loss, which is normally used as a threshold for clinically significant effect. Then, the analyst could ask again what is the half-point of effects between 0% and 10%, and the expert states 7%. This leads to the partial value function given in the figure, which can then be used to calculate the alternatives’ partial values and, once the weights are known, to rank the alternative treatments. The bisection procedure in principle is continued infinitely, but usually a few answers provide a good approximation of the “true” partial value function.
Weight information
Weights of the additive model express accepted tradeoffs over the criteria scale swings. For example, if the scale of mortality is [6%, 2%] and the scale of discontinuation is [50%, 10%], then if mortality has weight 100 and discontinuation weight 1, the increase of mortality from 2% to 6% is considered 100 times worse than the increase of discontinuation from 10% to 50%. Note that this is the only meaning of the weights—they do not express any kind of absolute importance. Therefore, elicitation questions such as “what is more important: all-cause mortality or myalgia?” are meaningless when dealing with additive value models.
In the above example, the criteria measurements are expressed as incidence rates and the partial value functions are taken to be linear, and therefore it may seem reasonable to trade off one event against another, rather than compare the incidence ranges. However, doing so depends heavily on the linearity assumption—that is, that a change on a single criterion from 2% to 1% is equally valuable to a change from 100% to 99%. This may decrease the precision as well as value of the elicited weight information because it would become less specific to the problem at hand. The importance of taking scale ranges into account is clearer when the decision involves outcomes such as blood pressure lowering in the treatment of hypertension. For example, the decision maker may face the dilemma of whether the difference in blood pressure lowering on the scale [–5 mm Hg, –15 mm Hg] outweighs the occurrence of serious adverse events on the scale [2%, 0%].
Weights can be elicited with the swing method, in which the decision maker is asked to judge the relative importance of the worst-best scale swings (as described above). However, all elicitation techniques resulting in exact weights are subject to behavioral biases. 20 Therefore, many modern MCDA approaches allow incorporation of weight information in imprecise or incomplete formats. Imprecise information can be modeled, instead of point estimates, as intervals for the tradeoff ratios. For example, instead of tradeoff ratio of 2, the decision maker could express imprecision with the ratio belonging to the interval [1.5, 2.5]. Incomplete information expresses exact but poor information, similar to pair-wise choices in conjoint analysis. The simplest form of incomplete weight information is (partial) ranking of scale swings (ordinal information).
Both incomplete and imprecise statements define linear constraints on the set of feasible weights, where the weights are always nonnegative and normalized to sum to a constant, usually to unity. For example, if the decision maker provides exact weight information in a 2-criterion problem, stating that the scale swing of the first criterion is twice as important as that of the second criterion, this would result in a single normalized weight vector [2/3, 1/3]. If instead the decision maker provides imprecise information stating that the tradeoff ratio belongs to the interval [1, 2], this results in a feasible weight space where the first weight is bound within interval [1/(1 + 1), 2/(2 + 1)] = [0.5, 2/3], and the second weight is 1 minus the first weight (and bound within [1/3, 0.5]). If the decision maker provides only incomplete (ordinal) information stating that the scale swing of the first criterion is more important than that of the second criterion, this restricts feasible weights so, that
Incorporate Uncertainty in the Analysis
The 2 main sources of uncertainty in benefit-risk assessment are the uncertain outcome estimates due to limited sample sizes in clinical trials and imprecise or incomplete weight information. To propagate the uncertainty in these inputs into uncertainty in the ranking of the treatments, we have previously proposed2,6 the application of stochastic multicriteria acceptability analysis (SMAA).21,22 This entails sampling a sufficient number of observations 23 from the measurement distributions and, for each of these, sampling a weight vector from a uniform distribution within the feasible weight space.24,25 In each of the Monte Carlo iterations, the alternatives are ranked from best to worst according to their total value, which is computed using the realized values of the weights and measurement outcomes (see Figure 4). Different realizations of the weights and measurement outcomes may translate into a different ranking of the treatments. In SMAA, this uncertainty is captured by computing the rank acceptability indices, which describe, for all possible combinations of ranks and treatments, the fraction of Monte Carlo iterations for which a treatment is ranked at a certain position.
Sampling weights uniformly from the feasible weight space specified with tradeoff intervals is not trivial, especially if there are many decision criteria. We recommend using the Hit-And-Run sampler, which is an efficient Markov Chain Monte Carlo technique.24,25 There exists an open source R package “hitandrun” that implements the sampling method and another package “smaa” for computing the SMAA rank acceptability indices. Both are freely available at the Comprehensive R Archive Network (CRAN) repository.
The choice of criterion scales has an effect on the simulation technique. For example, in our statins case, the 95% credible interval hull for transaminase is 1%–31%, but samples from the pooled distribution span the range 0%–86%. Therefore, if the partial value function is defined for 1%–31%, some of the samples will be outside this range. A simple approach to solve the problem is to extrapolate using extreme points of the function range. This will cause the simulation results to contain higher variance. However, as the rank acceptabilities are computed based on rank counts from the individual simulations, the out-of-range samples do not introduce excessive variance due to the ordinal nature of the computations.
By allowing imprecise weight information, the SMAA approach enables one to analyze the treatment benefit-risk profiles with increasingly precise weight information: missing, ordinal, tradeoff intervals, and exact tradeoffs. To illustrate the effects of increasing precision of weight information, we elicited weight information from an expert in cardiovascular medicine. We first asked for a ranking of the criteria scale swings and then elicited exact tradeoff statements. Afterward, we asked the expert to assess uncertainty of his exact statements to obtain tradeoff intervals. The elicitation protocol is presented in Appendix A (available online).
Results
The results of the weight elicitation are presented in Table 3. For exact and interval tradeoffs, transaminase and CK elevation had weights much smaller than the sampling error (<< 0.01), and therefore in those analyses we included only the other 4 criteria. We computed the SMAA rank acceptability indices using the 4 sets of weight information: missing, ordinal, interval tradeoffs, and exact tradeoff statements. The rank acceptabilities are illustrated in Figure 5. The analysis code and the full data set are freely available online. 26
Swing Weight Elicitation Results for the 4 Analyses with Increasingly Precise Weight Information
Weights of the different criteria:
Rank acceptability indices for the 6 treatments and control from the 4 analyses with increasingly precise weight information.
The results indicate that without any weight information (Figure 5, top-left panel), all treatments apart from the no-treatment alternative, control, have a possibility to be the preferred one, although atorvastatin, fluvastatin, and simvastatin have the highest first rank probabilities. When ordinal information is incorporated in the analysis, fluvastatin becomes clearly the likely candidate for being the preferred treatment (76% first rank acceptability), and pravastatin and rosuvastatin obtain approximately zero first rank probabilities. When more precise weight information (tradeoff intervals) is added to the analysis, the results remain approximately the same. The precise weight statements indicate that the hard clinical endpoints are very important in the analysis. For example, the scale swing of all-cause mortality was considered by our decision maker to be approximately 440 times more important than the scale swing of transaminase, which could be then excluded from the analysis altogether. The imprecise ratio bounds are quite tight, and therefore results from the analysis with exact tradeoff ratios (weights) are very similar to the ones from the analysis with interval tradeoffs. The remaining uncertainty is due to the imprecise relative effects that were obtained as pooled estimates in the network meta-analysis.
Discussion
Considering the benefits and harms of multiple treatment options has clear appeal for a variety of decisions in healthcare. Such considerations are an essential component of prescription decisions in clinical practice, development of clinical practice guidelines by expert committees, and benefit-risk assessments for market entry decisions in regulatory settings.27–31 MCDA offers a framework to explicitly compare and contrast and transparently trade off the benefits and harms of multiple healthcare interventions.
Our case on statins illustrated the key challenges in applying MCDA to comparative benefit-risk assessment. First, the set of criteria should be defined to capture all aspects relevant for the decision and to avoid double counting. Second, evidence from clinical trials should be synthesized through network meta-analysis to obtain relative treatment estimates, and these should be transformed to absolute effect estimates. The transformation requires a baseline estimate, which should be obtained taking into account the target population. Third, suitable scale ranges for the absolute scales should be defined for the weight elicitation. We recommend using the 95% credible interval hull from the absolute scale joint distribution. Fourth, weight information should be elicited at different levels of precision to understand the effect of imprecise and incomplete weights on the analysis results. And fifth, uncertainty on both the effect estimates and the weights should be incorporated into the analysis by applying a simulation-based technique, resulting in rank probabilities for all the different precisions of weight information.
Beyond the methodological considerations outlined in this paper, operationalizing the use of MCDA for comparative benefit-risk assessment has a number of limitations. While MCDA is gaining momentum in regulatory settings for evaluating the benefit and risk assessment of single agents, 28 drug licensing agencies such as the European Medicines Agency and the Food and Drug Administration are still reluctant to consider comparative evidence to evaluate the relative benefit-risk profiles of new drugs.32,33 When regulatory agencies adopt relative effectiveness as a criterion for licensing decisions in the future, network meta-analysis, and its combination with MCDA, would serve as a valuable tool to inform decision making in the regulatory setting.
Comparative benefit-risk assessment using MCDA has clear implications for routine clinical practice. However, using MCDA for making prescribing decisions in clinical practice faces a number of practical challenges. Incorporating MCDA models into evidence-based computerized decision aids would necessitate prespecifying, automating, and making available large parts of the MCDA model ahead of a clinical encounter. Decision aids such as SHARE-IT are already capable of automating and packaging key aspects of existing evidence into accessible summaries for patients. 34 Future efforts should focus on integrating MCDA capabilities to such decision aids. Patients could use these either prior to a clinic visit or during the patient-clinician encounter. This would allow the patient-provider interaction to focus primarily on patient preferences on various benefit and harm outcomes, which would inform the prescribing decision. As we argued previously, 35 we envision a future where computer decision aids are informed by systematic reviews and syntheses of all relevant clinical evidence on clinically meaningful benefit and harm outcomes. Combining evidence syntheses with MCDA would make feasible evidence-based decisions that are informed by provider expertise and knowledge and tailored according to patient preferences.
The approach presented in this paper is purely illustrative and is not intended to dictate prescribing decisions in clinical practice. To the contrary, our case study highlights the importance of carefully accounting for decision maker preferences when considering both benefit and harm outcomes and their tradeoffs. Irrespective of how the benefit-risk assessment is conducted, however, we appreciate that there are external factors that influence the final decision taken. For example, consider a new treatment that is compared against the standard of care. Even when this new treatment is considered to have a better benefit-risk balance, the decision may still fall in favor of the established treatment because of concerns with certain identified risks for which the available evidence was highly uncertain. Depending on the nature of the problem, the final benefit-risk decision may also be affected by various ethical, social, and economic aspects that do not directly influence a treatment’s benefit-risk balance but could nevertheless still have a profound impact on the acceptance of the decision by the public and other stakeholders. As such, MCDA approaches present an opportunity to guide and inform decisions rather than to dictate them.
Footnotes
TT, HN, AEA, AA, HLH, and DP declare no conflicts of interest. GvV has provided consulting services to Johnson & Johnson and (as a subcontractor of Deloitte) for UCB Pharma on the conduct of network meta-analyses.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
