The Impact of Using Informative Priors in a Bayesian Cost-Effectiveness Analysis

Abstract

Background. Bayesian methods have been proposed as a way of synthesizing all available evidence to inform decision making. However, few practical applications of the use of Bayesian methods for combining patient-level data (i.e., trial) with additional evidence (e.g., literature) exist in the cost-effectiveness literature. The objective of this study was to compare a Bayesian cost-effectiveness analysis using informative priors to a standard non-Bayesian nonparametric method to assess the impact of incorporating additional information into a cost-effectiveness analysis. Methods. Patient-level data from a previously published nonrandomized study were analyzed using traditional nonparametric bootstrap techniques and bivariate normal Bayesian models with vague and informative priors. Two different types of informative priors were considered to reflect different valuations of the additional evidence relative to the patient-level data (i.e., “face value” and “skeptical”). The impact of using different distributions and valuations was assessed in a sensitivity analysis. Models were compared in terms of incremental net monetary benefit (INMB) and cost-effectiveness acceptability frontiers (CEAFs). Results. The bootstrapping and Bayesian analyses using vague priors provided similar results. The most pronounced impact of incorporating the informative priors was the increase in estimated life years in the control arm relative to what was observed in the patient-level data alone. Consequently, the incremental difference in life years originally observed in the patient-level data was reduced, and the INMB and CEAF changed accordingly. Conclusions. The results of this study demonstrate the potential impact and importance of incorporating additional information into an analysis of patient-level data, suggesting this could alter decisions as to whether a treatment should be adopted and whether more information should be acquired.

Keywords

Bayesian cost-effectiveness informative priors decision making

Economic evaluation, an important tool for informing rational health care decision making, depends critically on the sources of evidence from which estimates of the relative costs and effects are derived. In the case of an economic evaluation conducted alongside a clinical trial (i.e., a patient-level analysis), cost and effect data would be determined for each patient in the study. These sample data could then be used to generate estimates for the mean costs and effects for patients under each of the treatments being compared. As these values represent estimates for the true population mean costs and effects, uncertainty around these sample values is often incorporated using the nonparametric bootstrap method.¹ The bootstrap method propagates uncertainty using only the information contained in the data, effectively ignoring all other sources of evidence external to the trial (e.g., literature). In contrast, in a Bayesian approach, the trial data as well as any external evidence can be taken into account through the combination of the prior distributions (i.e., external evidence) and the likelihood function (i.e., the data from the trial),² thus allowing for a more comprehensive approach to the incorporation of uncertainty.

Despite the importance of incorporating all available evidence to inform decision making,^{3

–6} a recent review⁷ of 16 trial-based Bayesian cost-effectiveness studies reported that 50% of the studies used noninformative or vague priors only in their analyses. This provides little guidance to policy makers on the potential of Bayesian methods to integrate all available evidence to capture the uncertainty inherent in decision making.⁷ Noninformative or vague priors are appropriate in those situations where there is a genuine lack of additional (i.e., prior or new external) information. However, in those situations where prior information exists, or new information becomes available either during the course of a trial or after its completion, failure to take this into account could impact the results. Through the use of the prior distribution, the Bayesian approach provides a mechanism by which this additional information can be incorporated into a trial-based cost-effectiveness analysis. At the very least, it would be useful to have a sense of what impact this external evidence might have on the trial results.

The objective of the current analysis was to compare the results of a traditional frequentist method (i.e., nonparametric bootstrap) that relies only on the information contained in the patient-level data to a Bayesian approach using informative priors to incorporate evidence in addition to the patient-level data. To inform the prior distributions, our analyses combined the results of several published studies, available at the time of the original analysis, in a meta-analysis. This article also makes use of 2 different types of informative prior distributions to reflect different potential valuations of the additional information (i.e., “face value” and “skeptical”). These prior distributions are then used to combine the additional information with the patient-level data from a published trial-based economic evaluation comparing endovascular aneurysm repair (EVAR) with open surgical repair (OSR).⁸

Case Study

A previous trial-based economic evaluation comparing elective EVAR and OSR for the treatment of abdominal aortic aneurysms for patients at high surgical risk provides the patient-level data for the current analysis.⁸ These data were based on a 1-year nonrandomized study conducted at a single site in Ontario, Canada. Total costs expressed in 2006 Canadian dollars and life years at 1 year were reported for 140 EVAR patients (treatment group) and 52 OSR patients (control group). The 2 groups were well matched in terms of clinical characteristics.⁸ The estimated mean costs indicated that EVAR ($34,147) was slightly less expensive than OSR ($34,170), and estimated mean life years indicated EVAR (0.96) was more effective than OSR (0.85). Thus, on the basis of point estimates only, EVAR dominated OSR in terms of incremental cost per life year gained. Sampling uncertainty in the trial data was incorporated using the standard nonparametric bootstrap method. Although additional evidence existed from other studies,⁹ this information was not included in the cost-effectiveness analysis.

Methods

The following describes the methods being compared and introduces the sources of evidence used to illustrate the potential impact of incorporating informative priors into a Bayesian trial-based economic evaluation.

The Bootstrapping Method

The bootstrapping method is nonparametric by nature, meaning it makes no assumption about the parametric distribution of the data. The method resamples with replacement from the original sample data to build an empirical estimate of the sampling distribution for the statistic of interest.¹ Although nonparametric bootstrapping does not assume any particular form of distribution, the choice of statistic used implicitly does. For example, if the sample mean is the statistic chosen to be monitored in the repeated samples, the results will be similar to those based on a parametric assumption of normality,¹⁰ provided the sample size is large enough.

Costs and effects were sampled simultaneously to generate 1000 bootstrap replicates to estimate the sampling distribution for the sample mean costs and effects for both the EVAR and OSR groups as well as for the incremental costs (ΔC) and effects (ΔE) of EVAR compared to OSR. The incremental net monetary benefit (INMB) was then obtained by rescaling the incremental effects between EVAR and OSR into a monetary value using a threshold willingness to pay for a life year gained (λ) of $10,000 and subtracting the incremental costs from this value (i.e., λΔE – ΔC).¹ Therefore, assuming the objective is to maximize health gains for a given budget, an INMB(λ) > 0 indicates that EVAR is optimal compared to OSR.¹¹ Using the percentile method, the limits of the 95% confidence intervals (CIs) around the various statistics of interest were calculated based on the 25th and 976th ordered values. Incorporating the uncertainty due to sampling error and uncertainty about the cost-effectiveness threshold, a cost-effectiveness acceptability curve (CEAC) was computed. From a frequentist perspective, the CEAC plots, as a function of λ, are 1 minus theP value of the 1-sided test of INMB(λ) = 0 versus INMB(λ) > 0. However, the CEAC is more commonly described from a Bayesian perspective as “the probability that the treatment is cost-effective” given the data.¹² To represent decision uncertainty in the current analysis, a cost-effectiveness acceptability frontier (CEAF) was constructed from the CEAC by identifying the range of values of λ over which each intervention had the highest mean net benefit (i.e., it was optimal).¹¹ The frontier indicates the probability that the intervention with the highest net benefit will be cost-effective. The decision uncertainty or the error probability is then 1 minus the value of the frontier.¹³

Bayesian Analysis

The basis for making inferences from a Bayesian perspective is Bayes theorem. In essence, Bayes theorem describes the combination of information from 2 sources, the likelihood and the prior.² The likelihood function summarizes all of the information that is contained in the data (e.g., a trial). In the current analysis, this refers to the patient-level data comparing EVAR and OSR.⁸ The prior distribution represents information that is available in addition to the data. In this analysis, the prior describes the information from the literature available at the time of the original analysis.⁹ In the absence of additional information, vague or noninformative prior distributions can be used. The less informative the prior, the more weight is given to the data in the analysis. The priors are combined with the data to generate the posterior distribution, which represents what is now known about the unknown quantity (e.g., mean effects) given the prior information and the data. The posterior is proportional to the product of the likelihood and the prior.²

Bivariate normal likelihood

The central limit theorem (CLT) states that for any population distribution of costs and effects, the distributions of the sample means will converge to normal distributions as the sample size increases.¹² In practice, the approximation is generally very good for sample sizes of 50 or more.¹⁴ The simulation results of Nixon and others,¹² which were based on different scenarios for sample size and skewness, further indicate that for moderate to large sample sizes (i.e., n > 50), the CLT performs well. Based on the current sample sizes of 140 and 52 patients for EVAR and OSR, respectively, we have invoked the CLT to justify the validity of the sample means as estimators for the population means. As noted previously, the use of sample means gives similar results to assuming normal distributions, especially for larger sample sizes, through the action of the CLT.¹⁰ To accommodate the correlation between costs and life years observed in the patient-level data for both the EVAR (–0.20) and OSR (–0.31) groups, the cost and effect data were first modeled using bivariate normal distributions.¹⁰ The 140 EVAR and 52 OSR patients were indexed by i, and the 2 study arms were indexed by j (i.e., j = 1 for EVAR and 2 for OSR):

C_{ij} \sim Normal (μ_{Cj,} σ_{Cj}^{2}),

E_{ij} \sim Normal (μ_{Eij,} σ_{Ej}^{2}), and

μ_{Eij} = μ_{Ej} + β_{j} (C_{ij} - μ_{Cj}) .

Here, the costs have a normal distribution with mean μ_Cj and standard deviation σ_Cj (equation 1). The effects have a normal distribution with mean μ_Eij and standard deviation σ_Ej (equation 2). As seen in equation 3, the mean of E_ij depends, through the parameter β_j, on how much the cost C_ij is above the mean cost μ_Cj. The model allows the correlation between costs and life years to be different in the 2 study groups, through the separate respective β_j parameters. The subtraction of μ_Cj ensures that μ_Ej remains interpretable as the overall mean effect in the jth arm of the study.¹⁰ As implied by the regression in equation 3, effects have been made a function of costs. To justify the assumption of a linear relationship between life years and costs, we plotted the residuals versus the costs and found no departure from linearity.

Vague priors

In addition to the likelihood function, a Bayesian analysis requires prior distributions for the unknown population parameters. In the initial analysis, vague priors (Table 1) were used so that the resulting inferences essentially depended only on the data. In that regard, we would expect the results from the Bayesian analysis to be similar to those from the nonparametric bootstrap approach.¹⁰

Table 1

Vague and Informative Prior Distributions for the Bayesian Models

		Informative Prior^a
Parameter	Vague Prior	Face Value Prior	Skeptical Prior
EVAR
Mean costs (μ_C1)	Normal(25,000, 1E11)	Not applicable	Not applicable
Precision costs (1/σ_C1 ²)	Gamma(0.50, 1E-07)	Not applicable	Not applicable
Mean life years (μ_E1)^b	Beta(1, 1)	Normal(0.95, 1.01E-03)	Normal(0.95, 8.38E-04)
		Normal(0.95, 4.73E-04)	Normal(0.95, 8.38E-04)
Standard deviation life years (σ_E1)	Uniform(0, 10)	Not applicable	Not applicable
Relationship between costs and life years (β₁)	Normal(0, 10,000)	Not applicable	Not applicable
Shape costs (ρ_C1)	Uniform(0, 100)	Not applicable	Not applicable
Scale costs (υ_C1)	Uniform(0, 100)	Not applicable	Not applicable
OSR
Mean costs (μ_C2)	Normal(25,000, 1E11)	Not applicable	Not applicable
Precision costs (1/σ_C2 ²)	Gamma(0.50, 1E-07)	Not applicable	Not applicable
Mean life years (μ_E2)^b	Beta(1, 1)	Normal(0.93, 2.21E-03)	Normal(0.93, 8.76E-03)
		Normal(0.93, 2.12E-03)	Normal(0.93, 8.76E-03)
Standard deviation life years (σ_E2)	Uniform(0, 10)	Not applicable	Not applicable
Relationship between costs and life years (β₂)	Normal(0, 10,000)	Not applicable	Not applicable
Shape costs (ρ_C2)	Uniform(0, 100)	Not applicable	Not applicable
Scale costs (υ_C2)	Uniform(0, 100)	Not applicable	Not applicable

Note: EVAR = endovascular aneurysm repair; OSR = open surgical repair.

Informative priors were only available for mean life years (μ_Ej) in the EVAR and OSR groups.

Two informative priors for each type derived from 2 sets of evidence (i.e., 2 high-risk and 8 mixed-risk studies).

Informative priors

To incorporate all available evidence in the Bayesian cost-effectiveness analysis, studies from a published systematic review,⁹ which were available at the time of the original patient-level analysis, were combined with the trial data. The review identified 8 nonrandomized studies conducted in high-risk patients. All 8 studies provided estimates of 30-day postoperative mortality. Two of the high-risk studies also provided estimates of longer term mortality but not at 1 year (i.e., mean follow-up of 26.8 months for EVAR and 27.6 months for OSR in one study¹⁵ and 15.6 months for EVAR and 19.8 months for OSR in the other study¹⁶). In addition to the body of evidence in high-risk patients, the review also contained information from another 8 nonrandomized studies that were not restricted to high-risk patients but measured mortality at 30 days and 1 year after treatment in a mixed population of low- to high-risk patients.

To estimate the 1-year mortality rate in a high-risk population, the 30-day mortality rates observed in the 8 high-risk studies were combined with conditional probabilities measuring the probability of being dead at 1 year given the patient was alive at 30 days. These conditional probabilities were calculated from 2 sets of evidence. First, the 2 high-risk studies reporting mortality data at around 2 years were used,^15,16 assuming that the 1- and 2-year probabilities of being dead conditional on being alive at 30 days were similar. The second set of evidence consisted of the 8 studies that measured mortality at 30 days and at 1 year in a mixed-risk population.⁹ For EVAR, the mortality rates for both 30 days and conditional on being alive at 30 days for the studies were fairly consistent with those from the trial (i.e., 3% v. 1% for 30-day mortality and 4% high risk, 5% mixed risk v. 6% for conditional rates). In contrast, the studies reported, on average, lower mortality rates for OSR compared to the trial (i.e., 6% v. 10% for 30-day mortality and 3% high risk, 3% mixed risk v. 9% for conditional rates). To estimate the 1-year mortality associated with EVAR and OSR in high-risk patients, for each of the 2 sets of evidence, binomial models were constructed in WinBUGS¹⁷ to combine information on 30-day mortality and longer term mortality. The details of the additional studies and the calculations are provided in the Appendix. In the absence of information on Canadian-specific costs from these studies, the informative priors were limited to effects.

For each of the 2 sets of data used to estimate the 1-year mortality to inform the prior on effects (i.e., 2 high-risk studies and 8 mixed-risk studies), 2 different types of informative prior distributions for μ_Ej were examined.¹⁸ These priors are described in Table 1. The first informative prior used in the analysis was labeled a “face value” prior because the additional information was taken at face value and no specific concession was made for any potential differences between the additional information and the patient-level data. This could reflect a belief that the evidence from the literature was of high quality and as reliable as the patient-level data. The second informative prior was labeled “skeptical.” This could reflect concerns regarding the risk level of the patients in the studies, the time periods over which mortality was measured, or the quality of the evidence. In that case, the additional information could be explicitly given less weighting than the patient-level data. To explicitly downweight the external evidence relative to the trial data, we initially used a prior variance for mean life years that was 4 times the variance of the patient-level data. This was based on a previous study by Sutton and Abrams.¹⁸ Such downweighting reflects skepticism regarding the additional evidence and would be appropriate in situations where a researcher believes that although this evidence provides some information, it should be treated with caution.¹⁸

Sensitivity analysis

While the use of a variance that was 4 times as large as that for the patient-level data corresponded to the variance inflation factor used by Sutton and Abrams,¹⁸ other choices are possible. The more the variance from the additional studies is inflated relative to the variance for the patient-level data, the more their evidence is downweighted relative to the patient-level data. To get a better understanding of the impact of using different inflation factors to explicitly downweight the additional information, a sensitivity analysis was conducted (e.g., inflating the variance by 2 rather than by 4).

To assess the sensitivity of the cost-effectiveness results to different cost distributions, the normal distributions in equation 1 were replaced with gamma distributions for both the EVAR and OSR arms:

C_{i j} \sim Gamma (ρ_{Cj}, υ_{Cj}) .

The gamma distributions for the costs were parameterized by their shape ρ_Cj and their scale υ_Cj. The mean was then ρ_Cj/υ_Cj, and the standard deviation was √ρ_Cj/υ_Cj. Otherwise, the formulation of the model was the same as before. Vague priors for the shape and scale parameters are given in Table 1.

Estimations

All posterior distributions of quantities of interest for both the informative priors and the Bayesian cost-effectiveness analyses were estimated in WinBUGS.¹⁷ For all Bayesian analyses, an initial burn-in of 100,000 iterations was discarded to ensure convergence. History plots, autocorrelation plots, and various diagnostics available in the Bayesian Output Analysis package,¹⁹ performed on 2 chains, were used to assess convergence. Posterior estimates were based on a subsequent sample of 100,000 iterations. These posterior distributions were summarized as posterior means and 95% credible intervals (CrIs). In contrast to a frequentist 95% CI, a Bayesian 95% CrI is an interval that has a 95% probability of containing the true parameter value. As with the nonparametric bootstrap, the estimated quantities included the sample means for costs and effects for EVAR and OSR as well as the mean cost difference (ΔC) and the mean effect difference (ΔE) between the EVAR and OSR groups. The INMB, CEACs, and CEAFs were also calculated.

Results

The Bootstrapping Method

Estimated values for mean costs and life years for both the EVAR and OSR groups and incremental costs and life years and their associated 95% CIs are presented in Table 2. The results closely correspond to those from the original study (i.e., ΔC = –$24(–$11,582, $9165) and ΔE = 0.11(0.02, 0.21)).⁸ The estimated mean INMB at a willingness to pay of $10,000 and its 95% CI are also reported. The positive mean value indicates that EVAR is optimal compared to OSR at λ equal to $10,000.

Table 2

Results for Nonparametric Bootstrap and Bayesian Bivariate Normal Models with Vague and Informative Priors for Mean Life Years

	Annual Mean Costs^a (μ_Cj), $		Incremental^a (ΔC), $	Mean Life Years^a (μ_Ej)		Incremental^a (ΔE)	Mean INMB,^a $ (λ = $10,000)
Model	EVAR	OSR	EVAR – OSR	EVAR	OSR	EVAR – OSR	λ ΔE – ΔC
Bootstrap	34,147 (32,485, 36,356)	34,170 (24,331, 45,606)	–23 (–11,586, 9692)	0.96 (0.93, 0.98)	0.85 (0.76, 0.93)	0.11 (0.02, 0.20)	1111 (–8893, 12,865)
Bayesian
Vague priors	34,150 (32,240, 36,050)	34,170 (23,620, 44,750)	–18 (–10,740, 10,660)	0.96 (0.93, 0.99)	0.85 (0.75, 0.94)	0.11 (0.01, 0.21)	1107 (–9893, 12,190)
Informative priors
2 high-risk studies^b
Face value	34,150 (32,250, 36,060)	32,840 (22,460, 43,110)	1318 (–9116, 11,870)	0.96 (0.93, 0.98)	0.89 (0.82, 0.96)	0.07 (–0.005, 0.14)	–645 (–11,370, 9938)
Skeptical	34,160 (32,270, 36,050)	33,630 (23,180, 44,030)	531 (–10,050, 11,160)	0.96 (0.93, 0.98)	0.87 (0.78, 0.95)	0.09 (0.002, 0.18)	383 (–10,510, 11,230)
8 mixed-risk studies^c
Face value	34,180 (32,290, 36,070)	32,790 (22,420, 43,030)	1387 (–9012, 11,900)	0.96 (0.93, 0.98)	0.89 (0.83, 0.96)	0.06 (–0.007, 0.13)	–748 (–11,430, 9807)
Skeptical	34,170 (32,270, 36,070)	33,600 (23,130, 44,010)	565 (–9986, 11,190)	0.96 (0.93, 0.98)	0.87 (0.78, 0.95)	0.09 (0.00002, 0.18)	336 (–10,570, 11,150)

Note: INMB = incremental net monetary benefit; EVAR = endovascular aneurysm repair; OSR = open surgical repair.

Values in parentheses represent 95% confidence intervals for nonparametric bootstrap and 95% credible intervals for Bayesian models.

High-risk studies reporting mortality after 30 days are used in the calculation of the informative priors.

Mixed-risk studies reporting mortality at 1 year are used in the calculation of the informative priors.

Bayesian Bivariate Normal Analysis

Vague prior distributions

The posterior mean estimates and 95% CrIs obtained from the Bayesian bivariate normal analysis with vague priors were similar to the mean estimates and 95% CIs from the nonparametric bootstrap (Table 2). The CEAFs were also quite similar (Figure 1). These results reflect the vagueness of the prior distributions and suggest that most of the information in the analysis is coming from the patient-level data.

Informative priors

As evidenced by the informative prior distributions given in Table 1 for the mean life years, the lower mortality rates for OSR reported in the literature translated into higher estimates for mean life years in the OSR group (i.e., 0.93) relative to the patient-level data (i.e., 0.85). The consistency of the mortality rates for EVAR led to estimates for mean life years that were roughly the same for both the informative priors (i.e., 0.95) and the patient-level data (i.e., 0.96). Consequently, and in contrast to the results for the vague prior distributions, the incorporation of informative priors for mean life years increased the posterior estimates for mean life years in the OSR group from 0.85 life years gained (LYG) to between 0.87 LYG and 0.89 LYG depending on the type of prior used (e.g., “face value”). The associated 95% CrIs shifted upwards and as a result of the added information became narrower. Likewise, the posterior estimates for mean life years and the associated intervals, although slightly narrower, were essentially unchanged for EVAR. These results were consistent across both types of priors for both sets of evidence (Table 2).

The extent of the increase relative to the mean values for OSR observed in the nonparametric bootstrap and vague models reflected the weight of the information associated with each type of informative prior. In the current analysis, the weight of the additional information relative to the data decreased as the priors moved from “face value” to “skeptical.” This was also apparent in the incremental estimates as the differences in mean life years between EVAR and OSR got progressively larger as the additional information was given less weight. In terms of the 95% CrIs associated with these incremental differences, they all shifted downwards and became more precise compared to those based on vague priors. These changes resulted in differing estimates for the mean INMB both between the 2 types of informative priors and relative to the vague and bootstrap models. As seen in Table 2, the estimated values for the “skeptical” priors remained positive, while for the “face value” priors, the values became negative, indicating that, for the latter, OSR would be optimal and, for the former, EVAR would be optimal.

Due to modeling the correlation between costs and effects, the informative priors on life years also impacted the mean costs. As the mean life years in the OSR group increased, the mean costs in the OSR group decreased. The combined impact of these changes in terms of both the optimal alternative based on existing information and the decision uncertainty can be seen in a comparison of the CEAFs. Figure 1 presents the range of values for λ over which EVAR or OSR had the highest mean net benefit and the approximate switch point where the current decision changes from one intervention to the other. The frontiers indicate that the more informative the prior, the larger the range over which OSR is optimal compared to EVAR and the higher the switch point or λ required for EVAR to be cost-effective. The switch point corresponds to the base incremental cost-effectiveness ratio (i.e., ΔC/ΔE) for the decision. For example, the λ for which EVAR would be optimal compared to OSR increased from $0 for the bootstrap and vague normal models to approximately $20,000 for the “face value” priors. The priors also impacted the decision uncertainty. As the priors became more informative, the probability that EVAR was optimal compared to OSR decreased for all λs, and consequently, the decision uncertainty increased.

Sensitivity analysis

In the initial analysis, the “skeptical” priors were downweighted by 4 times the variance of the patient-level trial data. Although this introduced a switch point ($6000–$7000) where the decision changed from OSR to EVAR (Figure 1), EVAR still had the higher mean net benefit compared to OSR for λ equal to $10,000 (Table 2). A sensitivity analysis revealed that the additional evidence for the OSR group would have to be downweighted by between 2 and 2.5 times the variance of the patient-level data in order for EVAR to remain optimal compared to OSR at a willingness to pay of $10,000. This would mean that the additional information for the OSR group would have to be downweighted by at least 50% relative to the patient-level data in order for the current decision to remain unchanged.

A second sensitivity analysis considered the impact on the results of using gamma distributions for the costs instead of normal distributions (Table 3). The main impact was in terms of the estimated precision for the mean costs in the OSR group. The gamma distribution increased this precision relative to the normal distribution. This increased precision around the mean costs was due to lower estimates for the variance in the data compared to what was estimated using a normal distribution and what was observed in the trial data itself. Accordingly, the precision of the estimates for the mean ΔC and mean INMB also increased. When informative priors were used, the estimates for the mean costs in the OSR group decreased, as before, but by less than with the normal distributions. Consequently, EVAR remained optimal compared to OSR for λ equal to $10,000 across all priors. In terms of decision uncertainty, the CEAFs in Figure 2 have the same pattern as those in Figure 1. The switch points, however, are about half the value they were with the normal distributions (i.e., approximately $10,000 and $3000, respectively, for the “face value” and “skeptical” priors).

Table 3

Sensitivity Analysis: Results for Gamma Cost Distributions with Vague and Informative Priors for Mean Life Years

	Annual Mean Costs^a (μ_Cj), $		Incremental^a (ΔC), $	Mean Life Years^a (μ_Ej)		Incremental^a (ΔE)	Mean INMB,^a $ (λ = $10,000)
Model	EVAR	OSR	EVAR – OSR	EVAR	OSR	EVAR – OSR	λ ΔE – ΔC
Vague priors	34,150 (32,600, 35,760)	34,180 (28,010, 41,660)	−35 (−7660, 6347)	0.96 (0.93, 0.99)	0.85 (0.76, 0.94)	0.11 (0.01, 0.21)	1125 (−5494, 8986)
Informative priors
2 high-risk studies^b
Face value	34,150 (32,600, 35,760)	33,600 (27,650, 40,690)	556 (−6707, 6704)	0.96 (0.93, 0.98)	0.89 (0.82, 0.95)	0.07 (−0.002, 0.14)	131 (−6156, 7516)
Skeptical	34,150 (32,620, 35,760)	33,930 (27,890, 41,200)	218 (−7194, 6476)	0.96 (0.93, 0.98)	0.87 (0.78, 0.95)	0.09 (0.005, 0.18)	704 (−5748, 8332)
8 mixed-risk studies^c
Face value	34,160 (32,620, 35,780)	33,570 (27,630, 40,670)	597 (−6644, 6760)	0.96 (0.93, 0.98)	0.89 (0.83, 0.96)	0.07 (−0.004, 0.13)	55 (−6248, 7388)
Skeptical	34,160 (32,610, 35,770)	33,940 (27,870, 41,250)	221 (−7233, 6503)	0.96 (0.93, 0.98)	0.87 (0.78, 0.95)	0.09 (0.004, 0.18)	688 (−5789, 8327)

Note: INMB = incremental net monetary benefit; EVAR = endovascular aneurysm repair; OSR = open surgical repair.

Values in parentheses represent 95% credible intervals for Bayesian models.

High-risk studies reporting mortality after 30 days are used in the calculation of the informative priors.

Mixed-risk studies reporting mortality at 1 year are used in the calculation of the informative priors.

Discussion

By comparing the nonparametric bootstrap to a Bayesian approach with both vague and informative priors, this study has sought to assess the potential impact of incorporating all available evidence into a trial-based economic evaluation. While the nonparametric bootstrap and the Bayesian approach using vague priors produced similar results, our study has demonstrated the potential for informative priors to impact both the decision about which alternative should be chosen based on existing information and whether more information should be acquired. Based on whether the additional information was incorporated into the analysis, and depending on a decision maker’s willingness to pay for a life year gained, this could result in very different funding decisions. The impact on decision uncertainty observed in the CEAFs suggests the synthesis of evidence from different sources could also play a role in decisions about future research, ensuring that resources are used efficiently. This could be particularly important in those situations where the additional information suggests something different from the patient-level data, as was observed in our case study.

In addition to exploring the potential impact of combining all available evidence, this study also considered how the additional information might be weighted or valued relative to the patient-level data from the original cost-effectiveness analysis. As the objective was to combine all available evidence to inform decision makers, this study provides insight into how multiple sources of evidence may be combined together in the prior and used in addition to the trial data. Integral to this process is an understanding of how to value the additional information relative to the patient-level data. Attempts were made to assess the impact on the cost-effectiveness results of different types of informative prior distributions. Specifically, 2 types of priors were examined (i.e., “face value” and “skeptical”).

In terms of deciding how much the additional information should contribute to the analysis, a more thorough consideration would need to be given as to why the mortality rates for OSR reported in the literature differed from the patient-level estimates. This could have implications both in terms of the weight ascribed to the additional information and to the potential need for future research. Unfortunately, none of the studies in the literature provided detailed information on all of the clinical characteristics necessary to evaluate the risk level of the patients. In addition, although both the trial-based economic evaluation and the studies from the literature were nonrandomized, the trial was well balanced in terms of patient characteristics, while there was evidence of covariate imbalance among the literature studies. Again, attempts to understand the potential impact of these imbalances are limited by the extent of missing covariate data.

In combination, these factors (i.e., real surgical risk level unknown in many studies and nonrandomized evidence) suggest that we may be unlikely to take the evidence from the literature at “face value.” As in our case study, this essentially gives the external evidence and the patient-level data for the OSR group equal weighting. Rather, some degree of downweighting would seem to be necessary. The results of the sensitivity analysis indicate that the additional information for the OSR group must be downweighted by at least 50% in order for EVAR to remain optimal compared to OSR. Whether this represents a reasonable valuation of the evidence in the literature relative to the patient-level data is not clear and likely would require additional research. Future research could also look at the feasibility of using models that elicit expert opinion concerning the rigor and relevance of the studies being combined.²⁰ Methods have also been proposed that use estimates from previously published meta-analyses to adjust and downweight studies.²¹ Again, the limited availability of covariate data would likely make any assessments regarding adjustment and downweighting difficult in the current analysis.

This article focused on the use of the prior distribution to combine all available evidence in a Bayesian cost-effectiveness analysis. A possible concern was the absence of data for life years at 1 year after treatment for the 8 high-risk studies from the literature. This meant that these values had to be estimated. Although actual data would have been preferable, the similarity of the outcomes for both sets of information (i.e., 2 high-risk and 8 mixed-risk studies) reinforced the results. Under ideal circumstances, additional information on total 1-year costs in EVAR and OSR patients would also have been available from the studies.

Another possible limitation is that in an empirical cost-effectiveness analysis, the true form of the distributions for the costs and effects remains unknown. If correct about the true population distributions, efficiency in estimating the population means could be gained. However, the use of estimators based on incorrect distributional assumptions can lead to totally misleading conclusions. Overall, the sample mean performs well.²² Accordingly, we have used the sample mean in the current analysis and assessed the sensitivity of the results to different parametric distributions for the costs.

Despite limitations, this study has demonstrated the potential importance of using all available evidence to inform decision makers. Where cost-effectiveness analyses and economic evaluations are a critical input to health care policy making, it is paramount that these policy decisions be based on the available evidence. This study contributes to the literature an example of how this may be achieved using actual data from a previous patient-level cost-effectiveness analysis and evidence available from the literature at the time of the original analysis. Future research could focus on further refinements, and of course, the approaches undertaken will likely vary depending on the context and availability of data.

Conclusions

This analysis indicates that ignoring specific sources of evidence could undermine cost-effectiveness results. Not only might it change decisions regarding the cost-effectiveness of one intervention compared to another, but it could also impact decisions regarding the need for future research. Only when all available evidence is taken into consideration can we be confident of well-informed health care decisions.

Footnotes

Appendix

Mortality Data for Patient-Level and Additional Studies

			30-Day Mortality^a				Mortality at End of Follow-up Conditional on Being Alive at 30 Days^b
			EVAR		OSR		EVAR		OSR
Study	Only High Risk	Length of Follow-up, d	Patients, N	Deaths, n (%)	Patients, N	Deaths, n (%)	Patients, N	Deaths, n (%)	Patients, N	Deaths, n (%)
Patient level
Tarride and others 2008⁸	Y	365	140	1 (1)	52	5 (10)	139	9 (6)	47	4 (9)
Additional: high risk
Du Toit 1998²³	Y	30	12	0 (0)	10	1 (10)
Carpenter 2002²⁴	Y	30	174	7 (4)	163	7 (4)
Forbes 2002²⁵	Y	30	7	0 (0)	31	0 (0)
Patel 2003²⁶	Y	30	16	0 (0)	35	6 (17)
Ianneli 2005²⁷	Y	30	34	0 (0)	28	1 (4)
Mendonca and others 2005¹⁵	Y	815 EVAR, 840 OSR	18	1 (6)	31	2 (6)	17	2 (12)	29	2 (7)
De Donato 2006²⁸	Y	30	19	1 (5)	8	1 (13)
Parmer and others 2006¹⁶	Y	475 EVAR, 602 OSR	52	0 (0)	46	0 (0)	52	0 (0)	46	0 (0)
Additional: mixed risk
Zarins 1999²⁹	N	365					185	8 (4)	60	2 (3)
Becquemin 2000³⁰	N	365					71	3 (4)	105	2 (2)
Cohnert 2000³¹	N	365					35	3 (9)	37	0 (0)
Ting 2003³²	N	365					26	1 (4)	24	0 (0)
Ballard 2004³³	N	365					22	1 (5)	107	2 (2)
Elkouri 2004³⁴	N	365					94	0 (0)	258	0 (0)
Greenberg 2004³⁵	N	365					199	7 (4)	78	3 (4)
Bush 2006³⁶	N	365					695	40 (6)	1120	77 (7)

Note: EVAR = endovascular aneurysm repair; OSR = open surgical repair; Y = yes; N = no.

Mortality data at 30 days for mixed-risk studies were not used in calculations.

Only applies to studies with mortality data beyond 30 days.

References

Drummond

Sculpher

Torrance

O’Brien

Stoddart

. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. Oxford: Oxford University Press; 2005.

Spiegelhalter

Abrams

Myles

. Bayesian Approaches to Clinical Trials and Health-Care Evaluation. Chichester, UK: John Wiley & Sons Ltd.; 2004.

Ades

Sculpher

Sutton

. Bayesian methods for evidence synthesis in cost-effectiveness analysis. Pharmacoeconomics. 2006;24(1):1–19.

Prevost

Abrams

Jones

. Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Stat Med. 2000;19:3359–76.

Sculpher

Claxton

Drummond

McCabe

. Whither trial-based economic evaluation for health care decision making. Health Econ. 2006;15:677–87.

Sutton

Cooper

Jones

. Evidence synthesis as the key to more coherent and efficient research. BMC Med Res Methodol. 2009;9:29.

McCarron

Pullenayegum

Marshall

Goeree

Tarride

. Handling uncertainty in economic evaluations of patient level data: a review of the use of Bayesian methods to inform health technology assessments. Int J Technol Assess Health Care. 2009;25(4):546–54.

Tarride

Blackhouse

De Rose

. Cost-effectiveness analysis of elective endovascular repair compared with open surgical repair of abdominal aortic aneurysms for patients at a high surgical risk: a 1-year patient level analysis conducted in Ontario, Canada. J Vasc Surg. 2008;48(4):779–87.

Hopkins

Bowen

Campbell

. Effects of study design and trends for EVAR versus OSR. Vasc Health Risk Manag. 2008;4(5):1011–22.

10.

Thompson

Nixon

. How sensitive are cost-effectiveness analyses to choice of parametric distributions? Med Decis Making. 2005;25:416–23.

11.

Fenwick

Claxton

Sculpher

. Representing uncertainty: the role of cost-effectiveness acceptability curves. Health Econ. 2001;10:779–87.

12.

Nixon

Wonderling

Grieve

. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Econ. 2010;19:316–33.

13.

Briggs

Sculpher

Claxton

. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press; 2006.

14.

O’Hagan

Stevens

. Bayesian methods for design and analysis of cost-effectiveness trials in the evaluation of health care technologies. Stat Methods Med Res. 2002;11:469–90.

15.

Mendonca

Moreira

RCR

Ribas

Jr. . Comparison between open and endovascular treatment of abdominal aortic aneurysms in high surgical risk patients. J Vasc Brasileiro. 2005;4:232–42.

16.

Parmer

Fairman

Karmacharya

Carpenter

Velazquez

Woo

. A comparison of renal function between open and endovascular aneurysm repair in patients with baseline chronic renal insufficiency. J Vasc Surg. 2006;44:706–11.

17.

Lunn

Thomas

Best

Spiegelhalter

. WinBUGS: a Bayesian modelling framework. Concepts, structure, and extensibility. Stat Comput. 2000;10:325–37.

18.

Sutton

Abrams

. Bayesian methods in meta-analysis and evidence synthesis. Stat Methods Med Res. 2001;10(4):277–303.

19.

Bayesian Output Analysis Program (BOA). Version 1.1: User’s Manual, 2005. Available from: URL: www.public-health.uiowa.edu/boa/BOA.pdf

20.

Turner

Spiegelhalter

Smith

GCS

Thompson

. Bias modeling in evidence synthesis. J R Stat Soc Ser A. 2009;172:21–47.

21.

Welton

Ades

Carlin

Altman

Sterne

JAC

. Models for potentially biased evidence in meta-analysis using empirically based priors. J R Stat Soc Ser A. 2009;172:119–36.

22.

Briggs

Nixon

Dixon

Thompson

. Parametric modelling of cost data: some simulation evidence. Health Econ. 2005;14:421–8.

23.

Du Toit

Saaiman

De Beer

Pretorius

. Endovascular stent-graft repair of abdominal aortic aneurysms–single-center experience and acute outcome. Cardiovasc J S Afr. 1998;88:C273–C81.

24.

Carpenter

Baum

Barker

. Durability of benefits of endovascular versus conventional abdominal aortic aneurysm repair. J Vasc Surg. 2002;35(2):222–8.

25.

Forbes

DeRose

Kribs

Harris

. A cost-effectiveness analysis of standard versus endovascular abdominal aortic aneurysm repair. Can J Surg. 2002;45(6):420–4.

26.

Patel

Langan

III Taylor

. An analysis of standard open and endovascular surgical repair of abdominal aortic aneurysms in octogenarians. Am Surg. 2003;69(9):744–7.

27.

Iannelli

Monaco

Di Tommaso

. Endovascular vs. open surgery of abdominal aortic aneurysm in high-risk patients: a single center experience. Thorac Cardiov Surg. 2005;53(5):291–4.

28.

De Donato

Chisci

Trovato

. Open and endovascular treatment of abdominal aortic aneurysm in octogenarians: a single center approach. Italian J Vasc Endovasc Surg. 2006;13(2):77–81.

29.

Zarins

White

Schwarten

. AneuRx stent graft versus open surgical repair of abdominal aortic aneurysms: multicenter prospective clinical trial. J Vasc Surg. 1999;29(2):292–308.

30.

Becquemin

Bourriez

d’Audiffret

. Mid-term results of endovascular versus open repair for abdominal aortic aneurysm in patients anatomically suitable for endovascular repair. Eur J Vasc Endovasc Surg. 2000;19(6):656–61.

31.

Cohnert

Oelert

Wahlers

. Matched-pair analysis of conventional versus endoluminal AAA treatment outcomes during the initial phase of an aortic endografting program. J Endovasc Ther. 2000;7(2):94–100.

32.

Ting

ACW

Cheng

SWK

. Endovascular repair for abdominal aortic aneurysms: initial experience of an endograft programme. Asian J Surg. 2003;26:17–21.

33.

Ballard

Abou-Zamzam

Teruya

Bianchi

Petersen

. Quality of life before and after endovascular and retroperitoneal abdominal aortic aneurysm repair. J Vasc Surg. 2004;39(4):797–803.

34.

Elkouri

Gloviczki

McKusick

. Perioperative complications and early outcome after endovascular and open surgical repair of abdominal aortic aneurysms. J Vasc Surg. 2004;39(3):497–505.

35.

Greenberg

Chuter

Sternbergh

III Fearnot

, Zenith Investigators. Zenith AAA endovascular graft: intermediate-term results of the US multicenter trial. J Vasc Surg. 2004;39(6):1209–18.

36.

Bush

Johnson

Collins

. Open versus endovascular abdominal aortic aneurysm repair in VA hospitals. J Am Coll Surg. 2006;202(4):577–87.