Probabilistic sensitivity analysis in health economics

Abstract

Health economic evaluations have recently become an important part of the clinical and medical research process and have built upon more advanced statistical decision-theoretic foundations. In some contexts, it is officially required that uncertainty about both parameters and observable variables be properly taken into account, increasingly often by means of Bayesian methods. Among these, probabilistic sensitivity analysis has assumed a predominant role. The objective of this article is to review the problem of health economic assessment from the standpoint of Bayesian statistical decision theory with particular attention to the philosophy underlying the procedures for sensitivity analysis.

Keywords

Bayesian decision theory health economic evaluation sensitivity analysis

1 Introduction

In recent years, health economics has become an increasingly important discipline in medical research, especially with the transition from the paradigm of evidence-based medicine to that of translational research,^1,2 which aims at making basic research applicable in the context of real practice, and under budget constraints, in order to enhance patients' access to optimal health care.

Since the late 1970s, methods like cost-effectiveness and cost-utility analysis have been established in the health care arena, especially in the Anglo-Saxon world.³ Moreover, in the past 10 years, health economic evaluations have built on more advanced statistical decision-theoretic foundations, effectively becoming a branch of applied statistics,^4,5 increasingly often under a Bayesian statistical approach.^6–10

Even though the process is, technically, a simple application of standard decision-theoretic precepts,¹¹ health economics is complicated by issues related to other important factors that play a major role in real practice medical decision making. Among these are the difficulty of applying standard cost-effectiveness techniques in the regulatory process,¹² and the necessity of properly accounting for uncertainty in the decision process, an issue known as sensitivity analysis.⁸ This latter in particular is fundamental: in some drug control regimes — for example, NICE (a glossary of all abbreviations/acronyms is provided at the end of the paper) in the UK¹³ — it is a required basic component of any new drug approval or reimbursement dossier.

The objective of this article is to review the problem of health economic assessment from the standpoint of Bayesian statistical decision theory, with specific attention to the basic statistical framework and the philosophy underlying the procedures for sensitivity analysis. In particular, in line with recent contributions to the literature, we advocate the use of an integrated vision based on value-of-information (VI) analysis, a procedure grounded in the theory of decision under uncertainty, and criticise the indiscriminate use of other approaches to sensitivity analysis.

2 Statistical framework

In a typical health economic problem, we are interested in the management of a particular clinical condition for which a set of interventions t ∈ = (0, 1,…, T) is available. We can apply a generic intervention t to any unit i in the relevant population and observe a (possibly multivariate) response, Y_i. Typically, Y_i will be represented by a suitable clinical outcome (e.g. blood pressure or occurrence of myocardial infarction), together with a measure of the costs associated with the given intervention.

The unit i might itself be a population, and the treatment t some population-level policy intervention. Sometimes, though not always, the relevant population-level response Y_i would be an average, or other summary (e.g. mean death rate or median time to onset of a disease) of individual-level responses within that population.

The objective of the health economic evaluation is to decide which treatment to apply to a new unit i′, judged as similar to, or, in statistical terms, exchangeable¹⁴ with all the others receiving the same treatment.

2.1 Example

We consider a health economic problem, which we use throughout the paper as a motivating example to discuss the rationale behind health economic modelling. We switch between theoretical considerations and the example to show the practical implications. The example is based on a real health economic model.¹⁵

Suppose the interest is in an infectious disease, for instance influenza, for which a new vaccine has been produced. Under the current management of the disease some individuals treat the infection by taking over the counter (OTC) medications. Some subjects visit their GP and, depending on the gravity of the infection, may receive treatment with antiviral drugs, which usually cures the infection. However, in some cases complications may occur. Minor complications will need a second GP visit after which the patients become more likely to receive antiviral treatment. Major complications are represented by pneumonia and can result in hospitalisation and possibly death. In this scenario, the costs generated by the management of the disease are represented by OTC medications, GP visits, the prescription of antiviral drugs, hospital episodes and indirect costs such as time off work.

The focus is on the clinical and economic evaluation of the policy that makes the vaccine available to those who wish to use it (t = 1) against the null option (t = 0) under which the vaccine will remain unavailable.

We note here that our purpose is in reviewing the statistical methodology and so our results should not be taken as contributing in any way to guidance as to an appropriate management of the disease under discussion: we refer to other publications for a detailed discussion of both the clinical and economic issues.^15–18

2.2 Modelling

The assumption of exchangeability essentially amounts to assuming the following data-generating process for the observables Y_i. First, we introduce a population parameter θ , generally involving treatment specific components: θ = (θ⁰,…, θ^t,…, θ^T). Current uncertainty about θ is formally described by a suitable probability distribution. This is computed starting from a (possibly subjective) prior distribution that describes the state of science about the parameters before observing any new data. For example, for each possible intervention t, patient-level data will generally be available (as produced by a set of randomised trials or observational studies) in the form ^t = {y_i : i = 1,…, n_t}. We generally refer to the whole set of background information as $D = ⋃_{t} D^{t}$ . The joint distribution of all the parameters is then p( θ ∣ ), from which it is possible to obtain every single marginal distribution p(θ^t ∣ ).

Conditionally on θ and a proposed treatment t, the second step of the data generation process consists in drawing the Y_i's independently from the probability distribution p(y ∣ θ^t), which describes the individual variability of the future (yet unobserved) health economic response. A similar general framework has been described in the health economic literature.⁶

2.3 Example (continued)

We describe here the model used for the vaccine problem. In a population made up of N individuals, we model V₁, the number of patients taking up the vaccine when available, using a Binomial(N, φ) distribution, depending on the vaccine coverage rate φ. Obviously, V₀ = 0 as the vaccine is not available in the status quo. For convenience, we denote the total number of patients in the two groups, vaccinated (v = 1) and non-vaccinated (v = 0), by n_tv with n_t1 ≔ V_t and n_t0 ≔ N − V_t, respectively.

Let the relevant clinical outcomes be defined as j = 1: influenza infection; j = 2: GP visit; j = 3: minor complications; j = 4: major complications; j = 5: hospitalisation; j = 6: death; and j = 7: adverse events of influenza vaccination. For each clinical outcome j, let β_j be its baseline rate of occurrence and let ρ_v be the proportional reduction in the chance of infection due to the vaccine. Vaccinated patients (the group v = 1) will experience a reduction in the chance of infection by a factor ρ₁; conversely, for v = 0, individuals are not vaccinated and so the chance of infection is just the attack rate β₁. This is equivalent to setting ρ₀ ≔ 0.

Under these assumptions, the number I_tv of individuals becoming infected in each group can be modelled using a Binomial(n_tv, π_v) distribution, where π_v ≔ β₁(1 − ρ_v) is the probability of infection. Among the infected subjects, the number visiting a GP for the first time is modelled as ${GP}_{tv}^{(1)} ~ Binomial (I_{tv}, β_{2})$ . Using a similar reasoning, among those who have had a GP visit, we can define: the number of individuals with minor complications ${GP}_{tv}^{(2)} ~ Binomial ({GP}_{tv}^{(1)}, β_{3})$ ; the number of those with major complications $P_{tv} ~ Binomial ({GP}_{tv}^{(1)}, β_{4})$ ; the number of hospitalisations $H_{tv} ~ Binomial ({GP}_{tv}^{(1)}, β_{5})$ ; and the deaths $D_{tv} ~ Binomial ({GP}_{tv}^{(1)}, β_{6})$ . The number of individuals experiencing adverse events due to vaccination is computed as AE_tv ∼ Binomial(n_tv, β₇) — obviously, this will be identically 0 in the status quo (t = 0) and among those individuals who choose not to take the vaccine up in the vaccination scenario (t = 1, v = 0).

We also include in the model suitable parameters to describe the remaining aspects of the clinical pathway described in Section 2.1, such as the chance of receiving a prescription after the first GP visit (γ₁) or following minor complications (γ₂) for a number of antiviral drugs (δ); of taking OTC medications (ξ); and of remaining off-work (η) for a number of days (λ). Combining these with the relevant populations at risk, we can then derive the expected number of individuals experiencing each of these events.

As for the costs, following the specification of the actual case¹⁵ on which we base our evaluation, we consider the relevant resources as h = 1: GP visits; h = 2: hospital episodes; h = 3: vaccination; h = 4: time to receive vaccination; h = 5: days off work; h = 6: antiviral drugs; h = 7: OTC medications; h = 8: travel to receive vaccination. For each, we define ψ_h to represent the associated unit cost for which we assume a suitable lognormal distribution, a convenient choice to model positive, continuous variables such as costs.

Finally, we include in the model suitable parameters to represent the loss in quality of life generated by the occurrence of the clinical outcomes. Estimations for the various clinical outcomes can be difficult to obtain. Nevertheless, it is possible to use validated instruments (like the EQ-5D¹⁹) to estimate the Quality Adjusted Life Years (QALYs²⁰), a combined measure of quantity and quality of life used to determine the number of extra years that would be added by an intervention. Let ω_j represent the QALYs lost when an individual experiences the jth outcome. We assume that GP visits do not generate loss in QALYs and therefore set ω₂ = ω₃ ≔ 0; the remaining ω_j's are modelled using a suitable lognormal distribution.

With respect to the discussion of Section 2.2, the assumptions encoded by this model are that we consider a population parameter θ = (θ⁰,θ¹), with the two components being defined as θ⁰ = (β_j, γ₁, γ₂, δ, ξ, η, λ, ψ_h, ω_j) and θ¹ = (φ, β_j, ρ_v, γ₁, γ₂, δ, ξ, η, λ, ψ_h, ω_j). We assume that the components of θ have the distributions specified in Table 1.

Table 1.

Distributional assumptions for the model

	Mean	2.5%	Median	97.5%	Distribution
φ	0.435	0.245	0.436	0.625	Beta(11.31, 14.44)
β₁	0.0701	0.0387	0.0680	0.1116	Beta(13.01, 172.38)
β₂	0.295	0.124	0.288	0.497	Beta(5.80, 13.80)
β₃	0.401	0.388	0.401	0.415	Beta(1909.50, 2851.86)
β₄	0.01339	0.00852	0.0132	0.01938	Beta(20.94, 1538.71)
β₅	0.000378	0.000223	0.000364	0.000616	lognormal(−7.91, 14.93)
β₆	0.000748	0.000366	0.000702	0.001331	lognormal(−7.26, 7.66)
β₇	0.1021	0.0255	0.0954	0.2265	Beta(3.50, 31.50)
ρ₁	0.688	0.593	0.686	0.794	lognormal(−0.374, 0.00524)
γ₁	0.420	0.417	0.420	0.423	Beta(45471.58, 62794.09)
γ₂	0.814	0.806	0.814	0.822	Beta(7701.86, 1759.89)
δ	6.97	2.00	7.00	12.00	Poisson(7.00)
ξ	0.950	0.940	0.950	0.9500.9	Beta(1804.05, 94.95)
η	0.900	0.890	0.900	0.909	Beta(3239.10, 359.90)
λ	2.90	1.22	2.69	5.97	lognormal(0.98, 0.17)
ψ₁	20.55	12.36	19.77	32.07	lognormal(3.00, 0.0606)
ψ₂	2661.92	1554.18	2575.67	4106.98	lognormal(7.85, 0.0606)
ψ₃	7.21	4.22	6.95	11.42	lognormal(1.95, 0.0606)
ψ₄	10.26	6.16	9.92	15.90	lognormal(2.29, 0.0606)
ψ₅	46.31	27.20	44.96	70.69	lognormal(3.80, 0.0606)
ψ₆	3.86	2.39	3.73	5.95	lognormal(1.31, 0.0606)
ψ₇	1.592	0.949	1.562	2.452	lognormal(0.44, 0.0606)
ψ₈	0.807	0.484	0.776	1.311	lognormal(−0.241, 0.0606)
ω₁	4.26	2.14	4.05	7.59	lognormal(1.40, 0.0993)
ω₄	6.39	3.81	6.23	9.82	lognormal(1.82, 0.0606)
ω₅	6.34	3.83	6.15	9.94	lognormal(1.82, 0.0606)
ω₆	15.20	9.09	14.88	23.34	lognormal(2.70, 0.054)
ω₇	0.556	0.316	0.541	0.932	lognormal(−0.634, 0.0717)

Notes: For each parameter, the distributions are chosen to model the available prior knowledge, represented by existing data or expert opinions. The mathematical form of the distributions is chosen according to the nature of the parameter (i.e. parameters describing probability of occurrence of an event are usually given a Beta distribution), while the values of the hyper-parameters are chosen so that the distribution is consistent with the prior information derived by the clinical literature or expert opinion.

The distributions of Table 1 are derived using suitable ‘hyper-parameters’ that have been set to encode knowledge available from previous studies and expert opinion (the relevant sources used to derive values that are consistent with real clinical practice are described elsewhere¹⁵). For example, a Beta distribution with hyper-parameters a = 13.01 and b = 172.38 has the property of having a mean value of approximately 0.069 and contains 95% of the probability mass in the interval [0.004 − 0.115], which is what is suggested by existing data on influenza incidence. It is therefore possible to encode the prior knowledge on β₁ using such a distributional assumption.

We note, however, that the choice of the prior distributions is a matter of context knowledge; for instance, parameters representing the probability of occurrence of an event, e.g. the vaccine attack rate φ, can (but do not have to) be modelled using a Beta distribution. This is just a mathematical convenience, and in fact, it is possible to use different functions to describe the existing knowledge.²¹ In practice, however, relatively ‘standard’ choices can be applied to reasonably approximate the prior information.

Conditionally on the relevant elements of θ , the observed health economic response is represented by $Y = (V_{t}, I_{tv}, {GP}_{tv}^{(1)}, {GP}_{tv}^{(2)}, P_{tv}, H_{tv}, D_{tv}, {AE}_{tv})$ . Its joint distribution is specified using the formulation described above and its components can be combined to implement the economic analysis.

3 Decision making in health economics

Suppose an intervention t is applied and results in outcome y. In health economic terms, we can quantify this situation by a combination of a measure of clinical effectiveness e (for instance measured in terms of QALYs) of the outcome y, and the costs c associated with the selected intervention t. With each situation (y, t), we thus associate a pair (e, c). The objective of health economic evaluations is to compare the proposed interventions in terms of their expected performances along these two dimensions of interest, benefit and cost. For example, we might consider the increment in mean effectiveness:

Δ_{e} : = E [e | θ^{1}] - E [e | θ^{0}]

and the increment in mean cost:

Δ_{c} : = E [c | θ^{1}] - E [c | θ^{0}] .

(note that these are functions of the unknown quantities θ⁰, θ¹ and as such are random variables).

Historically, health economic evaluations have been concerned with the calculation of the Incremental Cost Effectiveness Ratio (ICER), defined as ICER ≔ E[Δ_c]/E[Δ_e], where the expectations are now over the subjective distribution of θ .

The ICER, a pure number, represents the cost per incremental unit of effectiveness (e.g. cost per QALY gained, or cost per death/event averted). The use of the ICER has been widely criticised because of some major limitations.¹⁰ For example, knowing the sign of the ICER is not sufficient to identify the optimal treatment: an ICER of £100 can be derived by values of (E[Δ_e], E[Δ_c]) = (2, 200), indicating that the new treatment produces an increase in effectiveness of 2 units at the cost of extra £200, as well as by the values (E[Δ_e], E[Δ_c]) = (−2, −200), a case in which the new intervention is less effective, but cheaper. Moreover, from the statistical point of view the ICER can have infinite variance and it is generally difficult to perform interval estimation for its values.

A more effective way of describing the problem of allocating the best treatment to the new unit i′ is to use the formal theory of decision making under uncertainty, which is in this case generated by the imperfect knowledge of the random quantities (Y, θ ).^8,22

We take the standpoint of a body that is responsible for issuing guidance on the implementation of alternative interventions for specific public health matters. As suggested earlier, typically, a standard programme will be available and a new one is suggested to replace it, perhaps partially or only on specific sub-populations of individuals. The argument can easily be extended to T > 2 different treatments; however, for the sake of simplicity, we here confine attention to the case = (0, 1).

The overall value of applying treatment t and obtaining response y is supposed measured by a utility function u(y, t), assigning a numeric value to each combination of outcomes and costs. According to the precepts of Bayesian decision theory and on the basis of the current data , the value of taking decision t is the expected utility,

U^{t} : = E [u (Y, t) | D)] = \int \int u (y, t) p (y | θ^{t}) p (θ^{t} | D) d y d θ^{t} .

(1)

The expected utility is obtained by averaging over the uncertainty in both population (‘objective’) and parameters (‘subjective’) domains.

The overall utility is ^* ≔ max_t ^t, based on choosing the intervention t yielding this maximum value. Equivalently, we choose t = 1 if (and, henceforth ignoring ties, only if) EIB > 0, where

EIB : = U^{1} - U^{0}

(2)

is the expected incremental benefit (of treatment 1 over treatment 0). It is easy to see that

U^{*} = max {EIB, 0} + U^{0} .

(3)

Note that EIB is a fixed quantity, uncertainty in both domains having been averaged out. The Bayesian process thereby provides a ranking of the alternatives.

3.1 Choosing a utility function: the net benefit

The main difficulty in applying decision theory and (1) is that a form of the utility function must be specified. In a health economic problem, we need to combine the two measures e and c into a single real-valued utility measure, u(y, t) = f(e, c). While there are many possibilities, a common form of utility function is the (monetary) net benefit²³

u (y, t) = ke - c .

(4)

Here k is a ‘willingness-to-pay’ parameter used to put cost and benefits on the same scale and represents the budget that the decision maker is willing to invest to increase the benefits by one unit.

The main advantage of the net benefit over other possible forms of utility function (and the main reason for its widespread use) is that it has a fixed form, once the variables (e, c) are defined. Moreover, the net benefit is linear in (e, c), which makes for a simple interpretation and easy calculations. Nevertheless, the use of the net benefit presupposes that the decision maker is risk neutral, which is by no means always appropriate in health policy problems.²⁴ We consider this in more details in Section 7.

When the net benefit is used as utility function, cost-effectiveness analysis focuses on

EIB = E [k Δ_{e} - Δ_{c}] = k E [Δ_{e}] - E [Δ_{c}]

where the expectations are now over the subjective distribution of θ .

3.2 Example (continued)

In order to perform the economic analysis, we need to define suitable measures of cost and effectiveness. The total cost associated with each clinical resource can be computed by multiplying the unit cost ψ_h by the number of patients consuming it. For instance, the overall cost of GP visit is $({GP}_{tv}^{(1)} + {GP}_{tv}^{(2)}) \times ψ_{1}$ . If we indicate with N_tvh the total number of individuals consuming the hth resource under intervention t and in group v, we can then extend this reasoning and compute the average population cost under intervention t as

c_{t} : = 1 N \sum_{v = 0}^{1} \sum_{h = 1}^{8} N_{tvh} ψ_{h} .

Similarly, the total QALYs lost due to the occurrence of the relevant outcomes can be obtained by multiplying the number of individuals experiencing them by the weights ω_j. For example, the total number of QALYs lost to influenza infection can be computed as I_tv × ω₁. If we let M_tvj indicate the number of subjects with the jth outcome in intervention t and group v, we can define the population average measure of effectiveness for intervention t as

e_{t} : = 1 N \sum_{v = 0}^{1} \sum_{j = 1}^{7} M_{tvj} ω_{j} .

We ran the model described in Section 2.3 using a MCMC approach (R and WinBUGS/JAGS codes for the statistical and economic analysis are available from the authors on request). Using the results obtained by 1000 simulations of the model after convergence to the posterior distribution has been assessed, we can compute the expected utility ^t, based on the net benefit as utility function for each value of the willingness-to-pay parameter k ∈ [0 − 50 000]. This is then used to identify the optimal intervention.

Since it is likely that the decision maker is not certain about the value of the willingness-to-pay that they are likely to select in a given problem, the analysis is typically performed (and reported) on a grid of reasonable k values as shown in Figure 1, which depicts the EIB for the comparison between vaccination and the status quo.

From the graph, we can identify the break even point, i.e. the value of k for which the optimal decision is modified. In this case, for k ≤ k^* = 20 100, EIB < 0 and therefore maintaining the status quo is the optimal decision. Conversely, for all k > k^* vaccination is the most cost-effective strategy. The value of the break even point corresponds to the ICER and quantifies the point in which the decision maker is indifferent between the two options.

For the sake of simplicity, we can also select for definiteness a threshold value of k = 25 000 (usually suggested by NICE as the reference cost-per-QALY) and replicate the analysis assuming that this is the value used by the decision maker. If the decision maker were willing to set this particular value of k as the budget to allocate for the treatment of the disease under analysis, the EIB for t = 1 vs t = 0 would be 1.23. Vaccination would therefore prove the most cost effective intervention and it should be then selected to replace the status quo.

4 Uncertainty in the decision process

The above analysis shows how, in the Bayesian approach, both individual variations and uncertainty in the value of the parameters are averaged out. From the decision-theoretic point of view, identification of the overall expected utility is all that is needed to reach the best decision given the current state of knowledge available to the decision maker. This point has been argued in the context of health economics.²⁵

However, implementing an intervention is typically associated with some risks such as the irreversibility of investments, and therefore medical decision making can be viewed as a two-stage decision problem.⁴ If gathering additional data to supplement the background information is not an option, the decision maker must choose now whether to keep the standard programme t = 0, or to switch to the new one on the basis of some suitable cost-effectiveness measure of utility (e.g. the net benefit).

However, if deferring the final decision in order first to gather more data is an available option, then the standard intervention t = 0 will typically be maintained while additional evidence ℰ is collected, with the aim of resolving, at least partially, current uncertainty about the parameter θ . Once this evidence is available, the analysis can be updated and the utility for each possible intervention will be based on the new posterior density of the parameters of interest, p(θ^t ∣ , ℰ), which will induce a predictive distribution for some other future outcomes z (generally of the same nature as y). The option of postponing the decision on cost-effectiveness is typically associated with additional sampling costs.

For these reasons, it has been advocated in the literature that health economic evaluations should be subject to some form of Sensitivity Analysis (SA), in order to quantify and qualify the uncertainty underlying the decision process. Formally, SA is defined in risk assessment as the study of ‘how uncertainty in some model output can be apportioned, qualitatively or quantitatively, to different sources of uncertainty in the model input’.²⁶

Various different forms of SA have been recognised in the health economic literature.⁸ Marginalisation is implicit in Bayesian decision-theoretic procedures, such as (1); the relevant input can be represented by the value of the parameters of the model, θ , whereas the output is the future health economic outcomes on some reference unit. The uncertainty in all the random quantities is accounted for by the computation of the expected utilities used to determine the optimal decision, but is not analysed separately.

The second form of SA is Scenario Analysis (sometimes referred to as Deterministic Sensitivity Analysis, DSA). In this case, the experimenter selects a list of interesting values for (some of) the parameters of the model and evaluates the expected outcomes under all these different scenarios. This procedure is easy to implement when the number of parameters involved is relatively small. However, it fails to consider the possible correlation or the underlying uncertainty about the parameters of interest, only focusing on a set of arbitrarily chosen values, regardless of the likelihood of each of them occurring in reality.

5 Probabilistic sensitivity analysis

These limitations can be overcome by Probabilistic Sensitivity Analysis (PSA), a procedure in which all input parameters are considered as random quantities and are therefore associated with a probability distribution that describes the state of science (i.e. the background knowledge of the decision maker). This method is in line with the Bayesian analysis, but, instead of being marginalised out, as required by the decision-theoretic analysis, the uncertainty in the parameters is explicitly analysed by means of suitable indicators.

We acknowledge that recently some research has been devoted to PSA to structural uncertainty,²⁷ which concerns with assuming a probability distribution over a class of possible models (for both parameters and observables) and then produces a result obtained by averaging over the induced posterior distributions. While we reckon that this is a relevant issue, we focus our attention to PSA with respect to parameter uncertainty, i.e. the case in which, given a model, we are concerned with the impact of uncertainty in the parameters on the economic conclusions.

To understand the rationale behind PSA, let us consider a situation in which the information provided by the additional evidence ℰ is so accurate that p( θ ∣ , ℰ) is close to a one-point distribution at the true value: in this case, we shall have effectively learned θ . In other words, the uncertainty on the knowledge domain will be totally resolved.

If we then adopt intervention t, the ‘known-distribution’ expected utility will be

U (θ^{t}) : = E [u (Y, t) | D, E] = \int u (y, t) p (y | θ^{t}) dy .

(5)

Consequently, the overall utility is U^*( θ ) ≔ max_t U(θ^t) and we would choose treatment t = 1 if U(θ¹) > U(θ⁰), or equivalently if IB( θ ) > 0, where

IB (θ) : = U (θ^{1}) - U (θ^{0})

(6)

is the incremental benefit under parameter-pair θ . Note that, similarly to Equation (3), we have

U^{*} (θ) = max {IB (θ), 0} + U (θ^{0}) .

(7)

Obviously, in general we shall not be able to learn the value of θ with certainty. For this reason, IB( θ ) and U(θ^t) remain random quantities, whose current probability distributions are induced by p( θ ∣ ).

The idea behind PSA is to compare the actual decision process, based on the analysis of EIB and (2), to the ideal one, characterised by the (currently unknown) quantities computed in equations (5) and (6). This is done with a view to assessing whether the information provided by the current evidence is sufficient to take a decision on the optimal treatment, or it would be more effective to defer the final decision until after additional evidence ℰ is collected.

Although analytical methods have also been described,^28,29 PSA is typically conducted using a simulation approach.³⁰ For each of a sequence of iterations s = 1,…, S, a value θ _(s) is simulated from the distribution p( θ ∣ ). The decision analysis is then conducted using that specific value as if this were the realised one. By means of this procedure, it is possible to produce a sample from the distribution of U(θ^t), IB( θ ) or any other related random quantity; the resulting variability in the expected utilities and the influence of each component of θ can then be suitably summarised.

5.1 Example (continued)

Table 2 shows the results of the simulation exercise described in Section 3.2. For each of the 1000 iterations, we simulated values for all the parameters (shown in the upper and middle part of Table 2) from the distributions defined in Table 1.

Table 2.

PSA in practice.

s	φ	β₁	β₂	β₃	β₄	β₅	β₆	β₇	ρ₁	γ₁	γ₂	δ	ξ	η
1	0.55	0.06	0.29	0.38	0.01	0.00	0.00	0.10	0.68	0.42	0.81	3.00	0.96	0.91
2	0.41	0.09	0.30	0.41	0.01	0.00	0.00	0.10	0.68	0.42	0.81	5.00	0.95	0.91
3	0.56	0.06	0.16	0.41	0.01	0.00	0.00	0.17	0.79	0.42	0.82	7.00	0.95	0.90
4	0.56	0.09	0.48	0.41	0.01	0.00	0.00	0.01	0.63	0.42	0.82	9.00	0.95	0.90
5	0.60	0.07	0.29	0.41	0.01	0.00	0.00	0.15	0.69	0.42	0.81	10.00	0.95	0.90
6	0.35	0.05	0.46	0.40	0.02	0.00	0.00	0.07	0.68	0.42	0.82	7.00	0.96	0.90
…							…							…
1000	0.44	0.08	0.22	0.39	0.01	0.00	0.00	0.12	0.66	0.42	0.81	9.00	0.95	0.90
s	λ	ψ₁	ψ₂	ψ₃	ψ₄	ψ₅	ψ₆	ψ₇	ψ₈	ω₁	ω₄	ω₅	ω₆	ω₇
1	4.99	20.58	2192.76	8.34	8.37	37.73	2.97	1.56	1.25	6.73	3.78	6.99	16.03	0.59
2	1.47	23.17	2702.32	5.64	6.40	33.98	3.57	1.47	0.61	2.78	5.08	9.70	14.72	0.47
3	1.64	12.30	2734.48	9.45	12.44	59.77	4.93	1.40	0.94	4.71	7.03	5.89	11.97	0.60
4	2.90	20.88	2485.90	8.04	10.25	43.28	3.00	1.21	0.84	3.63	5.38	5.34	18.83	0.69
5	2.76	21.87	2445.67	5.00	12.12	48.94	2.41	1.10	0.63	4.65	5.28	8.77	22.00	0.78
6	2.91	22.25	1163.62	5.18	7.51	41.44	3.81	1.38	0.67	5.29	9.45	6.02	17.81	0.51
…							…							…
1000	2.21	18.96	3311.86	7.66	10.55	37.06	5.42	1.16	1.04	4.71	6.29	8.05	17.37	0.57

s	U(θ⁰)	U(θ¹)	U( θ )	IB( θ )	OL( θ )	VDI( θ )
1	−36.58	−38.72	−36.58	−2.14	2.14	−1.75
2	−27.93	−27.67	−27.67	0.25	—	7.15
3	−28.03	−33.37	−28.03	−5.34	5.34	6.80
4	−53.28	−47.14	−47.14	6.15	—	−12.31
5	−43.58	−40.40	−40.40	3.18	—	−5.58
6	−42.37	−33.09	−33.09	9.29	—	1.74
…			…			…
1000	−35.23	−32.60	−32.60	2.64	—	2.23
Average	⁰ = −36.05	¹ = −34.83	V^* = −32.41	EIB = 1.23	EVPI = 2.41	EVPI = 2.41

Notes: For each iteration, we first simulate a value for the parameters θ from the distributions described in Table 1. These are shown in the upper and mid part of this table. Then (lower part of this table), for each intervention and for each iteration s of the model the induced expected utilities are calculated, according to (8) and (9), and using a fixed value of k = 25 000; for each iteration, the maximum expected utility (i.e. of the optimal intervention) is typeset in italics. The last two columns of the lower part of this table are computed according to (10) and (12). For each column, the average is computed over the rows (model simulations).

Considering the net benefit as utility function, we can re-write (5) and (6), respectively, as

U (θ^{t}) = k E [e | θ^{t}] - E [c | θ^{t}]

(8)

and

IB (θ) = k Δ_{e} - Δ_{c} .

(9)

and use the simulated values for the parameters to calculate the expected utility of option t that would be obtained if the uncertainty on the parameters were resolved to the specific values simulated, which are shown in the lower part of Table 2.

While this process can be repeated for each value of k, in Table 2 we show again the computations for the reference value of k = 25 000. The last row of the table reports the average values computed over all the simulations of the model, while the last two columns are described in details in Section 6.2.

6 Summarising the results of PSA

Much of the recent theoretical work has been devoted specifically to the issue of reporting the results of PSA using suitable summary measures.^13,31–35 As suggested earlier, while the process of marginalisation performed computing the expected utilities ^t provides the ‘best’ decision given the current data, the essence of PSA is to use the induced distributions for the health economic indicators to qualify the extent to which uncertainty impacts on the decision process. We next review the main indicators used to this end, clarifying the basic differences in their nature.

6.1 Cost-effectiveness acceptability curves

In health economic evaluations, it is common to summarise the results of PSA by means of the cost-effectiveness acceptability curve (CEAC),³⁶ defined as

CEAC = ⪻ (IB (θ) > 0) .

If the net benefit is used as utility function, this can be re-expressed as CEAC = Pr(kΔ_e − Δ_c > 0), which depends on the willingness-to-pay parameter k. When EIB > 0, i.e. the optimal decision is treatment 1, this is the probability that learning the value of θ (resolving the uncertainty on the parameters) would not change that decision.

For the example of Section 2.1, Figure 2 shows the CEAC, again upon varying the value of the parameter k in the range [0; 50000]. For relatively small values of k the probability of cost-effectiveness is low, indicating higher uncertainty in the actual cost-effectiveness of the vaccination strategy. For k ≈ 20 000, it reaches a value of 0.5 (when the uncertainty as to what is the most cost-effective intervention is maximum).

Figure 2 seems to suggest that, for relatively low values of the willingness-to-pay parameter, there is substantial uncertainty as to whether t = 1 is in fact the optimal intervention; for large values of k, the probability that choosing it is the ‘correct’ decision is increasingly higher and is 0.53 for k = 25 000. This figure can be deduced from (the complete version of) Table 2 as the proportion of simulations for which IB( θ ) > 0 — cf. Equation (6).

By their very nature, CEACs provide a simple synthesis of the uncertainty about the cost-effectiveness of a given intervention³⁷ and have been widely used in the health economics literature.^8,9,38–40 The main advantage of CEACs is that they allow simple summarisation of the probability of cost-effectiveness upon varying the willingness-to-pay parameter, effectively performing a DSA on k. This circumstance proved to be particularly useful in presenting the results of economic analysis, as decision makers are often not ready to commit to a single value of k (i.e. a single utility function) prior to the analysis being performed.

Despite their wide use, some critical limitations have been pointed out, the main one being that CEACs do not contain a decision rule. For instance, they can only address the problem of how likely it is that resolving parameters uncertainty will change the optimal decision.⁴¹ However, no explicit reference is made to the possible change in the payoffs. More recently, it has been suggested that very different distributions for the IB can produce the same value of the CEAC, which makes it difficult to interpret and might lead to incorrect conclusions for policy makers.²⁴ Finally, CEACs are concerned only with currently available information, but do not consider explicitly the possibility of gathering additional evidence.

Consequently, by means of CEACs only a partial evaluation of the overall decision process is provided. For this reason, if sensitivity analysis is performed in the context described above (i.e. with the possibility of deferring the decision), then the use of CEACs is clearly not ideal.

6.2 The value of information

A purely decision-theoretic approach to PSA, avoiding the shortcomings of CEACs, is based on the VI analysis,⁴² an increasingly popular method in health economic evaluations.^4,41,43–48 In this approach, we compare the overall value of the decision process in the ideal scenario, represented by U^*( θ ), to that obtained in the actual evaluation, ^*.

The value of obtaining information on θ is defined as

VI (θ) : = U^{*} (θ) - U^{*},

(10)

which for each value of θ quantifies the difference in the utilities produced by the information on the parameters.

Again, since the value of U^*( θ ) in general cannot be determined with certainty, we synthesise its distribution considering the expected value of ‘perfect’ information (EVPI)

EVPI : = E [VI (θ)] = \int VI (θ) p (θ | D) d θ = V^{*} - U^{*},

(11)

where

V^{*} : = E [U^{*} (θ)] = \int U^{*} (θ) p (θ | D) d θ

is the expectation of the overall ‘known distribution’ utility U^*( θ ). Notice, however, that the additional information that may become available need not be ‘perfect’. In other words, we might not be able of learning the value of a parameter perfectly, although new data will generally make our estimation more precise, therefore leading to a less variable posterior distribution. Extensions of this argument are sometimes developed in terms of expected value of sample information, EVSI.⁴⁶

If τ = arg max_t^t is the intervention associated with the overall maximum expected utility, for each value of θ we can consider

OL (θ) : = U^{*} (θ) - U (θ^{τ}),

(12)

the opportunity loss derived by choosing the alternative associated with the highest overall expected utility, instead of the one associated with the highest known distribution utility. As is easy to see, the average value of OL( θ ) coincides with the value of EVPI:

E [OL (θ)] = \int [U^{*} (θ) - U (θ^{τ})] p (θ | D) d θ = \int U^{*} (θ) p (θ | D) d θ - \int U (θ^{τ}) p (θ^{τ} | D) d θ^{τ} = V^{*} - U^{*} = EVPI .

While the value of information VI( θ ) can take on negative values, OL( θ ) is necessarily non-negative. Consequently, by a simple application of Jensen's inequality, it can be proved that the EVPI is a non-negative quantity as well.

The EVPI places an upper limit to the amount that we would be willing to pay to obtain any information, perfect or imperfect, about θ and, by construction, it measures the weighted average opportunity loss induced by the decision that we make based on the EIB, the weight being the probability of incurring that loss. Therefore, this measure gives us an appropriately integrated indication of: (a) how much we are likely to lose if we take the ‘wrong’ decision, and (b) how likely it is that we take it, as is easily appreciated using Equations (3) and (7) to re-express Equation (11) as

EVPI = E [U^{*} (θ) - U^{*}] = E [max {IB (θ), 0}] - max {EIB, 0} = E [IB (θ) | IB (θ) > 0] \times ⪻ (IB (θ) > 0) - max {EIB, 0} = E [IB (θ) | IB (θ) > 0] \times CEAC - max {EIB, 0}

(the expectations are all taken with respect to the joint distribution of θ ).

If EIB > 0, then selecting the treatment t = 0 just because (from the analysis of CEAC) there is a large variability in IB( θ ) will impose unnecessary losses on society, as patients will not benefit from a potentially cost-effective treatment.²⁵ In contrast, the EVPI analysis provides the decision maker with a rational procedure overcoming this problem. If the large variability in IB( θ ) is associated with low cost for additional research, then the decision maker can postpone the decision, or perhaps select the treatment t = 1 only for a subset of the population.

The VI analysis is sometimes considered as a separate methodology, which can be performed independently on PSA. However, in our view, for the reasons explained above, it is a fundamental part of the sensitivity analysis process and is in line with the objective of identifying and quantifying the impact of parameters uncertainty on the decision process. Therefore, we view it as the proper method to perform PSA.

Figure 3 shows the analysis of the individual (i.e. per patient) expected VI as a function of the willingness-to-pay parameter k for the running example. The EVPI changes its shape around the break even point k = 21 100, since the optimal decision is reversed beyond that threshold.

As the value of k increases from 0, the value of reducing uncertainty becomes increasingly larger. Just after the break even point, it remains almost constant. Even for values of k where there is higher uncertainty in the optimal decision, the absolute magnitude of the patient-specific EVPI is fairly small (compared to the values of the payoffs) in this case. This implies that uncertainty in the parameters does not have a dramatic impact, contrary to what might be suggested by the CEAC analysis (which indicated a probability of cost-effectiveness of ‘just’ 0.53 for k = 25 000).

7 Including a risk aversion parameter in the net benefit

The previous analysis can be extended to consider a more general form for the utility function, to include explicitly the possibility that the decision maker is risk-averse. In this situation, before selecting a new intervention, the decision maker requires lower levels of uncertainty as to whether it will turn out to be the most cost-effective alternative. This has obvious implications on the level and quality of the evidence used to reach the decision.

We consider again the model for the infectious disease presented in Section 2.1, but instead of the simple linear utility function of Equation (4) used so far, we now define a more complex form

u (y, r) = 1 r [1 - \exp (- ry)],

(13)

where r > 0 represents a parameter of risk aversion²² — the higher the value of r, the more risk-averse the decision maker — and y ≔ ke − c now denotes the monetary net benefit.

Now, in line with the analysis of Section 5, the quantity that we should investigate for PSA of uncertainty in the parameters, the known distribution utility of Equation (5), has the more complex form

U (θ^{t}) = \int 1 r [1 - \exp (- ry)] p (y | θ^{t}) d y = 1 r [1 - M_{Y | θ^{t}} (- r)]

where M_Y∣θ^t(−r)≔E[exp(−rY)] is the moment generating function of the random quantity Y with respect to the distribution p(y ∣ θ^t), evaluated at the point −r. Notice that in this case, the utility function is no longer linear in (e, c) and will be in general not straightforward to compute analytically. However, it will typically be available via a simulation approach, such as the MCMC procedure we are using here.

7.1 Example (continued)

The incremental benefit is now a function of two parameters, the willingness-to-pay k and the risk aversion r. Figure 4 shows the EIB for various values of r as a function of k, and highlights the important effect on the overall decision process of including the risk propensity of the decision maker.

When r → 0, the decision maker is risk-neutral, and EIB is identical with that based on the monetary net benefit utility function. However, the break even point (i.e. the value of k for which vaccination becomes the best option, producing a positive EIB) does vary upon changing the value of r. As r increases from 0, EIB becomes increasingly non linear.

Figure 5 shows the analysis of the EVPI as a function of k and r; again, for r → 0 we retrieve the same analysis of Figure 3. When the decision maker's risk-aversion is taken into account, the expected VI becomes generally higher since now the decision maker is less prepared to commit to a given intervention.

While the analysis of the EVPI is appropriately sensitive to the choice of r, it is possible to prove with standard probability calculus that using the utility function of Equation (13) the CEAC is independent of the value of r. This is essential because r is a multiplicative scale parameter and as such, while it does modify the shape of the distributions of IB (and therefore the expected values used to compute the EVPI), it does not affect the probability that IB is positive. Consequently, irrespective of the risk propensity of the decision maker, if PSA is performed using the CEAC the results are the same (and identical with that depicted in Figure 2). Again, this feature is not ideal, as we would expect different decision makers with different attitude towards risk to arrive at different results.

8 Conclusions

We have reviewed the methodology of sensitivity analysis in health economics. Recent years have witnessed the establishment of formal statistical decision-theoretic foundations in this field, along with the increasing awareness of the relevance of monitoring uncertainty in the decision process.

Our standpoint is that PSA is an important component of any health economic analysis; however, we also believe that it should be consistent with the precepts of formal decision-theory, that is it should only concern those aspects that turn out to be crucial in determining the optimal decision. Care should be taken to consider the appropriate context, e.g. whether or not further information could be gathered.

Standard methodologies exist that allow the incorporation of risk aversion in the definition of the utility function. These should be exploited to represent more precisely the objectives of public decision makers seeking to identify an optimal strategy.

Footnotes

Acknowledgements

The authors are most grateful for the support for this research that was provided by the Leverhulme Foundation and the Economic and Social Research Council to University College London. No potential conflict of interest is to be reported. This work was partly undertaken at UCLH/UCL who received a proportion of funding from the Department of Health's NIHR Biomedical Research Centres funding scheme.

Glossary

References

Berwick

. Broadening the view of evidence-based medicine. Int J Qual Safe Health Care 2005 14: 315–316.

Lean

Mann

Hoek

Elliot

Schofield

. Translational research: from evidence-based medicine to sustainable solutions for public health problems. Brit Med J 2008, pp. 337.

Weinstein

Stason

. Foundations of cost-effectiveness analysis for health and medical practices. New Engl J Med 1977; 296(13): 716–721.

Briggs

Schulpher

Claxton

. Decision modelling for health economic evaluation, Oxford, UK: Oxford University Press, 2006.

Willan

Briggs

. The statistical analysis of cost-effectiveness data, Chichester, UK: John Wiley and Sons, 2006.

O'Hagan

Stevens

. A framework for cost-effectiveness analysis from clinical trial data. Health Econ 2001; 10: 303–315.

O'Hagan

Stevens

Montmartin

. Bayesian cost effectiveness analysis from clinical trial data. Stat Med 2001; 20: 733–753.

Parmigiani

. Modeling in medical decision making, New York, NY: John Wiley and Sons, 2002.

Spiegelhalter

Best

. Bayesian approaches to multiple sources of evidence and uncertainty in complex cost-effectiveness modelling. Statistics in Medicine 2003; 22: 3687–3709.

10.

Spiegelhalter

Abrams

Myles

. Bayesian Approaches to Clinical Trials and Health-Care Evaluation, Chichester, UK: John Wiley and Sons, 2004.

11.

Lindley

. Making Decisions, 2nd edition. London, UK: John Wiley and Sons, 1985.

12.

Baio

Russo

. A decision-theoretic framework for the application of cost-effectiveness analysis in regulatory processes. Pharmacoeconomics 2009; 27(8): 645–655.

13.

Claxton

Schulpher

McCabe

. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Economics 2005; 14: 339–347.

14.

de Finetti

. Theory of Probability 1974; Vol. 1, New York, NY: John Wiley and Sons.

15.

Turner

Wailoo

Cooper

Sutton

Abrams

Nicholson

. The cost-effectiveness of influenza vaccination of healthy adults 50-64 years of age. Vaccine 2006; 24: 1035–1043.

16.

Nichol

Mallon

KPM

. Cost benefit of influenza vaccination in healthy, working adults: an economic analysis based on the results of a clinical trial of trivalent live attenuated influenza virus vaccine. Vaccine 2003; 21(17-18): 2207–2217.

17.

Demicheli

Rivetti

Deeks

Jefferson

. Vaccines for preventing influenza in healthy adults (Cochrane Review). Vol. Issue 4, Cochr Lib, 2004. 4.

18.

Gross

Hermogenes

Sacks

Lau

Levandowski

. The efficacy of influenza vaccine in elderly persons — a meta analysis and review of the literature. Ann Intern Med 1995; 123(7): 518–527.

19.

Kind

Brooks

Rabin

. EQ-5D concepts and methods: a developmental history, Dordrecht, The Netherlands: Springer, 2005.

20.

Loomes

McKenzie

. The use of QALYs in health care decision making. Soc Sci Med 1989; 28: 299–308.

21.

Robert

. The Bayesian choice, 2nd edition. New York, NY: Springer Verlag, 2001.

22.

Raiffa

. Decision Analysis – Introductory Lectures on Choices under Uncertainty, Reading, MA: Addison Wesley, 1968.

23.

Stinnett

Mullahy

. Net health benefits: a new framework for the analysis of uncertainty in cost effectiveness analysis. Med Decis Making 1998; 18(Suppl): S68–S80.

24.

Koerkamp

Hunink

Stijnen

Hammitt

Kuntz

Weinstein

. Limitations of Acceptability Curves for Presenting Uncertainty in Cost-Effectiveness Analyses. Medical Decision Making 2007; 27(2): 101–111.

25.

Claxton

. The irrelevance of inference: a decision-making approach to stochastic evaluation of health care technologies. J Health Econ 1999; 18: 342–364.

26.

Saltelli

Tarantola

Campolongo

Ratto

. Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models, Chichester, UK: John Wiley and Sons, 2004.

27.

Jackson

Thompson

Sharples

. Accounting for uncertainty in health economic decision models by using model averaging. J Royal Stat Soc A 2009; 172(2): 383–404.

28.

Oakley

O'Hagan

. Probabilistic Sensitivity Analysis of Complex Models: A Bayesian Approach. Journal of the Royal Statistical Society B 2004; 66(3): 751–769.

29.

Oakley

. Decision-Theoretic Sensitivity Analysis for Complex Computer Models. Technometrics 2009; 51(3): 121–129.

30.

Doubilet

Begg

Weinstein

Braun

McNeil

. Probabilistic sensitivity analysis using Monte Carlo simulation. A practical approach. Med Decis Making 1985; 5: 157–177.

31.

Briggs

Goeree

Blackhouse

O' Brien

. Probabilistic Analysis of Cost-Effectiveness Models: Choosing between Treatment Strategies for Gastroesophageal Reflux Disease. Medical Decision Making 2002; 4: 290–308.

32.

Parmigiani

. Measuring uncertainty in complex decision analysis models. Stat Methods Med Res 2002; 11: 513–537.

33.

O'Hagan

Mc Cabe

Akehurst

Brennan

Briggs

Claxton

. Incorporation of uncertainty in health economic modelling studies. Pharmacoeconomics 2004; 23: 539–536.

34.

Griffin

Claxton

Hawkins

Sculpher

. Probabilistic Analysis and Computationally Expensive Models: Necessary and Required? Value Health 2006; 9: 244–252.

35.

O'Hagan

Stevenson

Madan

. Monte Carlo probabilistic sensitivity analysis for patient level simulation models: efficient estimation of mean and variance using ANOVA. Health Economics 2007; 16: 1009–1023.

36.

van Hout

Gordon

Rutten

. Costs, effects and C/E ratios alongside a clinical trial. Health Economics 1994; 3: 309–319.

37.

Fenwick

Claxton

Schulpher

. Representing uncertainty: the role of cost effectiveness acceptability curves. Health Economics 2001; 10: 779–787.

38.

Briggs

. Handling Uncertainty in Cost-Effectiveness Models. Pharmacoeconomics 2000; 22: 479–500.

39.

O'Hagan

Stevens

Montmartin

. Inference for the cost-effectiveness acceptability curve and cost-effectiveness ratio. Pharmacoeconomics 2000; 17: 339–349.

40.

O'Brien

Briggs

. Analysis of uncertainty in health care cost-effectiveness studies: an introduction to statistical issues and methods. Stat Methods Med Res 2002; 11: 455–468.

41.

Felli

Hazen

. A Bayesian approach to sensitivity analysis. Health Economics 1999; 8: 263–268.

42.

Howard

. Information value theory. IEEE Trans Syst Sci Cybernetics SCC-2 1966; 2(1): 22–26.

43.

Felli

Hazen

. Sensitivity analysis and the expected value of perfect information. Medical Decision Making 1998; 18: 95–109.

44.

Claxton

. Bayesian approaches to the value of information: implications for the regulation of new pharmaceutical. Health Economics 1999; 8: 269–274.

45.

Claxton

Neumann

Araki

Weinstein

. Bayesian Value-Of-Information Analysis. Int. J. Technol. Assess. Health Care 2001; 17: 38–55.

46.

Ades

Claxton

. Expected Value of Sample Information Calculations in Medical Decision Modeling. Medical Decision Making 2004; 24: 207–227.

47.

Brennan

Kharroubi

. Efficient Computation of Partial Expected Value of Sample Information Using Bayesian Approximation, UK: Department of Probability and Statistics, University of Sheffield, 2005. 560/05.

48.

Fenwick

Palmer

Claxton

Sculpher

Abrams

Sutton

. An Iterative Bayesian Approach to Health Technology Assessment: Application to a Policy of Preoperative Optimization for Patients Undergoing Major Elective Surgery. Medical Decision Making 2006; 26: 480–496.