Addressing Adoption and Research Design Decisions Simultaneously

Abstract

Methods to estimate the cost-effectiveness of technologies are well developed with increasing experience of their application to inform adoption decisions in a timely way. However, the experience of using similarly explicit methods to inform the associated research decisions is less well developed despite appropriate methods being available with an increasing number of applications in health. The authors demonstrate that evaluation of both adoption and research decisions is feasible within typical time and resource constraints relevant to policy decisions, even in situations in which data are sparse and formal elicitation is required. In addition to demonstrating the application of expected value of sample information (EVSI) in these circumstances, the authors examine and carefully distinguish the impact that the research decision is expected to have on patients while enrolled in the trial, those not enrolled, and once the trial reports. In doing so, the authors are able to account for the range of opportunity cost associated with research and evaluate a number of research designs including length of follow-up and sample size. The authors also explore the implications for research design of conducting research while the technology is approved for widespread use and whether approval should be withheld until research reports. In doing so, the authors highlight the impact of irrecoverable opportunity costs when the initial costs of a technology are compensated only by later gains in health outcome.

Keywords

Bayesian decision theory expected value of information research design cost-effectiveness analysis

Appropriate decisions about health care technologies require a number of questions to be addressed: 1) Is the technology expected to be cost-effective based on existing evidence? 2) Is additional evidence required to support the use of the technology? 3) If so, what type of evidence would be most valuable, and which research designs would be worthwhile? Because adoption of a technology may affect the prospects of being able to conduct certain types of research, both adoption and research decisions need to be addressed in a way that is consistent with the objectives and resource constraints of the health care system.¹

Methods to estimate the cost-effectiveness of technologies are well developed,^2,3 with increasingly long experience of their application to inform adoption decisions in a timely way. This is particularly well established in the process of appraisal used by the National Institute for Health and Clinical Excellence (NICE), which issues guidance to the National Health Service (NHS) in England and Wales on the use of technologies.⁴ This increasing application of cost-effectiveness analysis to inform adoption and reimbursement decisions is also reflected internationally.^5,6

However, the experience of using similarly explicit methods to inform research decisions is less well developed despite appropriate methods being available with an increasing number of applications,⁷ some of which have been commissioned to directly inform research decisions.⁸ Expected value of information (EVI) has a firm foundation in statistical decision theory,^9–14 with successful applications in other areas of research.^15–17 Over recent years, methods have developed further and have found increasing application in the evaluation of health care technologies.^2,18–22 However, most have been restricted to an analysis of the expected value of perfect information.^7,8,23–26 The application of expected value of sample information (EVSI) analysis and the evaluation of alternative research designs has been more limited,^27–32 with very few attempts to include such analysis within the type of cost-effectiveness analysis conducted to inform adoption decisions by bodies such as NICE.

In this article, we demonstrate that evaluation of both adoption and research decisions, including alternative research designs, is feasible in a timely way, even in situations in which data are sparse and formal elicitation is required. In addition, we examine and carefully distinguish the impact that the research decision is expected to have on patients enrolled in the trial, not enrolled in the trial, and once the trial reports. In doing so, we are able to account for the range of opportunity cost associated with the research decision and evaluate a number of research designs including length of follow-up and sample size. We also explore the implications for research design of conducting research while the technology is approved for widespread use and whether approval should be withheld until research reports. In doing so, we highlight the impact of irrecoverable opportunity costs when the initial costs of a technology are compensated only by later gains in health outcome.

Enhanced External Counterpulsation for Angina

Enhanced external counterpulsation (EECP) is a noninvasive procedure used to treat chronic stable angina. It involves wrapping inflatable pressure cuffs around a patient’s calves, lower thighs, and upper thighs. The cuffs are inflated and deflated to increase blood flow to the coronary arteries and decrease peripheral vascular resistance and cardiac workload.³³ The primary goal of the therapy is the symptomatic relief of angina symptoms.^34–36

The National Institute for Health Research Health Technology Assessment program in England and Wales, which commissions assessment reports to inform the NICE appraisal process, identified EECP as an important topic and commissioned a report to examine the clinical effectiveness and cost-effectiveness of EECP as an adjunct to standard therapy in patients with stable angina. Although the topic was not ultimately considered by NICE, it was commissioned in the same way and with the same resources as other assessment reports that inform NICE guidance. The full details are available elsewhere.³⁷ In summary, the review of clinical effectiveness identified only 1 randomized controlled trial (RCT; the MUST-EECP trial^34,38) comparing active with inactive EECP. This trial showed evidence of improved health-related quality of life (HRQoL) from EECP (see Table 1). However, follow-up was limited to 12 mo, so the degree to which improvement in HRQoL from EECP is sustained beyond 12 mo is uncertain.

Table 1

Parameters of the Decision Model for EECP Plus Standard Therapy Compared with Standard Therapy Alone

Parameter	Mean Value	Prior Distribution	Source
Baseline patient characteristics
Age, y	64	Fixed	MUST-EECP trial³⁸
Gender, % male	92	Fixed	MUST-EECP trial³⁸
HRQoL improvement from baseline to 1 y after treatment
EECP relative to standard therapy	0.0717	Beta (α = 3.641, β = 47.139)^a	MUST-EECP trial³⁸
Probability of sustaining HRQoL benefits in subsequent years^b
Year 2	0.757	Beta (α = 8.028, β = 2.577)	Expert elicitation³⁷
Year 3	0.742	Beta (α = 5.589, β = 1.943)	Expert elicitation³⁷
Year 4	0.719	Beta (α = 4.413, β = 1.726)	Expert elicitation³⁷
Year 5 onward	0.719	Beta (α = 4.413, β = 1.726)	Expert elicitation³⁷
Probability of repeat EECP sessions per year (decreased exponentially)	0.099	Beta (α = 194, β = 884)	Michaels and others⁴⁵
Resource use and unit costs
Capital cost of EECP machine per annum (10-y lifetime)	£10 822	Fixed	K. Miles, personal communication, 2008
Equipment replacement costs
1 set of cuffs per year	£139	Fixed	Vasogenics^c
1 set of hoses per year	£76	Fixed	Vasogenics^c
Pleth per every 2 y	£53	Fixed	Vasogenics^c
Consumables per patient			Vasogenics^c
Ultrasound scan	£75	Fixed	Vasogenics^c
Trousers	£16	Fixed	Vasogenics^c
Gel	£8	Fixed	Vasogenics^c
ECG electrodes	£110	Fixed	Vasogenics^c
Staffing and overhead costs per patient (based on 12 patients per year)	£3214	Fixed	W. Sheedy, personal communication, 2008
Total cost of EECP per patient	£4347	Fixed

Note: EECP = enhanced external counterpulsation; HRQoL = health-related quality of life; ECG = electrocardiogram.

It is assumed that EECP always improves HRQoL in the first year after treatment compared with standard therapy based on the results of MUST-EECP, but this improvement may not be sustained in subsequent years.

Refers to the probability of sustaining benefits during year t assuming that benefits have been sustained in year t − 1.

Vasogenics price list (effective from April 2007).

To establish the uncertainty associated with possible durations of treatment effect, formal elicitation of expert clinical judgment was undertaken.³⁹ Five experts with experience and knowledge of EECP in the United Kingdom independently completed an Excel-based exercise that elicited their judgments about the likelihood of sustaining HRQoL benefits from EECP in subsequent years after the first year.^a Because the uncertainty associated with any judgment is critical, a frequency chart format, in which experts place 20 crosses on a frequency chart to represent a distribution, was adopted.⁴⁰ The results from each expert were linearly pooled, with equal weight,⁴¹ providing the probability of continuing to respond to treatment in each subsequent year. The uncertainty associated with these pooled estimates was characterized by fitting beta distributions to pooled responses. Table 1 summarizes the mean values and distributions for the probability of sustaining HRQoL benefits in each year.

A simple probabilistic decision analytic model was used to estimate the expected cost-effectiveness of EECP and associated uncertainty. The decision model was structured to capture the HRQoL benefits and costs associated with EECP up to a period of 12 mo and to project these benefits to a lifetime horizon relevant to this patient population. Three health states were defined: 1) a “responders” state represented patients who continued to sustain HRQoL benefits from EECP in each yearly cycle, 2) a “nonresponders” state represented patients who lost their initial HRQoL benefits from treatment and reverted back to baseline HRQoL, and 3) “dead” represented deaths from disease-specific cardiovascular causes⁴² and other causes informed by UK life tables.⁴³ In the first year after treatment, it was assumed that patients achieved the HRQoL benefits reported in the 12-mo follow-up of MUST-EECP.³⁸ After 12 mo, the proportion of patients responding to treatment was based on the results of the elicitation exercise. After year 4, experts did not expect the probability of sustained response to be different in subsequent years. Therefore, a beta distribution, equivalent to that fitted to pooled year 4 responses, was assigned to year 5 onward.

A typical course of EECP involves a total of 35 h of therapy.⁴⁴ Some patients require a repeat or top-up procedure involving several additional sessions, which are generally given to help sustain the long-term benefits of treatment. Evidence suggests that 18% of patients require repeat EECP within 2 y of the initial course of treatment.⁴⁵ This was converted to an annual probability and decreased exponentially over time to reflect the diminishing attempts to continue to retreat if the patient is not responding (see Table 1). Resource use and costs associated with EECP were based on published unit costs. The cost for repeat procedures was based on an average of 10 additional sessions. Table 1 summarizes the key parameters for the decision model, the distributions assigned, and the sources of evidence used.

Is EECP Expected to Be Cost-Effective?

The expected incremental difference in lifetime costs and quality-adjusted life-years (QALYs) of EECP relative to standard therapy alone is £4744 and 0.2446 QALYS, respectively, giving an incremental cost-effectiveness ratio (ICER) of £19 392 (see Table 2). Therefore, EECP can be regarded as cost-effective and be approved for use, based on current evidence, if this ICER is lower than the cost-effectiveness threshold.

Table 2

Cost-Effectiveness Results of EECP Plus Standard Therapy Relative to Standard Therapy Alone

	Cost-Effectiveness at Thresholds
	£10 000 per QALY	£20 000 per QALY	£30 000 per QALY
Net benefits
EECP	£70 071	£144 887	£219 702
Standard therapy	£72 369	£144 738	£217 107
Probability EECP is cost-effective	0.067	0.428	0.700
Maximum net benefits under
Current information	£72 369	£144 887	£219 702
Perfect information	£72 453	£145 871	£220 143
EVPI per patient	£84	£984	£441
Population EVPI	£15 912 881	£185 743 378	£83 219 992
Population EVPPI
θ₁: HRQoL benefits in first year	£5 888 350	£162 537 654	£62 291 596
θ₂: Probability of sustaining HRQoL benefits in subsequent years	£396 787	£77 202 436	£2 716 506
θ₃: Probability of top-up sessions	£0	£4735	£0

Note: EECP = enhanced external counterpulsation; EVPI = expected value of perfect information; QALY = quality-adjusted life-year. Incremental mean costs =£4744; incremental mean QALYs = 0.2446; ICER for EECP = £19,392.

Equivalently, cost-effectiveness of an alternative, j(standard therapy, j = 0, or EECP, j = 1), can also be expressed as the difference between the expected health gains (Q_j), expressed in monetary terms using the cost-effectiveness threshold (k) and the expected costs (C_j) or the net monetary benefit (NB_j = k.Q_j – C_j).^46,47 Based on current evidence, the alternative that generates the highest expected net benefit (NB_*) should be selected,^1,48,49 that is,

N B * = \max_{j} E_{θ} N B (j, θ),

where θ represents the set of all parameters. The expected net benefit per patient for 3 cost-effectiveness thresholds is reported in Table 2. A decision maker should reject EECP at a threshold of £10 000 per QALY because standard therapy is expected to provide the highest net benefit (NB₀ = £72 369 ≥ NB₁ = £70 071). In contrast, EECP should be approved at the thresholds adopted by NICE of £20 000 and £30 000 per QALY. The difference in net benefit between EECP and standard care at these thresholds (£149 and £2595, respectively) represents the expected benefit of immediate approval of EECP based on existing evidence or, alternatively, the opportunity cost per patient of withholding approval of EECP if, for example, additional research is required that could not be conducted if it is approved.

Although EECP is expected to be cost-effective at thresholds of £20 000 and £30 000 per QALY, a decision to approve based on existing evidence is uncertain, with error probabilities of 0.572 and 0.3, respectively. Therefore, approval of EECP based on existing evidence runs the risk that it does not in fact offer the highest net benefit and that standard care would have been better.

Is More Evidence Required?

If the uncertainty could be immediately resolved, that is, with perfect information, the decision maker could select the alternative that maximizes the net benefit for each possible value of θ (max_j NB(j,θ)). However, the true values of θ are unknown, so the expected value of a decision taken when uncertainty is resolved is found by averaging these maximum net benefits over the joint distribution of θ,

E_{θ} \max_{j} N B (j, θ) .

This represents the maximum expected net benefit per patient that can be achieved. The maximum value of conducting further research to resolve uncertainty is the difference in expected net benefit between perfect and current information, that is, the difference between (2) and (1), or the expected value of perfect information (EVPI). The EVPIs per patient at thresholds of £10 000, £20 000, and £30 000 per QALY are £84, £984, and £441, respectively (Table 2).

The information generated by research can be used to inform treatment decisions for current and future patients. Therefore, the maximum societal value of further research requires an estimate of the population of patients who can potentially benefit from the additional information. A finite time horizon (T = 10 y) for the value of additional evidence about EECP is chosen as a proxy for how the entry of new technologies and changes in prices are likely to affect the value of evidence in future periods.²⁶ The British Heart Foundation estimates that the prevalence of angina in the United Kingdom is just under 1.1 million⁵⁰ and the annual incidence is about 95 000.⁵⁰ Assuming 10% of angina patients can potentially benefit from EECP (Michael Chester, personal communication, 2008), this implies a prevalent population (P₀) at time t = 0 of 109 800 and an annual incidence (I_t) of 9500 in each future time period t. Therefore, the EVPI for the population is given by^b

(P_{0} + \sum_{t = 1}^{T = 10} \frac{I_{t}}{{(1 + r)}^{t}}) (E_{θ} \max_{j} N B (j, θ) - N B *),

where the expected net benefits for future incident populations are discounted at a rate r = 3.5% in common with the costs and health benefits that determine NB_j. Population EVPI is reported in Table 2 for 3 thresholds.

What Type of Evidence Is Required?

The population EVPI at thresholds of £20 000 and £30 000 per QALY (the range used by NICE) is very likely to exceed the costs of further research (£186 million and £83 million, respectively). Therefore, it is important to establish which sources of uncertainty are most important, which evidence might be most valuable, and what type of research might be required. There are 3 groups of uncertain parameters that determine the expected cost-effectiveness of EECP: the HRQoL benefits from EECP in the first year after treatment (θ₁), the probability of sustaining HRQoL benefits in subsequent years (θ₂), and the probability of requiring repeat (or top-up) EECP sessions (θ₃).

The expected maximum net benefit if the uncertainty associated with each of these parameters could be resolved in turn can be established in a similar way to (2) above. For example, if uncertainty associated with θ₁ could be immediately resolved, the alternative that provides the maximum expected net benefit could be selected for each possible value that θ₁ could take (max_j E_θ₂,_θ₃ NB(j, θ)). Because the other parameters remain uncertain, the expected net benefits for each value of θ₁ are the expectation over the joint distribution of θ₂,θ₃, assuming θ₂,θ₃ are independent of θ₁. However, as before, the true values of θ₁ are unknown, so the expected value of a decision taken with perfect information about θ₁, but current information about θ₂,θ₃, is found by averaging these maximum net benefits over the distribution of θ₁,

E_{θ_{1}} \max_{j} E_{θ_{2}, θ_{3}} N B (j, θ) .

The difference between (4) and (1) represents the maximum value of research that would provide estimates of θ₁, or the expected value of perfect parameter information (EVPPI). As previously, information can be used to inform treatment decisions for current and future populations so EVPPI can also be expressed at a population level by multiplying the per-patient EVPPI by the discounted population in (3) above. The EVPPI for the 3 groups of uncertain parameters is reported in Table 2 for 3 thresholds. EVPPI is highest for HRQoL benefits in the first year after treatment (EVPPI = £162 million at a threshold of £20 000) and suggests that more precise estimates of the short-term effect of EECP on HRQoL maybe worthwhile. The probability of sustaining the HRQoL benefits in subsequent years is also associated with significant EVPPI of £77 million at a threshold of £20 000 per QALY. However, the EVPPI associated with the probability of requiring repeat EECP sessions is very low, and further research to provide more precise estimates of this parameter appears unnecessary. Because more precise estimates of treatment effect either in the short or longer term will require experimental design, if selection bias is to be avoided, the results of the EVPPI analysis suggest that a further RCT may well be worthwhile. However, whether such a trial should have a 12-mo or longer follow-up depends on whether the additional benefits of a longer follow-up to more precisely estimate the duration of effect exceeds the additional opportunity cost of a more costly and lengthy trial.

What Is the Value of Alternative Research Designs?

Different research designs can be evaluated by estimating the expected net benefits if the decision was informed by additional sample information. This can be estimated by predicting possible sample statistics D that could be obtained from a particular study with a sample size of n. These possible sample results are combined with prior knowledge about the parameters to form predicted posterior values. With sample information, the decision maker would be able to choose the alternative that generates the maximum expected net benefit for each possible sample result and predicted posterior. However, which of the possible sample results and therefore predicted posteriors will be realized once the trial reports is unknown, so the expected value of a decision with sample information is the expectation over the distribution of predicted sample results and posteriors.^2,27

For example, a sample from the prior distribution assigned to the probability of sustaining HRQoL benefits during year 2 (QOL ~ Beta(α = 8.028, b = 2.577)) can be used to predict a possible result of a sample of size n using the binomial distribution (nQOL ~ Bin(QOL, n)). This predicted sample result is combined with the prior, forming a predicted posterior (QOL′ ~ Beta((α + nQOL), (β + n – nQOL))). Given this particular sample result, D, and predicted posterior, the decision maker could choose the alternative that offers the greatest net benefit, that is, max_j E_θ|DNB(j,θ) However, because the actual results of the sample are not known in advance, the expected net benefit of the decision when it is informed by additional sample information (NB_**) is found by averaging these maximum expected net benefits over the distribution of possible values of D (found by repeatedly sampling from the prior and likelihood):

N B * * = E_{D} \max_{j} E_{θ | D} N B (j, θ) .

The value of the proposed design with a sample size of n, or the EVSI, is the difference between the expected net benefit of a decision made with sample information, in (5) above, and the expected net benefit with current information, in (1) above.

The length of follow-up of the trial is an important aspect of design. Designs with a shorter follow-up might be preferred if the additional benefits of a longer follow-up are less than the additional opportunity costs of waiting longer until the trial reports. This is especially important in this case, where it is unlikely that EECP could be approved for widespread NHS use while a trial is being conducted, because early approval is likely to make randomized design unethical and fully informed patients would be unwilling to participate.

The per-patient EVSI associated with trials of different follow-up can be established in the same way as described in (5) above but recognizing that a shorter follow-up will mean that sample information will be available to update estimates of only some of the parameters. The other parameters will remain uncertain when calculating expected net benefits for each sample result and predicted posterior. For example, a trial with a 1-y follow-up could inform the HRQoL benefits expected from EECP in the first year after treatment (θ₁). However, it would not provide information about the duration of treatment effect beyond year 1 (θ₂) or subsequent retreatment (θ₃).

The per-patient EVSI for trials with different lengths of follow-up is illustrated in Figure 1 at a cost-effectiveness threshold of £20 000 per QALY. The EVSI is always higher for designs with longer follow-up as the sample information is able to resolve more of the uncertainty surrounding the approval of EECP at each sample size. However, the differences in EVSI for different lengths of follow-up is small relative to the overall EVSI, that is, at a sample size of 1000, the EVSI for a 1-, 2-, 3-, and 4-y follow-up is £907, £926, £930, and £960 per patient, respectively. The choice of which particular trial follow-up and sample size is appropriate (e.g., whether a 4-y follow-up is most valuable or a 1-y follow-up is sufficient) will depend on the population expected benefits, taking account of the opportunity costs of longer follow-up and the expected costs of gathering the sample information.

What Is the Societal Value of Research?

The expected societal net benefit of particular trial designs depends on the EVSI per patient and the population of patients who can ultimately benefit from the information once the trial reports. It also depends, however, on the expected net benefits that accrue to those patients enrolled in the trial and those not participating in the trial but who are prevalent while the trial is being conducted, as well as the additional costs of the trial itself. Therefore, the value of sample information at a population level not only depends on the prevalent (P₀) and incident population (I_t) over the time horizon (T), as described in (3), but also on the sample size of the trial (the number of patients enrolled and not enrolled), how patients are allocated between the arms of the trial, and the length of follow-up. In addition, the population that can ultimately benefit once the trial reports will depend on the nature of the disease and the effects of treatment received while the trial is being conducted. It also depends on whether the prevalent population and those who are incident while the trial is being conducted will survive and continue to be eligible for EECP once the trial reports. Most applications of EVSI have assumed that the condition is acute, with costs and benefits occurring in one period,²⁹ or that patients enter the decision problem only once in the first period when they are incident, that is, treatment choice is made once when they present so it is not possible to switch treatments in later periods (e.g., a surgical procedure). In this example, stable angina is a chronic condition, so the population prevalent while the trial is being conducted can still benefit from the information (change treatment) if they survive and remain eligible for EECP once the trial reports. Therefore, if the trial is conducted, there are 2 different periods in which net benefits accrue, which need to be accounted for. These include the net benefits while the trial is being conducted (to both patients enrolled and not enrolled in the trial) and the net benefits when the trial reports (including patient incident after the trial reports and those who were prevalent while the trial was conducted).

Net Benefits Once the Trial Reports

There are 3 groups of patients who can ultimately benefit from information when the trial reports: 1) the patients who are incident after the trial reports, 2) the patients who are enrolled in the trial, and 3) the patients not enrolled but prevalent while the trial is conducted. We assume that the trial will recruit from the prevalent population in time t = 0 because the prevalent population is very large relative to possible sample sizes.^c Therefore, the trial will report and the information will become available to inform treatment choice in period t, equal to the trial follow-up, τ.

The expected net benefits for the population incident after the trial reports (t > τ) is straight forward and similar to (3) above; they receive the expected net benefit with sample information given in (5) above, discounted at 3.5%:

\sum_{t = τ + 1}^{T} \frac{I_{t}}{{(1 + r)}^{t}} . N B_{* * .}

The n patients enrolled in the trial are from the prevalent population, so when the trial reports, they will receive the expected net benefits with sample information but only over their remaining life expectancy (L). Therefore, it becomes necessary to distinguish and record net benefits that accrue in each time period ( $N B_{* *}^{t}$ ),which sum, over the time horizon of the model (L), to the expected net benefit (NB_**) in (5) above.^d In this case, $N B_{* *}^{t}$ reflects whether patients have survived and remain eligible for EECP at the time the trial reports, so the expected net benefit once the trial reports accruing to those enrolled in the trial is given by

n . \sum_{t > τ}^{L} N B_{* *}^{t} .

Those not enrolled in the trial include patients from the prevalent population (P₀ – n) and those incident in the periods before the trial reports (I_t≤τ) Once again, it is necessary to distinguish the net benefits expected to accrue in each period because once the trial reports, these patients will receive only the expected net benefits with sample information for the remaining periods of their life expectancy:

(P_{0} - n) . \sum_{t > τ}^{L} N B_{* *}^{t} + \sum_{t = 1}^{τ} \frac{I_{t}}{{(1 + r)}^{t}} . \sum_{t > τ}^{L} N B_{* *}^{t} .

Net Benefits while the Trial Is Conducted

Net benefits while the trial is being conducted will accrue to those enrolled and not enrolled in the trial. Those enrolled in the trial are from the prevalent population, with half allocated to EECP and half to standard care. Therefore, as previously, it is necessary to distinguish and record the per-period net benefit for patients treated with EECP ( $N B_{1}^{t}$ ) or standard care ( $N B_{0}^{t}$ ). The net benefits accruing to the population enrolled in the trial while it is being conducted is given by

\frac{n}{2} (\sum_{t = 1}^{τ} N B_{1}^{t} + \sum_{t = 1}^{τ} N B_{0}^{t}) .

The net benefits that accrue to those patients not enrolled in the trial while it is being conducted will depend on whether EECP is approved for use or not. For the reasons discussed above, a trial is unlikely to be possible if EECP is approved for widespread use. In these circumstances, both the prevalent population who were not enrolled and those incident while the trial is being conducted will receive the net benefits associated with standard care while the trial is being conducted, that is,

(P_{0} - n) . \sum_{t = 1}^{τ} N B_{0}^{t} + \sum_{t = 1}^{τ} \frac{I_{t}}{{(1 + r)}^{t}} . \sum_{t \leq τ}^{τ} N B_{0}^{t} .

If the type of research required could be conducted while the technology is approved for use, then those not enrolled in the trial could receive the alternative treatment expected to be cost-effective based on existing evidence with the associated per-period net benefit, that is, $N B_{* *}^{t}$ from (1) above.

The Costs of Research

The direct additional resource costs of conducting the research also need to be included in any estimate of the societal net benefits of conducting the trial. For simplicity, we assume an element of fixed costs associated with the research (C_f) plus a marginal cost per additional patient enrolled, per period of follow-up (C_m). Therefore, the total cost of the trial is simply

C_{f} + n . τ . C_{m} .

It should be noted that the additional costs of EECP are already included in the relevant estimates of net benefit, so the costs in (11) refer only to the costs of the research itself rather than the provision of the technology.

The total expected societal net benefit of a particular trial design will be the sum of each of these elements of net benefit, which accrue to sometimes different populations at different times. The total expected net benefit is the sum of the net benefits once the trial reports, (6) + (7) + (8), plus the net benefits while the trial is being conducted, (9) + (10), minus the direct costs of the research (11). However, to decide whether the trial is worthwhile, these net benefits must be compared with the net benefits that would accrue if the trial were not conducted.

Net Benefits If the Trial Is Not Conducted

Without additional information, the alternative that is expected to offer the highest net benefit (see (1) above) should be approved for both prevalent and future patient populations. Therefore, the expected population net benefits of deciding not to conduct the trial are

(P_{0} + \sum_{t = 1}^{T = 10} \frac{I_{t}}{{(1 + r)}^{t}}) . N B * .

The net value of a particular trial design, or the expected net benefit of sampling (ENBS), is the difference between the total expected societal net benefit of a particular trial design and the expected net benefit if further research is not conducted, that is,

ENBS = (6) + (7) + (8) + (9) + (10) - (11) - (12) .

This expression for the ENBS represents the net social value of conducting the research: if it is positive, further research is worthwhile and should be conducted. This can also be used to choose between alternative research proposals and designs. For example, the appropriate trial follow-up will be the design that provides the greatest ENBS. Similarly, the sample size that generates the greatest ENBS for each different design can be identified.

Evaluation of Alternative Research Designs

It should now be clear that whether research is worthwhile and which design is most appropriate depend on a number of factors, for example, whether the additional benefits of a longer follow-up outweigh the additional opportunity costs of waiting longer until the trial reports. It also depends on whether research can be conducted while the technology is approved for widespread use (i.e., approval with research [AWR]). However, if the type of experimental research that is required would not be possible once EECP is approved, then its use must be restricted to only in research (OIR) if the research is to be undertaken. In these circumstances, there will be a tradeoff between the expected net benefits of early approval of a cost-effective treatment and the expected gain in net benefit if approval is withheld until research reports.¹ This aspect of the opportunity cost of research when OIR is the only possibility is captured in (10) above in the expression for ENBS. Even if research is possible with approval, if there are irrecoverable costs associated with approval, then OIR might be better because it avoids the commitment of costs until the results of research are known.⁴⁸ All of these issues are illustrated in Figure 2, which reports the net social value of conducting research for alternative research designs (length of follow-up and sample size) when research is conducted while EECP is approved (AWR) and when approval is withheld (OIR).

At a threshold of £10 000 per QALY, standard therapy is expected to provide the highest net benefit (see Table 2), but further research is worthwhile (ENBS > 0) to reduce the uncertainty surrounding a decision to reject EECP based on current evidence (Figure 2a). A trial with a 2-y follow-up and sample size of 440 gives the highest ENBS at this threshold (marginal gains in net benefit for 2 y rather than 1 y of follow-up outweigh the opportunity costs incurred by withholding sample information for 1 additional year). Because EECP is not expected to be cost-effective, its AWR would simply impose additional opportunity costs, which in this case means that research would not be worthwhile (ENBS < 0 in Figure 2b).^e

At the thresholds of £20 000 and £30 000 per QALY, EECP is expected to be cost-effective, but a decision to adopt EECP is uncertain, and further research is worthwhile irrespective of concurrent approval (ENBS > 0 for OIR and AWR in Figures 2c-d). However, OIR appears to offer greater ENBS than AWR at £20 000 per QALY despite the fact that EECP is expected to offer improvements in expected net benefits. Therefore, withholding approval until the research is conducted (OIR) might be expected to impose opportunity costs compared with AWR. But in this example, OIR offers greater ENBS than AWR. This is because it avoids the commitment of costs that cannot be recovered if the decision to approve EECP is revised once the results of research are known.

Irrecoverable costs include situations in which initial losses (negative additional net heath benefits) of a technology are offset by later gains (positive net benefits). If early approval is revised (e.g., due to research revealing that the technology is not as effective as expected), then initial losses will have been incurred, but they will not be compensated by later gains. The impact of this time profile of cumulative incremental net benefit for EECP (illustrated in Figure 3) will be greater when a decision to approve is more likely to change and in the more immediate future, that is, when it is more uncertain and when research will be conducted and report in the near future.

The initial per-patient costs of EECP are high and far in excess of the immediate health benefits in the initial period of treatment. However, at thresholds of £20 000 and £30 000, these losses are offset by expected future health benefits after 14 and 3 y, respectively. Therefore, if research reports before losses are recouped, there is a chance that the results will indicate that EECP is not in fact cost-effective, approval will be withdrawn, and patients will be switched from EECP to standard care; initial losses will have been incurred, but they will not be compensated by later gains that were originally expected.

Consequently, OIR offers greater ENBS than AWR, so approval should be withheld even if research would have been possible while EECP was approved for widespread use. The difference between OIR and AWR is greater at a threshold of £20 000 rather than £30 000. This is for 3 reasons: 1) the opportunity costs (additional expected net benefit of EECP) of withholding approval if research confirms that EECP is cost-effective are lower, 2) the potential losses of net benefit are greater if the research indicates that EECP is not in fact cost-effective, and 3) it is more likely that research will indicate that EECP is not cost-effective once it reports. Therefore, the choice between OIR and AWR will depend on the threshold (at higher values, AWR would provide greater ENBS than OIR) and on the research design, especially optimal length of follow-up. Equally, aspects of research design are influenced by whether research will be conducted as part of OIR or AWR. Although in this particular example, a 1-y follow-up offers the highest ENBS irrespective of AWR and OIR, sample size does differ (at a threshold of £30 000), and in other circumstances, research optimal design may differ markedly.^30,48,51

Discussion

Most applications of EVI have been restricted to establishing the value of perfect information. EVPI provides a necessary condition for deciding whether more evidence is required, but it cannot address the design of any future research. EVSI provides a sufficient condition for deciding if further researchis worthwhile and can be used to identify efficient research design. The present study has demonstrated the feasibility of applying these methods to both adoption and research decisions in a timely way, even where data are sparse and formal methods of elicitation are required.

The net social value of any research design depends on the expected net benefits of the research, the costs of conducting the research, and the size of the population who can ultimately benefit from the information when it becomes available. The population who can benefit will depend on the nature of the disease and the effects of treatment received while the research is conducted. Most applications of EVSI to date have assumed that the nature of the disease is acute with costs and benefits accruing in one time period or that treatment choice is made once. However, for a condition that is chronic, the population prevalent while the research is being conducted can still benefit from the information and switch treatment if they survive and remain eligible when the research reports. Therefore, it is important to distinguish between different groups of patients who accrue different net benefits in different time periods: patients who are incident after the research reports, patients who are enrolled in the research, and patients not enrolled but prevalent while the research is conducted. We have carefully examined the impact that the research decision is expected to have on each of these groups of patients.

We have examined a range of alternative research designs, including length of follow-up, sample size, and policy options available. We have shown that designs with a shorter follow-up can be preferred if the additional benefits of a longer follow-up are less than the additional opportunity costs of waiting longer until the research reports. To our knowledge, this is the first EVSI study to examine the implications for the research decision of approving the technology for widespread use or withholding the technology from approval until the research is conducted. Although this particular example is associated with the type of appraisal conducted by NICE in the United Kingdom, the fundamental insights and methods have much wider application to other health care systems that are not characterized by a single budget constraint^f or even when explicit consideration of costs and cost-effectiveness is not explicitly undertaken^g in the decision-making process. In general, there is often an unavoidable tradeoff between the expected net benefits to current patients from early access to a cost-effective technology and the health benefits to patients in the future from withholding approval until valuable research is conducted. We have also provided a practical demonstration that, even when a technology is expected to be cost-effective, an OIR policy option may be more appropriate than AWR even when research could be conducted while the technology is approved for widespread use if there are opportunity costs that cannot be recovered should the results of the research indicate that its initial approval should be withdrawn. Such time profiles of cumulative net benefit are likely to be common, especially for intervention with mortality effects, and ought to be taken into account when making both adoption and research decisions.

Footnotes

Notes

References

Griffin

Claxton

Palmer

Sculpher

. Dangerous omissions: the consequences of ignoring decision uncertainty. Health Econ. 2011;20(2):212–24.

Briggs

Claxton

Sculpher

. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.

Drummond

Sculpher

Torrance

O’Brien

Stoddart

. Methods for the Economic Evaluation of Health Care Programmes. 3rd ed. Oxford: Oxford University Press; 2005.

National Institute for Health and Clinical Excellence. Guide to the Methods of Technology Appraisal. London: National Institute for Health and Clinical Excellence (NICE). Available from: URL: www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf . 2008.

O’Donnell

Pham

Pashos

. Health technology assessment in evidence-based health care reimbursement decisions around the world: an overview. Value Health. 2009;12(Suppl. 2):S1–5.

Sullivan

Watkins

Sweet

Ramsey

. Health technology assessment in health-care decisions in the United States. Value Health. 2009;12(Suppl. 2):S39–44.

Claxton

Sculpher

. Using value of information analysis to prioritise health research: some lessons from recent UK experience. Pharmacoeconomics. 2006;24(11):1055–68.

Colbourn

Asseburg

Bojke

. Preventive strategies for group B streptococcal and other bacterial infections in early infancy: cost effectiveness and value of information analyses. BMJ. 2007;335(7621):655.

Berger

. Statistical Decision Theory and Bayesian Analysis. New York: Springer-Verlag; 1985.

10.

Howard

. Information value theory. IEEE Trans Syst Sci Cybernet. 1966;2:22–6.

11.

Parmigiani

. Modeling in Medical Decision Making. Chichester (UK): John Wiley; 2002.

12.

Pratt

Raiffa

Schlaifer

. Introduction to Statistical Decision Theory. Cambridge (MA): MIT Press; 1995.

13.

Raiffa

Schlaifer

. Applied Statistical Decision Theory. New York: Wiley-Interscience; 2000.

14.

Schlaifer

. Probability and Statistics for Business Decisions. New York: McGraw-Hill; 1959.

15.

Hammitt

Cave

JAK

. Research Planning for Food Safety: A Value-of-Information Approach. Santa Monica (CA): RAND; 1991.

16.

Thompson

Evans

. The value of improved national exposure information for perchloroethylene (perc): a case study for dry cleaners. Risk Anal. 1997;17:253–71.

17.

Thompson

Graham

. Going beyond the single number: using probabilistic risk assessment to improve risk management. Hum Ecol Risk Assess. 1996;2:1008–34.

18.

Claxton

. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18:341–64.

19.

Claxton

. Bayesian approaches to the value of information: implications for the regulation of new pharmaceuticals. Health Econ. 1999;8:269–74.

20.

Felli

Hazen

. Sensitivity analysis and the expected value of perfect information. Med Decis Making. 1998;18:95.

21.

Griffin

Welton

Claxton

. Exploring the research decision space: the expected value of information for sequential research designs. Med Decis Making. 2010;30:155–62.

22.

McKenna

Chalabi

Epstein

Claxton

. Budgetary policies and available actions: a generalisation of decision rules for allocation and research decisions. J Health Econ. 2010;29(1):170–81.

23.

Bojke

Claxton

Sculpher

Palmer

Philips

. The cost-effectiveness and value of information associated with repeat screening for age related macular degeneration. Med Decis Making. 2008;28:33–43.

24.

Fenwick

Palmer

Claxton

Sculpher

Abrams

Sutton

. An iterative Bayesian approach to health technology assessment: application to a policy of preoperative optimization for patients undergoing major elective surgery. Med Decis Making. 2006;26:480.

25.

Hassan

Hunink

MGM

Laghi

. Value-of-information analysis to guide future research in colorectal cancer screening. Radiology. 2009;253:745–52.

26.

Philips

Claxton

Palmer

. The half-life of truth: what are appropriate time horizons for research decisions? Med Decis Making. 2008;28:287–99.

27.

Ades

Claxton

. Expected value of sample information calculations in medical decision modeling. Med Decis Making. 2004;24(2):207–27.

28.

Brennan

Kharroubi

. Expected value of sample information for Weibull survival data. Health Econ. 2007;16:1205–25.

29.

Conti

Claxton

. Dimensions of design space: a decision-theoretic approach to optimal research design. Med Decis Making. 2009;29:643–60.

30.

Eckermann

Willan

. Time and expected value of sample information wait for no patient. Value Health. 2008;11:522–6.

31.

Koerkamp

Nikken

Oei

Stijnen

Ginai

Hunink

MGM

. Value of information analysis used to determine the necessity of additional research: MR imaging in acute knee trauma as an example. Radiology. 2008;246:420–5.

32.

Stevenson

Oakley

Lloyd Jones

. The cost-effectiveness of an RCT to establish whether 5 or 10 years of bisphosphonate treatment is the better duration for women with a prior fracture. Med Decis Making. 2009;29(6):678–89.

33.

Sinvhal

Gowda

Khan

. Enhanced external counterpulsation for refractory angina pectoris. Heart. 2003;89:830–3.

34.

Arora

Chou

Jain

. The multicenter study of enhanced external counterpulsation (MUST-EECP): effect of EECP on exercise-induced myocardial ischemia and anginal episodes. J Am Coll Cardiol. 1999;33(7):1833–40.

35.

Lawson

Hui

JCK

Soroff

. Efficacy of enhanced external counterpulsation in the treatment of angina pectoris. Am J Cardiol. 1992;70(9):859–62.

36.

Lawson

Hui

JCK

Zheng

. Three-year sustained benefit from enhanced external counterpulsation in chronic angina pectoris. Am J Cardiol. 1995;75(12):840–1.

37.

McKenna

McDaid

Suekarran

. Enhanced external counterpulsation (EECP) for the treatment of stable angina and heart failure: a systematic review and economic analysis. Health Technol Assess. 2009;13(24):1–128.

38.

Arora

Chou

Jain

. Effects of enhanced external counterpulsation on health-related quality of life continue 12 months after treatment: a substudy of the Multicenter Study of Enhanced External Counterpulsation. J Investig Med. 2002;50(1):25–32.

39.

O’Hagan

Buck

Daneshkhah

. Uncertain Judgements: Eliciting Experts’ Probabilities. Chichester (UK): Wiley; 2006.

40.

van Noortwijk

Dekker

Cooke

Mazzuchi

. Expert judgment in maintenance optimization. IEEE Trans Reliabil. 1992;41(3):427–32.

41.

Cooke

. Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press; 1991.

42.

Briggs

Mihaylova

Sculpher

. Cost effectiveness of perindopril in reducing cardiovascular events in patients with stable coronary artery disease using data from the EUROPA study. Heart. 2007;93:1081–6.

43.

Government Actuary’s Department. Interim Life Tables. 2007. Available from: URL: http://www.gad.gov.uk/Demography%5FData/Life_Tables/Interim_life_tables.asp.

44.

Bonetti

Holmes

Jr. Lerman

Barsness

. Enhanced external counterpulsation for ischemic heart disease: what’s behind the curtain? J Am Coll Cardiol. 2003;41(11):1918–25.

45.

Michaels

Barsness

Soran

. Frequency and efficacy of repeat enhanced external counterpulsation for stable angina pectoris (from the International EECP Patient Registry). Am J Cardiol. 2005;95(3):394–7.

46.

Phelps

Mushlin

. On the (near) equivalence of cost-effectiveness and cost-benefit analyses. Int J Technol Assess Health Care. 1991;7(1):12–21.

47.

Stinnett

Mullahy

. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Med Decis Making. 1998;18(Suppl.):S68–80.

48.

Eckermann

Willan

. Expected value of information and decision making in HTA. Health Econ. 2007;16:195–209.

49.

Palmer

Smith

. Incorporating option values into the economic evaluation of health care technologies. J Health Econ. 2000;19:755–66.

50.

Allender

Peto

Scarborough

Boxer

Rayner

. Morbidity. Coronary Heart Disease Statistics. 15th ed. London: British Heart Foundation; 2007.

51.

Willan

Pinto

. The value of information and optimal clinical trial design. Stat Med. 2005;24:1791–806.

52.

Claxton

Paulden

Gravelle

Brouwer

Culyer

. Discounting and decision rules in the economic evaluation of health care technologies. Health Econ. 2011;20:2–15.

53.

Claxton

Walker

Palmer

Sculpher

. Appropriate perspectives for health care decisions. Centre for Health Economics, University of York CHE Research Paper 54. York (UK): University of York; 2010.