Adjusting Expected Value of Sample Information Using Realistic Expectations around Implementation

We thank Eckermann and Willan for their interest in our work. ¹ Although Eckermann and Willan ² recognize that our work represents a useful addition to the literature, they express concerns about the way that we have estimated the expected value of sample information (EVSI) under perfect implementation (we use the word “optimal” with regard to implementation). In particular, Eckermann and Willan assert that we have estimated the EVSI with perfect implementation “as though conducting no further research will result in implementation of the option with the lowest INB [incremental net benefit], rather than maximizing the INB given current evidence.” ²

Unfortunately, Eckermann and Willan seem to have misunderstood the way we have calculated EVSI with optimal implementation in our article. In estimating this figure, we agree that the “counterfactual” should be the option that maximizes the net monetary benefit (NMB; which they call INB) given current evidence. This is precisely the way we have calculated this EVSI figure: as the difference between the “factual” figure, representing the expected NMB if research takes place (£318.11 million, cell L in Table 2 of our article), and the “counterfactual” figure, representing the maximum NMB under current evidence (£293.12 million, cell C in Table 2). It should be obvious from our Table 2 that the figure we used for the counterfactual is indeed the highest of the NMB figures for current information. The resulting EVSI figure (£24.99 million) is given in Table 3 of our article.^1(p7)

We, of course, agree with Eckermann and Willan that decision context is important; taking this into account and expressing EVSI in a way that more closely reflects the interplay between information and implementation is the key motivation for our article. In doing so, we account for the fact that availability of additional information is expected to improve implementation, but it will not necessarily lead to instantaneous and perfect implementation. Eckermann and Willan seem to share this view in their letter. ² Given this, one can easily see why EVSI with improved implementation (i.e., higher than “current” implementation but lower than “optimal” implementation) is expected to result in higher NMB than EVSI with current implementation and lower NMB than EVSI with optimal implementation.

We thank Eckermann and Willan for recognizing the potential of our proposed framework to provide a useful tool for value of information analysis, and we hope that it is now clear to them that our calculations are correct, based on the counterfactual option that maximizes NMB given current evidence.