Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

Abstract

Background

Personalizing medical treatments based on patient-specific risks and preferences can improve patient health. However, models to support personalized treatment decisions are often complex and difficult to interpret, limiting their clinical application.

Methods

We present a new method, using machine learning to create meta-models, for simplifying complex models for personalizing medical treatment decisions. We consider simple interpretable models, interpretable ensemble models, and noninterpretable ensemble models. We use variable selection with a penalty for patient-specific risks and/or preferences that are difficult, risky, or costly to obtain. We interpret the meta-models to the extent permitted by their model architectures. We illustrate our method by applying it to simplify a previously developed model for personalized selection of antipsychotic drugs for patients with schizophrenia.

Results

The best simplified interpretable, interpretable ensemble, and noninterpretable ensemble models contained at most half the number of patient-specific risks and preferences compared with the original model. The simplified models achieved 60.5% (95% credible interval [crI]: 55.2–65.4), 60.8% (95% crI: 55.5–65.7), and 83.8% (95% crI: 80.8–86.6), respectively, of the net health benefit of the original model (quality-adjusted life-years gained). Important variables in all models were similar and made intuitive sense. Computation time for the meta-models was orders of magnitude less than for the original model.

Limitations

The simplified models share the limitations of the original model (e.g., potential biases).

Conclusions

Our meta-modeling method is disease- and model- agnostic and can be used to simplify complex models for personalization, allowing for variable selection in addition to improved model interpretability and computational performance. Simplified models may be more likely to be adopted in clinical settings and can help improve equity in patient outcomes.

Keywords

meta-model medical decision making personalized medicine schizophrenia

Personalizing medical treatment decisions based on patient-specific risks and preferences can significantly improve patient health. By accounting for patient-specific risks (e.g., demographic factors, biological factors, and medical history), patients can be allocated to treatments that they respond to favorably. By accounting for patient-specific preferences (e.g., preferences regarding treatment efficacy, side effects, and cost), patients can be allocated to treatments that they prefer.

A variety of approaches have been proposed for integrating evidence on treatment efficacy with patient-specific risks and preferences. These approaches variously account for population-level variations in risks and preferences,^1–5 patient-specific variations in risks,^6,7 patient-specific variations in preferences,^8–10 and patient-specific variations in both risks and preferences.¹¹

Such methods have not been widely adopted in clinical practice, for a variety of reasons. First, existing methods for treatment personalization often produce complex, black box models. It can be difficult for physicians and patients to understand and trust such models.^12,13 While empirical studies have not fully examined the extent of this problem in specific settings, it is clear that both physicians and patients must be educated to appreciate algorithmically determined diagnosis and treatment options, particularly those arising from complex models, before such models can attain broad use. As one article notes, such education is crucial, but also complicated.¹⁴

Second, these methods can require running large simulation models to evaluate treatment alternatives for each patient, which can be too computationally costly for practical use. This may occur, for example, with complex microsimulation models and/or when deep learning models are used to predict patient-specific risks and develop treatment plans using high-dimensional, patient-specific data. Lengthy computation times may preclude real-time use of such models; this, along with complex computer code and sophisticated computation platforms, may hinder the adoption of such models.

Third, treatment personalization is often based on all risks and/or all preferences. However, some variables can be difficult, risky, or costly to obtain for each patient. In some cases, evaluating patient risks could even potentially harm patients if, for example, invasive medical procedures are required. The potential benefits of excluding such variables must be balanced with the potential downsides of less informed treatment selection.¹¹

The Second Panel on Cost-Effectiveness in Health and Medicine suggests that efficient model emulators, or meta-models, could be developed to reduce the computational cost of complex models for health decision making.¹⁵ A meta-model is a statistical approximation of an originally constructed model. Besides directly replacing a model in predicting the model outputs, meta-models can also be used to improve model interpretability¹⁶ and aid in model calibration.¹⁷

Meta-models have been most commonly used in the physical sciences and engineering.^18–20 In health policy, meta-models have primarily been used to lower computation time of microsimulation and other complex models, to streamline value-of-information analysis, and to assess decision uncertainty due to uncertainty in model inputs.^16,21–31 Recent studies have described the use of meta-models to simplify microsimulation models of strategies for infant HIV testing and screening,³² colorectal cancer screening,³³ and hepatitis C testing and treatment in correctional settings.³⁴

In this article, we propose a general, disease- and model-agnostic method that uses machine-learning methods to develop meta-models that simplify complex models for personalization of medicine treatment decisions. Our method allows for determination of the optimal degree of personalization considering that some variables may be more difficult, risky, or costly to obtain than others and enables creation of models that are more interpretable and less computationally expensive than complex black box models. We consider 3 model categories: simple interpretable models, interpretable ensemble models, and noninterpretable ensemble models. We demonstrate our method by applying it to simplify an existing, complex model for personalizing the selection of antipsychotic drugs for patients with schizophrenia. We assess the net health benefit, in units of quality-adjusted life-years (QALYs), of using the meta-models and the original model to personalize treatment compared with no personalization, among other measures of model performance. We interpret the meta-models to the degree allowed by their model architectures.

Methods

Meta-modeling Framework

Our meta-modeling method (Figure 1) works as follows. We begin by running the original, complex model to personalize treatment for a demographically representative population of patients. We create a data set with, for each patient, features (demographics, risks, and preferences), projected mean net health benefit for each treatment, and the recommended treatment. We split this data set into training and test data sets.

We preprocess the data by standardizing numeric patient features. For each numeric variable, we subtract the sample mean and divide by the sample standard deviation. During prediction, we apply this same variable standardization to patient features before passing them to the model. This standardization has a variety of benefits: for example, it may allow for faster gradient descent during model training.

We then train machine learning meta-models on the training data set using various model architectures. We treat the treatment selection problem as a classification problem: that of identifying the best treatment for a patient. We include steps for hyperparameter tuning and variable selection. We consider interpretable and noninterpretable models. We define interpretable models as those that a physician could easily understand.

For the interpretable models, we consider commonly used decision tree models including classification and regression trees, conditional inference trees, and C5.0 decision trees.^35–37 We consider decision tree models as they are easy for physicians to understand since the models visually represent key questions to ask patients and the recommended treatment in each case. We constrain the trees to have a maximum depth of 4; other maximum tree depths could be used.

For the noninterpretable models, we consider supervised learning models that have previously been identified as potentially useful for health services research, including generalized linear models, deep learning models, extremely randomized trees, distributed random forests, and gradient boosting machines.³⁸

Models that combine predictions from multiple individual models can often yield improved performance. Thus, we consider both noninterpretable and interpretable ensemble models created using the SuperLearner algorithm, which has been shown to be asymptotically optimal.^39–42 This algorithm allows for a combination of predictions from multiple individual models and uses cross-validation to prevent overfitting. We classify the ensemble of interpretable models as interpretable since physicians could still understand the component models. However, such a model is less interpretable than its component models.

We evaluate model performance by computing the net health benefit, in units of QALYs, compared with full personalization based on all risks and preferences using the original model. Specifically, we compute mean projected net health benefit with the best patient-specific treatments recommended by the meta-model as compared with the best treatment for the population with no personalization from the original model data set. We also compute the mean projected net health benefit with the best patient-specific treatments from the original model data set compared with the best treatment for the population with no personalization from the original model data set. We divide these 2 values and multiply by 100%; this is the percentage of net health benefit retained by personalization using the meta-model compared with the original model.

During the process of training the meta-models, we use hyperparameter tuning to improve model performance. Hyperparameter tuning is a way of testing various parameters used to control a machine learning process (e.g., maximum tree depth) to determine the best values for those parameters. The number of hyperparameters and potential values depend on the machine learning model architecture. We use grid search to identify the best hyperparameter values. We perform hyperparameter tuning on the training data set using cross-validation.

We also perform variable selection; that is, we determine which variables from the original model to include in a meta-model. Fitting models with all possible combinations of variables is frequently too computationally expensive; thus, we use a greedy approach. For a given model, we exclude one variable at a time from the model, refit the model, and calculate the projected net health benefit based on the new treatment recommendations.

Importantly, we allow for penalties to be included for difficult, risky, or costly-to-obtain variables. A variable may be difficult to obtain if it requires time and resources to evaluate: for example, eliciting patient preferences may require both patient and staff time and possibly an electronic interface.⁴³ A variable may be risky to obtain if collection of the data may adversely affect patient health (e.g., a risk that requires an invasive medical procedure to evaluate). A variable may be costly to obtain if it requires use of medical equipment, such as imaging devices. If a variable is difficult or risky to obtain, we add QALYs following its removal equal to the QALY penalty for inclusion of that variable. If a variable is costly to obtain, a willingness-to-pay threshold must be specified and then the cost can be converted to QALYs as is typical in calculating net health benefit. In the United States, a willingness-to-pay threshold of $50,000 to $100,000/QALY may be appropriate.⁴⁴ The cost penalty in units of QALYs can then be applied in the same way as before. These are not mutually exclusive categories: for example, a variable may be both risky and costly to obtain. In this case, the penalties, in units of QALYs, can be summed before they are applied.

We then select the best-performing reduced model, retune the hyperparameters, and refit the model. We repeat this process until there are no variables left to exclude. We compare the performance of these models on the training data set using cross-validation to determine the optimal degree of personalization for each model architecture. We select the best-performing model in each category—interpretable, interpretable ensemble, and noninterpretable ensemble—and project their performance using the test data set.

Finally, we interpret the models. We compute measures such as variable importance. For interpretable trees, we plot them.

Personalizing Schizophrenia Treatment

We illustrate our method by applying it to simplify an existing model for personalized selection of antipsychotic drugs for patients with schizophrenia.¹¹ Briefly, the schizophrenia treatment selection model considers 15 antipsychotic drugs and placebo. It accounts for patient-specific risks and preferences for 9 different possible outcomes (efficacy, drug cost, years of life lost, and 6 potential side effects: weight gain, extrapyramidal side effects, prolactin increase, QTc prolongation, sedation, and agranulocytosis). It also accounts for patient demographics including age, sex, and race (Black, White, other). It personalizes treatment by simulating 10,000 identical versions of each patient on each treatment alternative over a period of 5 y and selecting the treatment that maximizes expected net health benefit. Analysis using this model found that, compared with treating all patients with the drug that yields the greatest net health benefit on average (amisulpride), personalizing the selection of antipsychotic drugs for schizophrenia patients would be expected to yield 0.33 QALYs (95% credible interval [crI]: 0.30–0.37) per patient over a 5-y model period at an incremental cost of $4849/QALY gained (95% crI: dominant–$12,357).

Using the full model, both patient-specific risks and preferences influenced treatment selection. Clozapine, which is the most effective drug but has a relatively severe side effect profile, was generally selected for patients with high baseline Positive and Negative Syndrome Scale (PANSS) scores and preferences for drug efficacy. Amisulpride, which is the second most effective drug and has a better side effect profile, was generally selected for patients with lower baseline PANSS scores and preferences for drug efficacy. Amisulpride was also generally selected for patients with higher risks of or preferences for avoiding weight gain. Other drugs including aripiprazole, haloperidol, and lurasidone were selected for patients with less common sets of risks and preferences. Placebo tended to be selected for patients who are more susceptible to and have stronger preferences for avoiding side effects.

The full model may be difficult to implement in a clinical setting. It requires evaluation of risks and preferences for 9 different outcomes for each patient. It is not readily interpretable, so physicians and others may be less likely to trust the recommendations of the model. In addition, it requires running a relatively complex and computationally expensive simulation model for each patient.

We applied our method to simplify this model. We first used the original model to simulate 10,000 patients that are representative of the adult US population with schizophrenia, as previously described.¹¹ For each patient, we recorded patient features, drug scores, and the recommended drug. We split this data set into training and test data sets using a 90:10 split.

We trained machine learning models using the training data set and all patient features. We first followed the procedure described above but omitted the steps for variable selection and variable penalties. We computed model performance on the test data set in terms of percentage of net health benefit of personalization retained compared with using the original model.

We then repeated this process, now including variable selection and variable penalties as described above. We did not penalize inclusion of risks as they are all relatively easy to obtain for this treatment selection problem. We penalized inclusion of each preference by 0.05 QALYs because preferences require time and effort to elicit in clinical settings. These penalties are illustrative; further work could be directed to more accurately determine their values. We considered other penalties in sensitivity analysis. We computed performance on the test data set as percentage of net health benefit of personalization retained compared with using the original model.

Finally, we interpreted the machine learning models to the extent permitted by their model architectures. For all models, we computed standardized variable importance. For ensemble models, for each variable, we computed variable importance as the mean of standardized variable importance for the component models. We plotted the optimal interpretable tree.

All analyses were programmed in R version 3.5.1,⁴⁵ with input data and statistical code for replication and extension of our analysis published at https://sdr.stanford.edu concurrent with publication.

Results

Figure 2 shows the performance of the meta-models fit without variable selection on the test data set. Supplementary Table S1 shows the selected hyperparameter values. Personalization using the models was able, in varying degrees, to achieve much of the net health benefit of personalization using the original model as compared with no personalization. Among the interpretable models, the C5.0 decision tree model gained 62.0% as many QALYs as the original model, and the ensemble model gained 62.5% as many QALYs as the original model. As expected, the noninterpretable models generally performed better than the interpretable models did. The best-performing noninterpretable models, the gradient boosting machine model and the noninterpretable ensemble model, yielded 96.5% and 96.9% as many QALYs as the original model, respectively.

Supplementary Table S2 shows the accuracy of the models, computed as the percentage of meta-model treatment recommendations that match those from the original model. The best interpretable model was 62.7% accurate in treatment selection, the best interpretable ensemble model was 63.7% accurate, and the best noninterpretable ensemble model was 88.9% accurate. When we compute accuracy assuming that all misclassifications that result in a net health benefit loss of less than 0.05 QALYs (approximately half a month) are in fact correct, these accuracy values increase to 68.8%, 69.6%, and 94.5%, respectively.

Figure 3 shows the results when we fit the meta-models additionally using steps for variable selection and penalties for difficult, risky, or costly-to-obtain variables. We selected the best-performing model in each category (interpretable, interpretable ensemble, and noninterpretable ensemble models). Supplementary Table S3 shows the selected hyperparameter values. The best interpretable model gained 60.5% (95% crI: 55.2–65.4) as many QALYs as the original model did, the best interpretable ensemble model gained 60.8% (95% crI: 55.5–65.7) as many QALYs, and the best noninterpretable ensemble model gained 83.8% (95% crI: 80.8–86.6) as many QALYs.

Supplementary Table S4 shows the accuracy of the models, computed as the percentage of meta-model treatment recommendations that match those from the original model. The best interpretable model was 62.2% accurate in treatment selection, the best interpretable ensemble model was 62.3% accurate, and the best noninterpretable ensemble model was 74.9% accurate. When we compute accuracy assuming that all misclassifications that result in a net health benefit loss of less than 0.05 QALYs (approximately half a month) are in fact correct, these accuracy values increase to 68.2%, 68.4%, and 80.8%, respectively.

The models were generally stable to changes in the QALY penalty per preference. In the base-case analysis, the penalty per preference was 0.05 QALYs: for each additional preference we included, we decremented the per-person QALYs by 0.05. The meta-models were unchanged when the penalty was higher or when the penalty was up to approximately a factor of 5 lower. For example, the best noninterpretable model did not change until the penalty was 0.014 QALYs per preference or less. At this point, the preference for weight gain entered the model. With no penalty, all preferences entered the model.

Computation time for the meta-models was orders of magnitude faster than for the original model (Supplementary Figure S1). The original model took approximately 15 s per patient to compute. The meta-models took less than 0.4 s per patient.

The performance of the meta-models with variable selection (Figure 3, Supplementary Table S4) was worse than the performance of the models when variable selection was not used (Figure 2, Supplementary Table S2) because our method penalizes the use of difficult, risky, or costly-to-obtain variables in the variable selection process. For the noninterpretable ensemble model, the optimal set of variables included all risks but no preferences. The interpretable models selected a subset of these risks likely due in part to the constraint on maximum tree depth. The noninterpretable ensemble model performed better than the interpretable ensemble model, which performed better than the interpretable model. These results make intuitive sense as the more complex model architectures can capture additional structure within the data.

We interpreted our best-performing model in each category to the degree allowed by the model architecture. Figure 4 shows the variable importance for the optimal interpretable, interpretable ensemble, and noninterpretable ensemble models. Variable importance was similar across the different models, with the most important variables being the risks of 3 side effects—QTc prolongation, weight gain, and prolactin increase—as well as the patient’s PANSS score (a measure of disease severity). In the interpretable ensemble model, patient age was also important; the noninterpretable ensemble model additionally considered the patient’s sex and race. These results make intuitive sense: the original model allocated most patients to amisulpride and clozapine, and the most important variables for the meta-models were those risks for which the treatment effects for amisulpride and clozapine were most different in our base-case analysis.

Figure 5 shows the best interpretable model, a C5.0 decision tree. Patients are variously allocated to clozapine, amisulpride, aripiprazole, or placebo depending on risk of QTc prolongation, weight gain, and prolactin increase as well as PANSS score. We note that, using the original model, 96.4% of patients were assigned to 1 of these 4 treatment options (out of 16 treatment options). Physicians can use the decision tree to select a drug for a patient by following the nodes from top to bottom. Thus, the physician would first determine whether the patient has a QTc interval less than or equal to 425 ms. If so, the physician would follow the branch to the left; if not, the physician would follow the branch to the right. Assuming the patient has a QTc interval less than or equal to 425 ms, the physician would next assess patient weight. If the patient weighs more than 67 kg, the physician would follow the branch to the right and see that drug 2, amisulpride, is recommended for that patient. If the patient weighs less than 50 kg, then drug 1, clozapine, is recommended. If the patient weighs between 50 and 67 kg, the appropriate drug depends on the patient’s PANSS score: if the score is less than 73 (less severe schizophrenia symptoms), then amisulpride is recommended, and if the score is greater than 73 (more severe symptoms), then clozapine is recommended. Similar decisions can be determined for patients with a QTc interval greater than 425 ms.

The personalized treatment decisions as determined by this decision tree gained approximately 60.5% as many QALYs as the original model and allocated similar percentages of patients to the different drugs. The simple model allocated 28.5% to clozapine versus 35.6% in the original model, 57.0% to amisulpride versus 43.0%, 0.0% to haloperidol versus 1.2%, 9.4% to aripiprazole versus 11.4%, 0.0% to lurasidone versus 2.4%, and 5.1% to placebo versus 6.4%.

Discussion

Our method using machine learning to generate meta-models can simplify models for personalization of medical treatment decisions. Our method has 3 primary benefits: it allows for determination of the optimal degree of personalization, which is important since evaluating all risks and preferences for all patients is often infeasible in a clinical setting; it allows for otherwise complex, black box models to be interpreted; and it reduces the computation time needed to determine a personalized treatment.

In applying our modeling framework to simplify a model for personalized selection of antipsychotic drugs for patients with schizophrenia, we found that much simpler models, with fewer variables and shorter computation time, can be used while still preserving much of the health benefits of full personalization based on all risks and preferences using the original model. As expected, noninterpretable model architectures generally performed better than the interpretable model architectures.

The optimal model likely depends on the health care setting. An interpretable or noninterpretable ensemble machine learning model could be run on a computer in the physician’s office to project the optimal treatment for each patient. An interpretable ensemble machine learning model would likely result in worse health outcomes than a noninterpretable model but would be easier for physicians to understand and interpret, which might lead to more widespread clinical adoption. Alternatively, a simple, interpretable model such as the decision tree we developed for schizophrenia treatment selection could be used by the physician without a computer to project the optimal treatment for each patient. This approach would likely result in the worst health outcomes of the 3 approaches but could easily be used and understood by physicians without requiring the use of a computer. Such a model could be implemented in the form of a clinical guideline for personalized treatment, similar to those used for treatment of other conditions such as hypertension.⁴⁶

Methods are currently being developed to address some of the disadvantages of less interpretable meta-models. For example, methods are being developed to allow such models to be better understood.⁴⁷ Techniques such as partial dependence plots, local interpretable model-agnostic explanations, and others can help convince physicians that the models are behaving appropriately. Although these advances are helpful and important, such models are generally still more difficult to understand than interpretable models and will likely remain that way. In addition, user-friendly interfaces are being developed to allow less interpretable meta-models to be more easily used in a clinical setting. For example, such an interface was recently developed to support decisions about infant HIV testing and screening.³² These advances will likely help mitigate this particular limitation of less interpretable models.

There is debate over when meta-model performance is sufficient to replace an original, complex model.³² This question might be best addressed through future empirical research to measure or project real-world consequences of treatment personalization using a meta-model. Outcomes of interest might include model adoption rates, health outcomes, and disparities. From this information, cost-effectiveness analysis and related methods might be used to compare the use of a meta-model versus the original model.

Our approach has several limitations. A meta-model shares the same limitations as the original model. For example, if the original model was developed with biased data, then the meta-models will likely learn the same biases. Moreover, meta-models simply learn from the underlying model, so meta-model performance will likely be no better than the performance of the original model. However, it is possible that use of the meta-models may result in better patient health outcomes if they recommend exclusion of variables that are risky to obtain (i.e., could cause adverse events). Importantly, meta-models may also result in better patient health outcomes if they are more widely adopted than the more complex original models. Models for personalizing medical treatment decisions, many of which are complex black boxes, have not been widely adopted to date.

A variety of promising areas for further research remain. Research is needed to evaluate different personalized medicine methods in clinical practice, in particular to compare them and to determine the degree to which they increase patient health outcomes in practice as compared with their projections. Lack of such evidence has been cited as a barrier to the use of personalized treatment models.¹² It would be useful to continue to develop methods for evaluating patient-specific risks and preferences that can readily be incorporated into clinical workflows. It would also be useful to apply personalization methods to additional medical treatment decisions, particularly for common chronic conditions in which a variety of treatment choices are available.

We have considered the creation of a meta-model to emulate a single complex simulation model. As an alternative approach, different complexities of simulation models could be developed to personalize treatment and the resulting treatment outcomes (e.g., costs, QALYs) compared. This approach might be viable for simple medical decisions but might be challenging for more complex medical decisions as it would require developing, running, and evaluating a separate simulation model for every possible combination of risks and preferences and potentially other factors such as structural assumptions. In our illustrative example, which has 9 patient-specific risks and preferences, there would be at least 2¹⁸ = 262,144 models to develop, run, and evaluate if one wanted to comprehensively assess which risks and preferences should be included when making a treatment decision. Moreover, the resulting simulation models would likely be less interpretable than the models we develop and more computationally expensive.

Our method for simplifying models for personalization of medical treatment decisions can generate more readily interpretable and less computationally expensive models that achieve much of the health benefits of full personalization. Simplified models of this type may be more likely to be adopted in a clinical setting than complex models, thereby improving patient outcomes compared with no personalization. Moreover, such simplified models can improve equity in patient health outcomes, as all patients can receive the same personalized, high-quality, evidence-based care regardless of setting. Without use of this framework, only those patients with experienced doctors and in higher-resource settings might receive personalized care.

Supplemental Material

sj-doc-1-mdm-10.1177_0272989X211037921 – Supplemental material for Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes

Supplemental material, sj-doc-1-mdm-10.1177_0272989X211037921 for Personalization of Medical Treatment Decisions: Simplifying Complex Models while Maintaining Patient Health Outcomes by Christopher Weyant and Margaret L. Brandeau in Medical Decision Making

Footnotes

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Financial support for this study was provided in part by a National Science Foundation Graduate Fellowship (DGE-114747) and in part by a Stanford University Kaseberg Doolan Graduate Fellowship. It was also provided in part by grant R37-DA15612 from the National Institute on Drug Abuse. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

ORCID iD

Christopher Weyant

Supplemental Material

Supplementary material for this article is available on the Medical Decision Making website at .

References

Tervonen

van Valkenhoef

Buskens

Hillege

Postmus

A stochastic multicriteria model for evidence-based decision making in drug benefit-risk analysis. Stat Med. 2011;30(12):1419–28.

van Valkenhoef

Tervonen

Zhao

de Brock

Hillege

Postmus

Multicriteria benefit-risk assessment using network meta-analysis. J Clin Epidemiol. 2012;65(4):394–403.

Tervonen

Naci

van Valkenhoef

, et al. Applying multiple criteria decision analysis to comparative benefit-risk assessment: choosing among statins in primary prevention. Med Decis Making. 2015;35(7):859–71.

Wen

Zhang

Yang

Two approaches to incorporate clinical data uncertainty into multiple criteria decision analysis for benefit-risk assessment of medicinal products. Value Health. 2014;17(5):619–28.

Broekhuizen

Groothuis-Oudshoorn

CGM

Hauber

Jansen

IJzerman

MJ.

Estimating the value of medical treatments to patients using probabilistic multi criteria decision analysis. BMC Med Inform Decis Mak. 2015;15:102.

Lynd

Najafzadeh

Colley

, et al. Using the incremental net benefit framework for quantitative benefit-risk analysis in regulatory decision-making—a case study of alosetron in irritable bowel syndrome. Value Health. 2010;13(4):411–7.

Choi

Brandeau

Basu

Dynamic treatment selection and modification for personalised blood pressure therapy using a Markov decision process model: a cost-effectiveness analysis. BMJ Open. 2017;7(11):e018374.

Broekhuizen

IJzerman

Hauber

Groothuis-Oudshoorn

CGM

. Weighing clinical evidence using patient preferences: an application of probabilistic multi-criteria decision analysis. Pharmacoeconomics. 2017;35(3):259–69.

Chim

Salkeld

Stockler

Mileshkin

Weighing up the benefits and harms of a new anti-cancer drug: a survey of Australian oncologists. Intern Med J. 2015;45(8):834–42.

10.

Kaltoft

Turner

Cunich

Salkeld

Nielsen

Dowie

Addressing preference heterogeneity in public health policy by combining cluster analysis and multi-criteria decision analysis: proof of method. Health Econ Rev. 2015;5:10.

11.

Weyant

Brandeau

Basu

Personalizing medical treatment decisions: integrating meta-analytic treatment comparisons with patient-specific risks and preferences. Med Decis Making. 2019;39(8):998–1009.

12.

Fröhlich

Balling

Beerenwinkel

, et al. From hype to reality: data science enabling personalized medicine. BMC Med. 2018;16(1):150.

13.

Baxter

Zhou

Zhang

The practical implementation of artificial intelligence technologies in medicine. Nat Med. 2019;25(1):30–6.

14.

Watson

Krutzinna

Bruce

, et al. Clinical applications of machine learning algorithms: beyond the black box. BMJ. 2019;364:l886.

15.

Neumann

Kim

Trikalinos

, et al. Future directions for cost-effectiveness analyses in health and medicine. Med Decis Making. 2018;38(7):767–77.

16.

Jalal

Dowd

Sainfort

Kuntz

Linear regression metamodeling as a tool to summarize and present simulation model results. Med Decis Making. 2013;33(7):880–90.

17.

Yuan

Nian

Meng

A simultaneous calibration and parameter ranking method for building energy models. Appl Energy. 2017;206:657–66.

18.

O’Hagan

Bayesian analysis of computer code outputs: a tutorial. Reliab Eng Syst Saf. 2006;91(10):1290–300.

19.

Wang

Shan

Review of metamodeling techniques in support of engineering design optimization. J Mech Des. 2006;129(4):370–80.

20.

Kleijnen

JPC

Sargent

RG.

A methodology for fitting and validating metamodels in simulation. Eur J Oper Res. 2000;120(1):14–29.

21.

Strong

Oakley

Brennan

Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. Med Decis Making. 2014;34(3):311–26.

22.

Strong

Oakley

Brennan

Breeze

Estimating the expected value of sample information using the probabilistic sensitivity analysis sample: a fast, nonparametric regression-based method. Med Decis Making. 2015;35(5):570–83.

23.

Jalal

Goldhaber-Fiebert

Kuntz

KM.

Computing expected value of partial sample information from probabilistic sensitivity analysis using linear regression metamodeling. Med Decis Making. 2015;35(5):584–95.

24.

Brennan

Chilcott

Kharroubi

O’Hagan

A two level Monte Carlo approach to calculating expected value of perfect information: resolution of the uncertainty in methods. Presented at: 24th Annual Meeting of the Society for Medical Decision Making;October 23, 2002; Baltimore, MD.

25.

Degeling

IJzerman

Lavieri

Strong

Koffijberg

Introduction to metamodeling for reducing computational burden of advanced analyses with health economic models: a structured overview of metamodeling methods in a 6-step application process. Med Decis Making. 2020;40(3):348–63.

26.

Rojnik

Naveršnik

Gaussian process metamodeling in Bayesian value of information analysis: a case of the complex health economic model for breast cancer screening. Value Health. 2008;11(2):240–50.

27.

Andrianakis

Vernon

McCreesh

, et al. Bayesian history matching of complex infectious disease models using emulation: a tutorial and a case study on HIV in Uganda. PLOS Comput Biol. 2015;11(1):e1003968.

28.

Tappenden

Chilcott

Eggington

Oakley

McCabe

Methods for expected value of information analysis in complex health economic models: developments on the health economics of beta interferon and glatiramer acetate for multiple sclerosis. Health Technol Assess. 2004;8(27):iii, 1–78.

29.

Merz

Small

Fischbeck

PS.

Measuring decision sensitivity: a combined Monte Carlo–logistic regression approach. Med Decis Making. 1992;12(3):189–96.

30.

Stevenson

Oakley

Chilcott

JB.

Gaussian process modeling in conjunction with individual patient simulation modeling: a case study describing the calculation of cost-effectiveness ratios for the treatment of established osteoporosis. Med Decis Making. 2004;24(1):89–100.

31.

Alam

Briggs

Artificial neural network metamodel for sensitivity analysis in a total hip replacement health economic model. Expert Rev Pharmacoecon Outcomes Res. 2020;20(6):629–40.

32.

Soeteman

Resch

Jalal

, et al. Developing and validating metamodels of a microsimulation model of infant HIV testing and screening strategies used in a decision support tool for health policy makers. MDM Policy Pract. 2020;5(1):2381468320932894.

33.

Koffijberg

Degeling

IJzerman

Coupé

VMH

Greuter

MJE

. Using metamodeling to identify the optimal strategy for colorectal cancer screening. Value Health. 2021;24(2):206–15.

34.

Zhong

Brandeau

Yazdi

, et al. Metamodeling for policy simulations with multivariate outcomes. Working Paper, Stanford University, 2021.

35.

Timofeev

Classification and regression trees (CART) theory and applications. Berlin: Humboldt University; 2004.

36.

Hothorn

Hornik

Zeileis

ctree: conditional inference trees. Compr R Arch Netw. 2015;1–34.

37.

Kuhn

Johnson

, eds. Classification models. In: Applied Predictive Modeling. New York: Springer; 2013. p 245–460.

38.

Doupe

Faghmous

Basu

Machine learning for health services researchers. Value Health. 2019;22(7):808–15.

39.

Wolpert

Stacked generalization. Neural Netw. 1991;5(2):241–59.

40.

Breiman

Stacked regressions. Mach Learn. 1996;24:49–64.

41.

van der Laan

Polley

Hubbard

. Super learner. J Am Stat Appl Genet Mol Biol. 2007;6:Article25.

42.

Polley

Rose

van der Laan

MJ.

Super learning. In: Targeted Learning: Causal Inference for Observational and Experimental Data. New York: Springer; 2011. p 43–66.

43.

Saloniki

E-C

Malley

Burge

, et al. Comparing internet and face-to-face surveys as methods for eliciting preferences for social care-related quality of life: evidence from England using the ASCOT service user measure. Qual Life Res. 2019;28(8):2207–20.

44.

Neumann

Cohen

Weinstein

MC.

Updating cost-effectiveness—the curious resilience of the $50,000-per-QALY threshold. N Engl J Med. 2014;371(9):796–7.

45.

R Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2018. Available from: https://www.R-project.org/

46.

James

Oparil

Carter

, et al. 2014 evidence-based guideline for the management of high blood pressure in adults: report from the panel members appointed to the Eighth Joint National Committee (JNC 8). JAMA. 2014;311(5):507–20.

47.

Hall

Ambati

Phan

Ideas on interpreting machine learning. O’Reilly Media. 2017. Available from: https://www.oreilly.com/radar/ideas-on-interpreting-machine-learning/. Accessed March 20, 2021.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB