Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

Abstract

Background. Patients may find clinical prediction models more useful if those models accounted for preferences for false-positive and false-negative predictive errors and for other model characteristics. Methods. We conducted a discrete choice experiment to compare preferences for characteristics of a hypothetical mortality prediction model among community-dwelling patients with chronic lung disease recruited from 3 clinics in Philadelphia. This design was chosen to allow us to quantify “exchange rates” between different characteristics of a prediction model. We provided previously validated educational modules to explain model attributes of sensitivity, specificity, confidence intervals (CI), and time horizons. Patients reported their interest in using prediction models themselves or having their physicians use them. Patients then chose between 2 hypothetical prediction models each containing varying levels of the 4 attributes across 12 tasks. Results. We completed interviews with 200 patients, among whom 95% correctly chose a strictly dominant model in an internal validity check. Patients’ interest in predictive information was high for use by themselves (n = 169, 85%) and by their physicians (n = 184, 92%). Interest in maximizing sensitivity and specificity were similar (0.88 percentage points of specificity equivalent to 1 point of sensitivity, 95% CI 0.72 to 1.05). Patients were willing to accept a reduction of 6.10 months (95% CI 3.66 to 8.54) in the predictive time horizon for a 1% increase in specificity. Discussion. Patients with chronic lung disease can articulate their preferences for the characteristics of hypothetical mortality prediction models and are highly interested in using such models as part of their care. Just as clinical care should become more patient centered, so should the characteristics of predictive models used to guide that care.

Keywords

discrete choice experiment patient-centered decision making predictive modeling

Clinical prediction models are increasingly used to guide care. Such models combine historical data with statistical techniques to forecast—reducing uncertainty in prognosis or reducing risks of high-stakes clinical decisions. Those who develop prediction models make choices—implicitly or explicitly—about the relative costs of false-positives, false-negatives, and other model characteristics such as time horizon that help set decision thresholds. Typically absent, however, is input from the patients for whom the models might be relevant. If these prediction models are to usefully guide patient-centered clinical care, they ought to be designed less around the preferences of the modelers and more around the preferences of the patients they aim to serve.

People living with a serious illness face high prognostic uncertainty and wrenching decisions about care goals. Prognostic models, if tailored to the patients making such decisions, might inform choices these patients encounter such as targeted deployment of intensive outpatient management, referral for palliative care, or whether to engage in advance care planning. However, there are technical and logistical barriers to accurately incorporating patient preferences into model development, including difficulties—real or perceived—in identifying an appropriate method for quantifying patient preferences for characteristics of prognostic models. As a consequence, most clinical prediction models, such as those for patients with breast cancer,¹ coronary artery disease,² and chronic obstructive pulmonary disease,^3,4 do not account for patient preferences for different error types or other model characteristics.

Therefore, we sought to quantify preferences for the predictive time horizon (the future period over which a model makes a prediction, e.g., the probability of some event in the next 6 months), sensitivity, specificity, and the confidence interval (CI) around these estimates for a hypothetical mortality prediction model in a population of community-dwelling patients living with advanced chronic lung disease. We employed a discrete choice experiment (DCE) to identify patients’ relative utilities for these model characteristics. We hypothesized that patients would have preferences about tradeoffs between model characteristics that could inform the development of future prediction models that better reflect their goals. Such an approach could also serve as an example of preference elicitation to inform the development of a patient-centered predictive model in other populations.

Methods

We performed a cross-sectional study of adult, community-dwelling patients with chronic lung disease who visited an outpatient pulmonary clinic. We conducted a DCE to determine the part-worth utilities of different characteristics of a hypothetical prediction model that identified the future risk of in-hospital death. We followed best practices for design and reporting of conjoint analysis studies for health-related applications.⁵

Population and Sample Size

We screened the daily appointment schedule at 3 outpatient academic pulmonary clinics within the University of Pennsylvania Health System. Patients were eligible for inclusion if they had any recent International Classification of Disease–10 code for a chronic lung disease (Supplemental Table S1) and also met at least 1 of the following criteria for disease severity: 1) forced expiratory volume in 1 second (FEV₁) <50%, 2) peripheral oxygen saturation <88% recorded during a recent outpatient encounter, or 3) daytime supplemental oxygen therapy documented in the electronic health record. We chose to include patients with higher disease severity because they are more likely to benefit from a prognostic model and likely have different decision-making biases and preferences than healthier patients do.⁶ Patients meeting eligibility criteria were approached by research coordinators on the day of their visit in the waiting area, and patients providing informed consent were enrolled. All participants received $20 for completing the study. There was no additional follow-up. The University of Pennsylvania Institutional Review Board approved the study (#826986).

Task and Study Design

We used 12 forced-choice tasks (Figure 1) with a full-profile, partial-factorial design. Each task required the participant to choose 1 of 2 hypothetical prediction models with different characteristics (Table 1). The levels for each task were chosen using the ChoicEff macro with SAS software (SAS Institute Inc., Cary, NC). An additional task in which one model strictly dominated the other was also included as a check for participant understanding and internal validity (Supplemental File, final task on page 39). Each participant was presented with all 13 tasks in a randomized order.

Figure 1

Sample task from the discrete choice experiment.

Table 1

Attributes and Levels of a Hypothetical Clinical Prediction Model Used in the Discrete Choice Experiment

Attribute	Description	Levels
Sensitivity	The true-positive rate of the prediction model	65, 85, 95 (%)
Specificity	The true-negative rate of the prediction model	65, 85, 95 (%)
Confidence interval	The 95% confidence interval of the sensitivity and specificity	±3, ±20 (%)
Time horizon	The future period over which the model makes a prediction	1, 12, 60 (months)

Instrument Development

Attributes were selected based on input from the study team with expertise in the development of clinical prediction models. Specifically, sensitivity, specificity, CIs, and predictive time horizon were chosen because of their centrality to the model development process and influence on a model’s eventual predictions. In addition, the inherent tradeoffs between these attributes offered several opportunities to link patient preferences directly to analytic choices in model development.

First, a classification threshold that determines a particular combination of sensitivity and specificity could be determined with knowledge of the relative utilities of the predicted outcomes. For example, in the case of predicting mortality among patients living with lung disease, this decision could account for the value of correctly identifying the true risk of death, whether high or low. This must be balanced against the cost of mistakenly predicting high or low risk of death when in fact the opposite is true.

Second, the role of CIs in communicating risk information is not yet clearly defined.⁷ CIs around predicted probabilities that are perceived as narrow may convey a degree of certainty around a prediction that could engender trust in the model. How a patient defines wide and narrow is unknown and may vary with respect to the consequences associated with a prediction. Analytically, the width of the CI depends on the number of observations used to train the model, the number of variables and their signal, and the extent of model overfitting, among other factors.

Third, predictive model developers must select a relevant time horizon based on an optimal tradeoff between performance and utility. That is, utility as a function of the predictive time horizon for a prediction model most likely follows an inverted U-shaped curve, while its performance follows the opposite pattern. For example, for a patient in clinic, it is probably easy to train a high-performing model to predict death within the next hour. But a model at this time horizon likely yields little utility because such a prediction is obvious in most cases. Similarly, it would be easy to achieve high performance yet uninformative to predict death occurring sometime in the next 100 years. As the time horizon is shifted away from either temporal extreme, the predictive task becomes more difficult, thus increasing the marginal utility of relevant prognostic information for middling time horizons.⁸ The choice of the predictive time horizon should be long enough to allow for some intervention or change in behavior while being short enough to feel relevant to a decision in the present.⁹

The feasibility of communicating and understanding these statistical concepts using best practices for risk communication^10–12 was demonstrated in a previous pilot study.¹³ Levels for each attribute were selected to include those of commonly published prediction models.^14–16 We also chose attribute levels to avoiding extreme values that could induce a grounding effect based on guidelines for DCE development in the absence of prior data from similar experiments.⁵ The model outcome of in-hospital death used for all tasks was selected based on results of a combined free-listing and multiple-choice exercise with 30 volunteers in the outpatient clinic. Participants in this pilot survey were not included in the main study. A copy of the final instrument is available in the Supplement.

Additional Measures

Participants reported their sociodemographic characteristics and their interest in the results of a clinical prediction model for use by themselves and by their doctor, and they completed the short Numeracy Understanding in Medicine instrument (S-NUMi).¹⁷ The S-NUMi is an 8-item instrument that measures multiple domains of numeracy, is strongly associated with other commonly used numeracy instruments, and is also associated with understanding of statistical concepts such as sensitivity and specificity that were presented to patients in this study.^13,17 The S-NUMi score ranges from 0 (low) to 8 (high).

Statistical Analysis

For the primary analysis, we calculated the odds of choosing one hypothetical prediction model over another using an explanatory, multivariable logistic regression model adjusted for each attribute. We included a crossed random effect for both participant and task. However, the variance of the random intercept for participant was near zero and so was removed in subsequent analyses and the model refit with a single random intercept for task. In a separate explanatory model, we also examined interactions between sensitivity and specificity and between sensitivity and time horizon.

In the absence of prior literature on preferences for predictive model characteristics, we assumed a linear utility function.¹⁸ As a check against this assumption, explanatory models were fit separately using attributes as continuous and categorical variables. Model fit between these approaches was compared with the Akaike information criterion (AIC) and analysis of variance (ANOVA). Ratios of fitted coefficients and their CIs were calculated using the delta method with a nonlinear Wald test to determine the “exchange rates” between characteristics of the hypothetical prediction model.^19,20 The ratios of the marginal rates of substitution among pairs of attributes represent the preferences for the rate at which participants might trade off one attribute for the other.

Given the complexity of the presented information in each task, we also conducted a sensitivity analysis in which we repeated the primary analysis while excluding all those participants who chose the strictly dominated prediction model in the internal check question and another in which we excluded those with any dominant preferences (i.e., those who always chose the prediction model with the better of a single attribute). With multiple modeling approaches available for analysis of DCE data,²¹ we also repeated the primary analysis using a conditional logit model to see how the results were affected by specification of the explanatory model and also refit the main model using a zero intercept.

As a secondary analysis, we fit separate explanatory models adjusted for self-perceived risk and for numeracy. Self-perceived risk was modeled as a continuous variable between 0 and 1 based on each participant’s assessment of their own risk of dying in the hospital in the next 12 months. We adjusted for numeracy using the S-NUMi score as described above modeled as a continuous variable. We compared characteristics between groups using the chi-squared and t test for categorical and continuous variables, respectively. Binomial CIs were reported for events with binary outcomes. Using a mixed-effects logistic regression model, to detect a 0.65 log-odds effect across all parameters for main effects and 2 interaction terms with $a = 0.05$ and $β = 0.20$ , we estimated a required sample size of 200.²² All analyses were conducted in R version 3.5.0 with additional packages^23–26 and in Stata version 14.2 (StataCorp LP, College Station, TX).

Results

We surveyed 200 patients during their visit to an outpatient pulmonary clinic (Figure 2) between January 31, 2018, and February 14, 2019. Of these, 118 (59%) were women, 127 (64%) were white, and the median age was 68 years (interquartile range [IQR], 59 to 75 years; Table 2). The task with a choice of a strictly dominant option was answered correctly by 189 (94.5%, 95% CI 90.4 to 97.2) participants.

Figure 2

Cohort screening and enrollment.

Table 2

Characteristics of the Study Population

Variable	Count (%)
Enrolled patients	200
Women	118 (59)
Race
Asian	2 (1)
Black or African American	63 (32)
Mixed race or multiracial	4 (2)
Other	4 (2)
White or Caucasian	127 (64)
Ethnicity
Unknown	1 (1)
Hispanic	3 (2)
Non-Hispanic	196 (98)
Education
Associate’s degree (2-y college)	44 (22)
Bachelor’s degree (4-y college)	38 (19)
Doctoral degree (PhD, EdD, MD, JD, ScD, DDM)	9 (5)
GED or equivalent	6 (3)
High school (grades 9–12)	86 (43)
Master’s degree (MBA, MPH, MSW, MSEd, MS, MFA)	14 (7)
Middle school (grades 6–8)	3 (2)
Age, median (interquartile range)	68 (59 to 75)
Marital status
Divorced	34 (17)
Married or domestic partnership	102 (51)
Single, never married	42 (21)
Widowed	22 (11)
s-NUMI score, median (interquartile range)	5 (4 to 6)

Patients’ interest in predictive information was high for use by themselves (n = 169, 85%) and by their physicians (n = 184, 92%). In unadjusted analyses, interest did not differ by gender, race, numeracy, or education (P > 0.05 for all comparisons). However, patients who were interested in using predictive information themselves were younger (mean age 65.5 v. 71.0 years, 95% CI for difference 1.44 to 9.45, P = 0.009) and had higher estimates of their own 1-year in-hospital mortality (10.7% v. 4.0%, 95% CI for difference 2.8% to 10.5%, P < 0.001). Patients who were interested in their physicians using predictive information were also younger (mean age 65.9 v. 71.9 years, 95% CI for difference 0.91 to 11.02, P = 0.023), but their mortality risk was not significantly different from patients who did not express such interest (10.1% v. 5.2%, 95% CI for difference −0.1% to 9.8%, P = 0.055).

The odds of choosing a given model increased along with improvements in all attribute categories (Figure 3). Estimated coefficients of the explanatory model using continuous rather than categorical attributes are reported in Supplemental Table S3. There was no difference in fit between these 2 model approaches as determined by AIC (4026 and 4020) and ANOVA (P = 0.999).

Figure 3

Estimated regression coefficients of each attribute modeled as a categorical variable and each task with a random intercept. Coefficients represent relative utilities for each level of each attribute associated with a hypothetical mortality prediction model.

Patients’ interests in maximizing sensitivity and specificity were similar (0.88 percentage points of specificity equivalent to 1% point of sensitivity, 95% CI 0.72 to 1.05; Table 3). Patients were willing to accept a reduction of 6.10 months (95% CI 3.66 to 8.54) in the predictive time horizon for a 1% increase in specificity. Examples of equivalent hypothetical prediction models based on these exchange rates are presented in Figure 4.

Table 3

Ratios of Coefficients as Log Odds from the Mixed-Effects Logistic Regression Model

Numerator	Denominator	Ratio	95% CI	Interpretation
Sensitivity	Specificity	0.88	0.72 to 1.05	“I would accept a 1% reduction in sensitivity for a 0.88% increase in specificity.”
	Time horizon	5.40	3.08 to 7.71	“I would accept a 1% reduction in sensitivity for a 5.4 month increase in the predictive time horizon.”
	Confidence interval	−0.28	−0.31 to −0.24	“I would accept a 1% reduction in sensitivity for a 0.28% reduction in the half-width of the confidence interval.”
Specificity	Time horizon	6.10	3.66 to 8.54	“I would accept a 1% reduction in specificity for a 6.1 month increase in the predictive time horizon.”
	Confidence interval	−0.31	−0.35 to −0.27	“I would accept a 1% reduction in specificity for a 0.31% reduction in the half-width of the confidence interval.”
Confidence interval	Time horizon	−19.6	−26.9 to −12.3	“I would accept a 1% increase in the half-width of the confidence interval for a 19.6 month increase in the predictive time horizon.”

Ratios are equivalent to “exchange rates” between different characteristics of a mortality prediction model.

Figure 4

Examples of models with equivalent utility based on estimated exchange rates.

Self-reported probability of 1-year in-hospital death ranged from 0% to 79%, with median a of 0% (IQR 0 to 10), and it was uncorrelated with age (Pearson’s r 0.08, 95% CI −0.06 to 0.22). Neither baseline perceived mortality risk (odds ratio [OR] 1.00, 95% CI 1.00 to 1.00) nor numeracy (OR 1.00, 95% CI 0.95 to 1.06) was associated with choice of a predictive model after adjustment for model attributes. In addition, the estimated odds ratios for each attribute were not meaningfully changed following adjustment for numeracy (Supplemental Table S4) or perceived risk of death (Supplemental Table S5).

We observed 11 (5.5%) participants who incorrectly chose the strictly dominated model and 63 (31.5%) who exhibited dominant preferences for a single attribute, the most common of which was the CI (Supplemental Figure S1). When separately or together excluding these categories of participants, we observed small numeric differences in coefficient estimates and their ratios, but the direction of the effects and their relative magnitudes remained unchanged (Supplemental Tables S6, S7, S8). Similarly, when repeating the primary analysis using a conditional logit model, there were some numeric differences in the estimates, while the relative effect sizes and directions were preserved (Supplemental Tables S9 and S10). When repeating the primary analysis with a zero intercept, we also observed small numeric changes but with the same direction and relative magnitude of the effects (Supplemental Table S11).

Discussion

This study of outpatients with chronic lung diseases provides several insights regarding the usefulness and future design of patient-centered prediction models. First, we found that patients express high interest in using mortality prediction models in clinical care, with strong majorities saying they would be interested in using such models themselves and having their physicians use them. Although this interest was consistent among patients living with chronic lung disease across many demographic categories, older patients were less likely to express interest in the use of such models.

Second, we demonstrated the feasibility of quantifying specific “exchange rates” for prediction model characteristics. These findings have several implications for analytic choices made early in the development of mortality prediction models by investigators studying patients with life-limiting lung disease. This approach also serves as a generalizable demonstration of an often-overlooked step in the patient-centered development of clinical prediction models.

The fact that “exchange rates” can be quantified directly centers the preferences of eventual users (patients in this case) in analytic decisions typically made by researchers for default or convenience reasons. For example, the choice of a classification threshold for predicting a binary outcome depends on the relative utilities of sensitivity and specificity. In the development of a previously published prediction model to identify patients with chronic obstructive pulmonary disease at high risk of mortality and therefore appropriate for palliative care, a sensitivity of 100% was chosen so as to minimize the false-negative rate.³ However, the study does not account for the potential harms of false-positive errors because of patient and family distress²⁷ and from overburdening the palliative care referral service.²⁸ In contrast, our findings suggest that, if anything, false-negative errors may be more acceptable to such patients than false-positive errors. Although we did not ask follow-up questions about the reasons for patients’ stated preferences, we hypothesize that this preference, if real, may be due to desires to avoid unnecessary stress associated with a false prediction of impending death. These 2 studies cannot be directly compared as they tested different outcomes and associated interventions, and the interval between predictions (e.g., 1-time prediction v. annual) was not specified. Clarifying the tradeoff between sensitivity and specificity, and hence of false-positive and false-negative errors, in a specific clinical circumstance will support the development of mortality prediction models that are aligned with patient preferences.

The quantification of utilities for sensitivity and specificity also permits the use of decision curve analysis and net benefit approaches as methods of selecting the optimal classification threshold. These approaches are powerful and visually intuitive, but they rely on the enumeration of utilities for different predicted outcomes. In the original article describing decision curve analysis, the authors present an estimate of 9-to-1 for the disutility of a false-negative error compared with a false-positive one.²⁹ This estimate is taken from a hypothetical surgeon’s preference for balancing the risks of cancer recurrence with urinary incontinence and impotence. Future work on prediction modeling in any clinical scenario should quantify these tradeoffs empirically and recognize that preferences between patients, clinicians, and other stakeholders may diverge. Other tradeoffs evaluated in this study are also directly relevant to analytic decisions in model development. The finding that patients preferred longer time horizons is consistent with some published models.^3,15,30 While 30-day time horizons are of interest because of national policies around hospital readmission,³¹ our findings suggest that predictive information over this time horizon is likely less relevant to patients.^14,32,33 In addition, the decisions made by the developers of predictive models about trading off sensitivity or specificity and the predictive time horizon should also be sensitive to user preferences for their relative part-worth utilities. As the predictive time horizon increases, a model will more easily capture events of interest, and thus sensitivity will increase at the cost of specificity. The most relevant time horizon will depend on the planned intervention in response to the prediction and may depend on particular patient preferences for temporal discounting, which were not accounted for in this study.

The strong preferences we observed for narrower CIs suggest that statistical uncertainty measurements are important for patients relying on a mortality prediction model.⁷ This is in contrast to prior work in other contexts suggesting that the CI may not be influential in decision making.³⁴ These discordant findings may also suggest heterogeneity among individuals in their ability to tolerate ambiguity.³⁵ Strong preferences for narrow CIs highlight the need to reduce variance in predictions through sufficient data collection before any model development begins and through reduction of overfitting in the training phase.

Many participants also exhibited dominant preferences for CIs, for which there are at least 2 interpretations.³⁶ First, for some patients, the reliability of a model’s predictions (indicated by the narrowness of the CI) may be the single most important feature upon which the value of the model rests. In such a case of noncompensatory decision making, there are no other attributes that could sufficiently offset the disutility associated with poor reliability. Second, there may be a hidden threshold effect not detected by the levels tested in this study. In such a scenario, there may be a maximum CI width above which any model is deemed unreliable and thus of no value to a decision maker. If the threshold were at 10%, it would not have been detected in this analysis, which tested CIs of ±3% and ±20%. However, this study design was not able to determine the underlying decision-making strategy that produced these results, including elimination by aspects, lexicographic ordering of preferences, or others.^37,38 In any case, these findings suggest that increased attention to reporting of CIs in prediction model evaluation is warranted.

This study should be interpreted in light of its limitations. First, the hypothetical prediction model presented in the DCE tasks included a composite outcome of mortality and hospitalization, which may be difficult for patients to predict and for which a particular intervention may be difficult to justify. All-location mortality may be a more relevant outcome in some settings, and so findings should not be generalized to prediction models that provide risk scores for different outcomes. Second, there is likely a diminishing marginal utility at the extremes of all model performance measures that was unable to be assessed here but may be relevant for determining decision thresholds at extremes of performance. Third, results may not be generalizable to other populations, as this sample is from a single tertiary referral health system and did not include some common lung diseases, such as asthma. Finally, we did not vary the color of the figures presented to patients and thus could not determine their effect on the preferences for different attributes.

In conclusion, it is possible to quantify the tradeoffs between performance and other characteristics of a hypothetical mortality prediction model based on patients’ stated preferences. Among outpatients with chronic lung disease, interest in predictive information for future mortality risk is very high. These results should inform analytic choices in model development for predicting in-hospital mortality in this population and serve as an example for developing patient-centered prediction models in other populations.

Supplemental Material

dce_supplement_R1_2.18.2020.rjf_online_supp – Supplemental material for Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

Supplemental material, dce_supplement_R1_2.18.2020.rjf_online_supp for Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment by Gary E. Weissman, Kuldeep N. Yadav, Trishya Srinivasan, Stephanie Szymanski, Florylene Capulong, Vanessa Madden, Katherine R. Courtright, Joanna L. Hart, David A. Asch, Sarah J. Ratcliffe, Marilyn M. Schapira and Scott D. Halpern in Medical Decision Making

Supplemental Material

Survey_instrument.rjf_online_supp – Supplemental material for Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment

Supplemental material, Survey_instrument.rjf_online_supp for Preferences for Predictive Model Characteristics among People Living with Chronic Lung Disease: A Discrete Choice Experiment by Gary E. Weissman, Kuldeep N. Yadav, Trishya Srinivasan, Stephanie Szymanski, Florylene Capulong, Vanessa Madden, Katherine R. Courtright, Joanna L. Hart, David A. Asch, Sarah J. Ratcliffe, Marilyn M. Schapira and Scott D. Halpern in Medical Decision Making

Footnotes

Acknowledgements

We are grateful to Thomas Valley, MD, MSc, at the University of Michigan for insightful comments on an early draft of this article and to all of the patients who agreed to participate in the study.

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr. Weissman received support from NIH K23HL141639, NIH L30HL138788, Penn LDI/CHIBE Roybal Center (P30AG034546). Dr. Halpern received support from NIH K24HL143289. Financial support for this study was provided in part by multiple grants from the National Institutes of Health. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

Authors’ Note

This work was completed by faculty and staff primarily within the Division of Pulmonary, Allergy, and Critical Care Medicine and within the Palliative and Advanced Illness Research (PAIR) Center, both in the Department of Medicine, Perelman School of Medicine, University of Pennsylvania. Preliminary results of this study were presented as a poster at the American Thoracic Society International Conference 2019 in Dallas, Texas.

ORCID iDs

Gary E. Weissman

Joanna L. Hart

Marilyn M. Schapira

Supplemental Material

Supplementary material for this article is available on the Medical Decision Making Web site at .

References

Karapanagiotis

Pharoah

PDP

Jackson

Newcombe

PJ.

Development and external validation of prediction models for 10-year survival of invasive breast cancer. Comparison with PREDICT and CancerMath. Clin Cancer Res. 2018;24(9):2110–5. doi:10.1158/1078-0432.CCR-17-3542

Rapsomaniki

Shah

Perel

, et al. Prognostic models for stable coronary artery disease based on electronic health record cohort of 102 023 patients. Eur Heart J. 2014;35(13):844–52. doi:10.1093/eurheartj/eht533

Duenk

Verhagen

Bronkhorst

, et al. Development of the ProPal-COPD tool to identify patients with COPD for proactive palliative care. Int J Chron Obstruct Pulmon Dis. 2017;12:2121–8. doi:10.2147/COPD.S140037

Almagro

Yun

Sangil

, et al. Palliative care and prognosis in COPD: A systematic review with a validation cohort. Int J Chron Obstruct Pulmon Dis. 2017;12:1721–9. doi:10.2147/COPD.S135657

Bridges

JFP

Hauber

Marshall

, et al. Conjoint analysis applications in Healtha Checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14(4):403–3. doi:10.1016/j.jval.2010.11.013

Halpern

Arnold

RM.

Affective forecasting: an unrecognized challenge in making serious health decisions. J Gen Intern Med. 2008;23(10):1708–12. doi:10.1007/s11606-008-0719-5

Politi

Han

PKJ

Col

NF.

Communicating the uncertainty of harms and benefits of medical interventions. Med Decis Making. 2007;27(5):681–95. doi:10.1177/0272989X07307270

Goldstein

Pencina

Montez-Rath

Winkelmayer

WC.

Predicting mortality over different time horizons: which data elements are needed?

J Am Med Inform Assoc. 2017;24(1):176–81. doi:10.1093/jamia/ocw057

Rolnick

Weissman

GE.

Early warning systems: the neglected importance of timing. J Hosp Med. 2019;14(7):3.

10.

Hawley

Zikmund-Fisher

Ubel

Jancovic

Lucas

Fagerlin

The impact of the format of graphical presentation on health-related knowledge and treatment choices. Patient Educ Couns. 2008;73(3):448–55. doi:10.1016/j.pec.2008.07.023

11.

Fagerlin

Zikmund-Fisher

Ubel

PA.

Helping patients decide: ten steps to better risk communication. J Natl Cancer Inst. 2011;103(19):1436–43. doi:10.1093/jnci/djr318

12.

Talboy

Schneider

SL.

Improving understanding of diagnostic test outcomes. Med Decis Making. 2018;38(5):573–83. doi:10.1177/0272989X18758293

13.

Weissman

Yadav

Madden

, et al. Numeracy and understanding of quantitative aspects of predictive models: a pilot study. Appl Clin Inform. 2018;9(3):683–692. doi:10.1055/s-0038-1669457

14.

Lindenauer

Grosso

Wang

, et al. Development, validation, and results of a risk-standardized measure of hospital 30-day mortality for patients with exacerbation of chronic obstructive pulmonary disease: Hospital Mortality Measure for COPD. J Hosp Med. 2013;8(8):428–35. doi:10.1002/jhm.2066

15.

Smith

L-JE

Moore

Ali

Smeeth

Stone

Quint

. Prognostic variables and scores identifying the end of life in COPD: a systematic review. Int J Chron Obstruct Pulmon Dis. 2017;12:2239–56. doi:10.2147/COPD.S137868

16.

Chung

Winship

Phung

Lake

Waterer

Five-year outcome in COPD patients after their first episode of acute exacerbation treated with non-invasive ventilation. Respirology. 2010;15(7):1084–91. doi:10.1111/j.1440-1843.2010.01795.x

17.

Schapira

Walker

Miller

, et al. Development and validation of the numeracy understanding in medicine instrument short form. J Health Commun. 2014;19(suppl 2):240–53.

18.

van der Pol

Currie

Kromm

Ryan

. Specification of the utility function in discrete choice experiments. Value Health. 2014;17(2):297–301. doi:10.1016/j.jval.2013.11.009

19.

Feiveson

AH.

Stata | FAQ: explanation of the delta method. Available from: https://www.stata.com/support/faqs/statistics/delta-method/

20.

Gary

Oehlert. A note on the delta method. American Statistician. 1992;46(1):27–9. doi:10.1080/00031305.1992.10475842

21.

Hauber

González

Groothuis-Oudshoorn

CGM

, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force. Value Health. 2016;19(4):300–15. doi:10.1016/j.jval.2016.04.004

22.

de Bekker-Grob

Donkers

Jonker

Stolk

EA.

Sample size requirements for discrete-choice experiments in healthcare: a practical guide. Patient. 2015;8(5):373–84. doi:10.1007/s40271-015-0118-z

23.

R Core Team. R: A Language and Environment for Statistical Computing. Vienna (Austria): R Foundation for Statistical Computing; 2016.

24.

Bates

Mächler

Bolker

Walker

Fitting linear mixed-effects models using lme4. J Stat Softw. 2015;67(1):1–48. doi:10.18637/jss.v067.i01

25.

Dowle

Srinivasan

Data.Table: Extension of ‘data.Frame’. 2018. Available from: https://github.com/Rdatatable/data.table

26.

Wickham

Ggplot2: Elegant Graphics for Data Analysis. New York: Springer-Verlag; 2016.

27.

Elmore

Barton

Moceri

Polk

Arena

Fletcher

SW.

Ten-year risk of false positive screening mammograms and clinical breast examinations. N Engl J Med. 1998;338(16):1089–96. doi:10.1056/NEJM199804163381601

28.

Courtright

Cassel

Halpern

SD.

A research agenda for high-value palliative care. Ann Intern Med. 2018;168(1):71. doi:10.7326/M17-2164

29.

Vickers

Elkin

EB.

Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74. doi:10.1177/0272989X06295361

30.

Blagev

Collingridge

Rea

, et al. The Laboratory-Based Intermountain Validated Exacerbation (LIVE) score identifies chronic obstructive pulmonary disease patients at high mortality risk. Front Med (Lausanne). 2018;5. doi:10.3389/fmed.2018.00173

31.

Ody

Msall

Dafny

Grabowski

Cutler

DM.

Decreases in readmissions credited to Medicare’s program to reduce hospital readmissions have been overstated. Health Aff. 2019;38(1):36–43. doi:10.1377/hlthaff.2018.05178

32.

Chang

Sullivan

Karalus

Mills

McLachlan

Hancox

RJ.

Predicting early mortality in acute exacerbation of chronic obstructive pulmonary disease using the CURB65 score. Respirology. 2011;16(1):146–51. doi:10.1111/j.1440-1843.2010.01866.x

33.

Echevarria

Steer

Heslop-Marshall

, et al. Validation of the DECAF score to predict hospital mortality in acute exacerbations of COPD. Thorax. 2016;71(2):133–40. doi:10.1136/thoraxjnl-2015-207775

34.

Schapira

Aggarwal

Akers

, et al. How patients view lung cancer screening: the role of uncertainty in medical decision making. Ann Am Thorac Soc. 2016;13(11):1969–76. doi:10.1513/AnnalsATS.201604-290OC

35.

Han

PKJ

Klein

WMP

Lehman

Massett

Lee

Freedman

. Laypersons’ responses to the communication of uncertainty regarding cancer risk estimates. Med Decis Making. 2009;29(3):391–403. doi:10.1177/0272989X08327396

36.

Scott

Identifying and analysing dominant preferences in discrete choice experiments: An application in health care. J Econ Psychol. 2002;23(3):383–98. doi:10.1016/S0167-4870(02)00082-X

37.

Tversky

Elimination by aspects: a theory of choice. Psychol Rev. 1972;79(4):281–99.

38.

Rothrock

Yin

Integrating compensatory and noncompensatory decision-making strategies in dynamic task environments. In: Pardalos

Kugler

Smith

Connolly

Son

Y-J

, eds. Decision Modeling and Behavior in Complex and Uncertain Environments. Vol 21. New York: Springer; 2008:125–41. doi:10.1007/978-0-387-77131-1_6

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

7.02 MB

0.05 MB