Translating Evidence Updates to International Standards: Is More Certainty Needed for International Standards on Decision Aids?

Abstract

Medical Decision Making recently published evidence updates from the International Panel of Decision Aid Standards (IPDAS) that will inform the next iteration of decision aid standards.^1–14 These evidence updates are from an international group of experts and have the goal of providing the state of the science on patient decision aid development and implementation (e.g., adoption, use, and sustained use of decision aids). Unfortunately, the updates are variably supported by the questions and evidence needed for standards, raising questions about implications for standard development.

Standards are meant to provide best evidence rules for action by an established authority.¹⁵ As such, they should ask and answer the questions about what processes and interventions (and/or measures) work best, use the highest-quality evidence to answer these questions, rigorously determine the strength and certainty of evidence for making recommendations, and set a standard only when there is moderate to high certainty of substantial net benefit (e.g., benefits outweigh harms and costs). Absent this, standards could worsen overall quality rather than improve it.

For decision aid standards, questions about what processes and interventions work best center around what development processes and decision aid components produce decision aids most likely to optimize decision making, health outcomes, and implementation and what implementation processes and supports are most likely to optimize implementation of decision aids, decision making, and health outcomes. Optimization generally means “produces the most net benefit,” but ultimately it might also mean “produces the most net value” (i.e., the most net benefit plus the best experience of care (e.g., care as patient centered, equitable, efficient, and sustainable). Questions about what measures work best is a separate question and focuses instead on what measures most accurately assess what is intended and the truth at one point in time and repeatedly over time.

For questions of what processes and interventions work best, the highest-quality evidence is generally considered to come from 1) systematic reviews or meta-analyses of large, well-done randomized controlled trials or randomized comparative effectiveness trials or 2) from a single very large, well-done randomized comparative effectiveness trial comparing all relevant interventions of interest in the same population and setting using the same methods and outcomes.^16,17 For questions of what measures work best, the highest-quality evidence instead comes from psychometric studies (e.g., factor analysis and validity studies)¹⁸ or from studies that assess the accuracy of measures relative to the best possible measure of the truth and the test-retest reliability over time. Unfortunately, needed evidence is often not available and, perhaps in part because of it, IPDAS has taken a variety of approaches to the evidence to inform decision aid standards (e.g., systematic reviews and meta-analyses of randomized trials, narrative and scoping reviews of all types of study designs, and systematic reviews of expert processes and decision aid content).

When evidence is available, individually and in aggregate, it must be of sufficient strength and certainty to warrant a standard. For decision aid standards, strong and certain evidence directly answers questions of interest, provides an assessment of the net benefit of what is being studied (or, for measurement studies, the internal consistency reliability, content and construct validity, accuracy, and/or test-retest reliability of measures), has low or moderate risk of scientific bias, is precise, is consistent in its findings, and is broadly applicable to the settings in which it is intended to be applied.^19,20 Table 1 provides an assessment of IPDAS evidence and its strength and certainty for supporting standards.

Table 1

Overview of IPDAS Evidence and Its Strength to Support Standards

Decision Aid Development, Intervention, or Implementation Components	Question of Interest for Standard Development	Approach to Evidence by IPDAS in 2021	Net Benefit	Directness of Evidence^a,b	Risk of Bias^a	Precision^a	Consistency^a	Applicability^a
Questions best answered by randomized trials (or meta-analyses thereof)
S ystematic development process¹²	What development process yields the highest-quality decisions and the best outcomes?	Review of a recent systematic review of expert development processes of decision aids C ontent analysis of all decision aids citing the last IPDAS development process	Not evaluated	Indirect (do not directly test outcome)	Not assessed	Not assessed	Inconsistent	Unclear
S ynthesis of scientific evidence³	What type of scientific evidence and evidence translation processes yield the highest-quality decision aids and best outcomes?	Narrative review of definitions and key processes related to incorporating evidence into decision aids C ontent analysis of the types of evidence incorporated into decision aids	Not evaluated	Indirect (do not directly test outcome)	Not assessed: narrative review	Not applicable: narrative review	Not assessed	Unclear
Presenting probabilities^2,11	What numerical and graphical formats yield the best understanding of net benefit?	Narrative review (all study designs)	Not evaluated	Indirect (do not directly test outcome)	Not assessed: narrative review	Not applicable: narrative review	Not assessed	Unclear
P roviding balanced information⁵	What features of presentation yield a perception of complete, unbiased, and neutral information about available options and their net benefit?	Scoping review on: • conceptualizations of balance, • alternate presentation of balance, and • measurement of perceived balance	Not evaluated	Indirect (do not directly test outcome)	Not assessed: scoping review	Not applicable: scoping review	Not assessed	Unclear
Values clarification¹³	What values clarification methods lead to the best decisions (e.g., value concordant decisions to which patients will adhere)?	Meta-analysis (RCTs and comparative effectiveness RCTs)	Not evaluated	Direct	No overall assessment of risk of bias (but 9 of 43 high risk of bias on at least 1 quality element)	Precise	Inconsistent	Variable, depending on type of method (i.e., most effective method [mathematical modeling] not likely to be widely applicable
Coaching and guidance⁷	Do coaching or guidance interventions improve decision making and health outcomes alone or as an adjunctive support to decision aids?	Scoping review on guidance (RCTs only) M eta-analyses on coaching (RCTs only)	Both benefits and harms assessed but not quantitatively	Direct	Guidance: Not reported C oaching: No overall assessment of risk of bias (but 5 of 21 high risk of bias on at least 1 quality element)	Guidance: Not reported C oaching: Variable depending on comparison and outcome	Guidance: Not reported C oaching: Variable depending on comparison and outcome	Guidance: Not reported C oaching: Variable depending on type of method and route of delivery
A ddressing health literacy in IPDAS⁶	What features of decision aids allow the best decisions for those with low, marginal, and adequate health literacy?	Content analysis of readability level of decisions aids in RCTs	Not assessed	Indirect (do not directly test outcome)	Not applicable: content analysis	Not applicable: content analysis	Not assessed	Unclear
D ecision aids for socially disadvantaged populations¹⁴	Do decision aids and/or their adjunctive supports provide equal outcomes in socially disadvantaged (and nondisadvantaged) groups?	Meta-analysis (RCTS)	Both benefits and harms assessed	Direct	Variable depending on outcome	Variable depending on outcome	Variable depending on outcome	Unclear applicability (dependent on access and resources in settings in which to be applied)
Personal stories⁸	Do personal stories improve engagement with decision aids, knowledge, motivation, values clarity, skills, behavior, decision making, and/or health outcomes?	Scoping review (all study designs)	Both benefits and harms assessed but not quantitatively	Direct	Not assessed: scoping review	Not applicable: scoping review	Not assessed	Unclear
I mplementation of decision aids⁴	What implementation processes and supports provide optimal adoption, use, and sustainability of decision aids in practice?	Rapid realist review of factors related to implementation	Not applicable	Indirect (do not directly test outcome)	Not assessed: realist review	Not applicable: realist review	Not assessed	Unclear
A ddressing conflicts of interest⁹	What strategies to reduce and/or manage conflicts of interest produce the most unbiased and trustworthy decision aids?	Narrative review of: • effects of alternate strategies to manage conflicts of interests and • public perceptions of conflicts of interest	Benefits and harms assessed but not quantitatively	Direct	Alternate strategies in health care delivery: not assessed; narrative review	Alternate strategies in health care delivery: not assessed; narrative review	Not assessed	Unclear
Questions best answered by psychometric studies (for scale measures) and diagnostic accuracy studies (for clinical measures)
U sing decision process and quality measures¹⁰	What process and outcome measures provide the best validity and reliability in capturing decision quality?	Content analysis of measures in RCTs of decision aids	Not applicable	Indirect	Not applicable: content analysis	Not applicable: content analysis	Not assessed	Unclear

RCTs, randomized controlled trials.

Shaded boxes indicate that these features are not generally assessed for the study design chosen.^bDirectness of evidence is used broadly here to indicate a direct answer to the question of interest, a direct assessment of the outcome of interest, and/or a direct comparison of interventions of interest.

Table 1 illustrates that, in many cases, IPDAS authors and leaders do not ask and answer the questions needed for standard development or provide strong and certain enough evidence to support an absolute standard (i.e., one to which every decision aid is expected to adhere for certification and widespread use). For instance, the questions asked do not always focus on what is best, but instead on what has been done or what is believed to be best. Further, the methods do not always rely on a single, large, well-done trial or meta-analyses of randomized trials to answer what is best, require quantitative estimates of net benefit, or sufficiently acknowledge the risk of scientific bias, the lack of precision, the indirect nature and/or inconsistency (e.g., heterogeneity) of data or the potential problems with applicability to real-world settings. All of these issues should make us question whether it is appropriate to establish an absolute standard.

Leaving measurement studies aside, some readers may question whether or not randomized trials (or meta-analyses thereof) can answer every question regarding what processes and interventions are best; I believe, with a little ingenuity, they can. Randomized trials can certainly make the relevant comparisons of alternate content, design, format, delivery route, delivery routine, and messenger for development processes, decision aids, and implementation processes and supports. Further, randomized subgroup analyses can determine what works, for whom, in what situations and settings. The keys are to use theory to guide development of processes and interventions, study what works in both ideal and real-world settings, recruit representative samples of practices and patients, and, as several of the IPDAS chapter authors have taught us, to disaggregate both processes and interventions into their component parts for study, making what differs between study groups only what is under study. Successful parts can then be reaggregated into optimized decision aids, supports, and processes, which can then be tested.

At its core, IPDAS is trying to promote the development and implementation of high-quality tools to help people make and adhere to decisions; thus, it seems that they should help decision aid developers and implementers make high-quality decisions about how to build the best tools and get people to use them and how to best measure tool effects. In the short term, this means that authors and leaders need to be transparent about the strength and certainty of evidence on what is best and encourage Delphi participants to make absolute standards only when there is moderate or high certainty of substantial net benefit. In the long run, it also means authors and leaders need to continue to refine the questions asked and answered, continue to work toward providing the highest-quality answers for every questioned asked, and advocate for needed evidence.

IPDAS has done this field a tremendous service, and its current standards may truly represent what is best, but I, for one, would like a little more certainty if I am to be held to an absolute standard on what is best.

Footnotes

The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

The author received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Stacey L. Sheridan

References

Stacey

Volk

Leads

IEU

. The International Patient Decision Aid Standards (IPDAS) Collaboration: evidence update 2.0. Med Decis Making. 2021;41:729–33.

Bonner

Trevena

Gaissmaier

, et al. Current best practice for presenting probabilities in patient decision aids: fundamental principles. Med Decis Making. 2021;41:821–33.

Hoffmann

Bakhit

Durand

Perestelo-Perez

Saunders

Brito

. Basing information on comprehensive, critically appraised, and up-to-date syntheses of the scientific evidence: an update from the International Patient Decision Aid Standards. Med Decis Making. 2021;41:755–67.

Joseph-Williams

Abhyankar

Boland

, et al. What works in implementing patient decision aids in routine clinical settings? A rapid realist review and update from the International Patient Decision Aid Standards Collaboration. Med Decis Making. 2021;41:907–37.

Martin

Brogard Andersen

O’Brien

, et al. Providing balanced information about options in patient decision aids: an update from the International Patient Decision Aid Standards. Med Decis Making. 2021;41:780–800.

Muscat

Smith

Mac

, et al. Addressing health literacy in patient decision aids: an update from the International Patient Decision Aid Standards. Med Decis Making. 2021;41:848–69.

Rahn

Jull

Boland

, et al. Guidance and/or decision coaching with patient decision aids: scoping reviews to inform the International Patient Decision Aid Standards (IPDAS). Med Decis Making. 2021;41:938–53.

Shaffer

Brodney

Gavaruzzi

, et al. Do personal stories make patient decision aids more effective? An update from the International Patient Decision Aids Standards. Med Decis Making. 2021;41:897–906.

Thompson

Paskins

Main

, et al. Addressing conflicts of interest in health and medicine: current evidence and implications for patient decision aid development. Med Decis Making. 2021;41:768–79.

10.

Trenaman

Jansen

Blumenthal-Barby

, et al. Are we improving? Update and critical appraisal of the reporting of decision process and quality measures in trials evaluating patient decision aids. Med Decis Making. 2021;41:954–9.

11.

Trevena

Bonner

Okan

, et al. Current challenges when using numbers in patient decision aids: advanced concepts. Med Decis Making. 2021;41:834–47.

12.

Witteman

Maki

Vaisson

, et al. Systematic development of patient decision aids: an update from the IPDAS Collaboration. Med Decis Making. 2021;41:736–54.

13.

Witteman

Ndjaboue

Vaisson

, et al. Clarifying values: an updated and expanded systematic review and meta-analysis. Med Decis Making. 2021;41:801–20.

14.

Yen

Smith

Engel

, et al. A systematic review and meta-analysis of patient decision aids for socially disadvantaged populations: update from the International Patient Decision Aid Standards (IPDAS). Med Decis Making. 2021;41:870–96.

15.

Qaseem

Forland

Macbeth

, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156:525–31.

16.

LeLorier

Gregoire

Benhaddad

Lapierre

Derderian

. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997;337:536–42.

17.

Ioannidis

Haidich

Pappa

, et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. JAMA. 2001;286:821–30.

18.

Sepucha

Scholl

. Measuring shared decision making: a review of constructs, measures, and opportunities for cardiovascular care. Circ Cardiovasc Qual Outcomes. 2014;7:620–6.

19.

Owens

Lohr

Atkins

, et al. AHRQ series paper 5: grading the strength of a body of evidence when comparing medical interventions—agency for healthcare research and quality and the effective health-care program. J Clin Epidemiol. 2010;63:513–23.

20.

Sawaya

Guirguis-Blake

LeFevre

Harris

Petitti

; on behalf of the U.S. Preventive Services Task Force. Update on the methods of the U.S. Preventive Services Task Force: estimating certainty and magnitude of net benefit. Ann Intern Med. 2007;147:871–5.