A Universal Framework for the Benefit-Risk Assessment of Medicines

Abstract

A universal framework for the evaluation of the benefit-risk assessment of medicines during development by pharmaceutical companies and in the regulatory review by regulatory authorities is considered of value, as it would result in the systematic structured approach to support transparency in decision making. Several organizations have developed frameworks over the past few years, including those recommended by pharmaceutical companies such as the PhRMA BRAT (Pharmaceutical Research and Manufacturers of America Benefit-Risk Action Team) and the BRAIN (Benefit-Risk Assessment in New and Old Drugs) as well as frameworks advanced by regulatory agencies, including the FDA 5-step framework and the EMA PrOACT-URL. However, a review of the criteria—including logical soundness, comprehensiveness, acceptability of results, practicality, specificity and sensitivity, presentation (visualization), and scope proposed for the development of a universal framework—demonstrated that all these different frameworks described can be incorporated into UMBRA (Universal Methodology for Benefit-Risk Assessment).

Keywords

universal methodology for benefit-risk assessment multicriteria decision making

Introduction

Currently, regulatory agencies may make different decisions despite having the same data on new medicines submitted for their assessment, leading to increased pressure to improve agency transparency and accountability and to establish appropriate document governance for decision-making processes. A universal benefit-risk assessment framework applicable to pharmaceutical companies and regulatory agencies could serve as a standardized structured model for benefit-risk assessment to support transparency in decision making.¹

A survey conducted among pharmaceutical companies and regulatory agencies demonstrated that the main hurdle to the establishment of a universal framework was the lack of an accepted, validated, and internationally relevant model.² Although harmonizing requirements that could be applied across different jurisdictions and scenarios are challenging, establishing a universal framework with the participation of major regulatory agencies may ensure its uptake by other regulators across the world.

The EMA Benefit-Risk Methodology Project was aimed at the development and testing of tools and processes for balancing multiple benefits and risks, which could be used as an aid to informed, science-based regulatory decisions about medicinal products.^3,4 This project consisted of 5 consecutive work packages. The second work package examined the applicability of 18 quantitative approaches for assessing the benefit-risk balance and 3 qualitative frameworks—namely, the framework of the PhRMA BRAT (Pharmaceutical Research and Manufacturers of America Benefit-Risk Assessment Team), the 7-step framework developed by the Centre for Innovation in Regulatory Science (CIRS), and the benefit-risk framework developed by the FDA.⁵

In the EMA’s evaluation of quantitative approaches, it was concluded that any quantitative method or approach would require a qualitative framework within which the model could be effectively developed. Combinations of approaches could prove useful in situations that required a review of the contributions, such as the magnitude of favorable effects, the seriousness of unfavorable effects, uncertainties, transitions in health states and the time spent in each state, and trade-offs between effects. Therefore, an overarching benefit-risk assessment framework with the capacity to incorporate various quantitative methods would be ideal.

The International Society for Pharmacoeconomics and Outcomes Research Risk-Benefit Management Working Group conducted a study to review and compare published quantitative benefit-risk assessment methodologies employed by regulatory agencies and/or the pharmaceutical industry to identify unique characteristics of techniques applicable to a specific drug evaluation scenario or therapeutic indication. It found that each quantitative method had unique advantages and disadvantages based on data requirements and statistical properties. Because there was a limited number of empirical applications of proposed techniques and no consensus regarding a clear gold standard, the authors recommended the use of multiple benefit-risk assessment approaches across different therapeutic indications and treatment populations.⁶ This was similar to the EMA opinion regarding the need to select the most appropriate tools available for effective benefit-risk assessment, which should be governed by an overarching framework.

In the report of methods for benefit and harm assessment in systematic reviews by the Agency for Healthcare Research and Quality, some principles for a review protocol development were highlighted. First, the key potential benefits and harms should be identified; then, the approaches used in the reporting of outcomes should be described, including the assumptions undertaken, such as numbers needed to treat and numbers needed to harm. Patients’ preferences should also be considered in the assessment and sensitivity analyses conducted to determine the impact of varying preferences.⁷

Mussen and colleagues⁸ conducted a literature review of tools for the assessment of medicines and argued that the development of any new framework ought to achieve the following objectives:

match current regulatory agency practices for benefit-risk assessment so that it could be used in the scope of those practices;

take into account the data in a marketing authorization application and the scientific data otherwise available to regulatory agencies;

not require additional analyses or reanalyses of source clinical data or additional clinical meta-analyses;

be fit for use for initial registration and postapproval reassessment of existing medicines;

be applicable to all kinds of medicines, including vaccines and nonprescription medicines;

be considered a tool for regulatory agencies and pharmaceutical companies for assessing the benefit-risk balance of medicines but not as a substitute for decision making; and

be validated.

A study was conducted to explore the current status of and need for a universal benefit-risk framework for medicines in regulatory agencies and pharmaceutical companies.² It found that most agencies and companies believed that a benefit-risk framework should be applied throughout the life cycle of the medicine with the emphasis on its applicability to product registration and to supporting health technology assessment. The general consensus was that a universal framework would enhance the quality of communication and enable the ongoing assessment of benefit-risk management plans and that it should be utilized by both agencies and companies. Advantages of a universal framework cited by survey respondents included its ability to provide documentation for structured discussion, to act as a tool for communication among peers within an organization and among stakeholders, as well as to provide the opportunity to enhance communication and document transparent and accountable decisions.

The objective of this research was to build on these experiences to develop a universal framework for the assessment of medicines that supports a systematic approach to benefit-risk decision making.

Methods

The potential components of a universal benefit-risk framework that would facilitate decision making were collated from a systematic review of published literature and reports from workshops conducted since 2007 by the CIRS. Five frameworks were reviewed and assessed against the following criteria: logical soundness, comprehensiveness, acceptability of results, practicality, specificity and sensitivity, presentation (visualization), and scope. Finally, the overarching applicability of the proposed framework—namely, the Universal Methodology for Benefit-Risk Assessment (UMBRA)—was evaluated by comparing its components with those of the assessed frameworks to determine if it encompassed the common essential elements.

Results

Requirements of a Universal Framework

The criteria identified by the EMA Benefit-Risk Methodology Project⁴ and subsequently used by Leong and colleagues in their survey of pharmaceutical companies and regulatory agencies² were employed here to assess the suitability of this universal framework (Table 1). Among these criteria, “comprehensiveness” includes the use of multiple assessment tools within an overarching framework, the appropriateness of which had been previously detailed by several investigators.^5,6

Table 1.

Criteria influencing the quality of a universal benefit-risk framework.

Criterion	Function
1. Logical soundness	Provides an approach that is sound and allows decisions that are coherent and aid rational thinking
2. Comprehensiveness	Provides an approach that handles all forms of data (including qualitative and quantitative, subjective and objective information) and allows for multiple criteria
3. Acceptability of results	Provides an approach that checks for inconsistencies in data and judgment and a realistic approach to the evaluation of benefits and risks
4. Practicality	Provides an approach with minimum burden on resources and ease of use
5. Specificity and sensitivity	Provides a statistical perspective underpinning the reliability of the decision
6. Presentation (visualization)	Provides outcomes in an easily understandable format, such as charts and plots
7. Scope	Provides a consistent approach throughout drug development and postapproval monitoring

Identification of a Suitable Framework

Five frameworks currently used for the assessment of the benefits and risks of medicines were reviewed (Table 2). Of these, 2 were used by regulatory agencies and 2 by pharmaceutical companies, and the 7-step CIRS framework had been reviewed by both these stakeholders.

Table 2.

Selected current frameworks for the assessment of the benefits and risks of medicines.

Source	CIRS	EMA	FDA	PhRMA	Novo Nordisk
Name of framework	7-step framework	8-step PrOACT-URL	5-step benefit-risk framework	6-step BRAT framework	8-step BRAIN framework
Basis of framework	MCDA	MCDA		MCDA	MCDA
Reviewed by	Regulatory agencies and pharmaceutical companies	EU regulatory agencies	US regulatory agency	Pharmaceutical companies	Pharmaceutical companies

BRAIN, Benefit-Risk Assessment in Old and New Drugs; CIRS, Centre for Innovation in Regulatory Science; MCDA, multicriteria decision analysis; PhRMA BRAT, Pharmaceutical Research and Manufacturers of America Benefit-Risk Action Team; PrOACT-URL, problem, objectives, alternatives, consequences, trade-offs, uncertainty, risk tolerance, linked decisions.

The platform on which 4 of the 5 reviewed frameworks are based—multicriteria decision analysis (MCDA)—is a process described by Dodgson and associates, which defines the individual contributing aspects of the decision-making process allowing these outcomes to be systematically organized to form the basis of the decision.⁹ The 3 key phases of the MCDA process are as follows: identifying and structuring the problem, taking the decision maker’s preferences into account, and developing an action plan. The various steps in MCDA that can be applied to the benefit-risk assessment of medicines are found in Table 3.

Table 3.

Steps in multicriteria decision analysis (MCDA).

Steps	Actions
1. Establish the decision context.	Establish aims of the MCDA; identify decision makers and other key players. Design the sociotechnical system for conducting the MCDA. Consider the context of the appraisal.
2. Identify the options to be appraised.
3. Identify objectives and criteria.	Identify criteria for assessing the consequences of each option. Organize the criteria by clustering them under high- and lower-level objectives in a hierarchy.
4. Scoring: Assess the expected performance of each option against the criteria; then, assess the value associated with the consequences of each option for each criterion.	Describe the consequences of the options. Score the options on the criteria. Check the consistency of the scores on each criterion.
5. Weighting: Assign weights for each of the criteria to reflect their relative importance to the decision.
6. Combine the weights and scores for each option to derive an overall value.	Calculate the overall weighted scores at each level in the hierarchy. Calculate the overall weighted scores.
7. Examine the results.
8. Sensitivity analysis.	Conduct a sensitivity analysis: Do other preferences or weights affect the overall ordering of the options? Look at the advantage and disadvantage of the selected options and compare pairs of options. Create possible new options that might be better than those originally considered. Repeat the above steps until a “requisite” model is obtained.

A number of factors make the MCDA approach relevant to the benefit-risk assessment of medicines. It does not require additional analyses or aim to replace decision making. Additionally, scoring, weighting, and sensitivity analyses facilitate the required acceptability, specificity, and sensitivity of results. By providing a structured flow of information leading to a decision, an MCDA-based framework is a tool for communicating a transparent and consistent decision. In support of its use in a universal benefit-risk framework, the MCDA model is applicable to all types of medicines at different life cycle decision stage gates and by multiple stakeholders, including industry and regulatory and health technology assessment agencies.

Although MCDA does not provide any form of visualization to facilitate understanding of outcomes, it can enhance the consistency, objectivity, and transparency of benefit-risk decision-making processes by providing a structured and systematic approach with appropriate documentation for tracking the process and providing greater accountability for a decision. It also facilitates the reviewing of past decisions and experiences to ensure the consistency of regulatory decisions across a therapeutic area, and it allows an understanding of the context of differing agency decisions when based on the same or substantially similar dossiers.

Accordingly, because of its MCDA basis, as well as its independent development and input from regulatory agencies and pharmaceutical companies, the CIRS 7-step framework was chosen as the basis for the further development of a universal framework.

Development of the Framework

The processes of the CIRS 7-step framework were reviewed to identify areas for enhancement. Step 1, “decision context,” is the identification and structuring of the problem, while steps 2 to 5 are the development of decision-maker preferences—that is, the benefit-risk criteria. Because step 7, “expert judgment,” provides an action plan leading to a decision, it is correlated to the final key phase of MCDA. Step 6, “visual presentation,” was added to fulfill the previously identified requirement for a universal framework.

Unlike the other frameworks, the CIRS 7-step framework had been applied to real-world use to only a limited degree. To impart universal utility and increase the likelihood of global uptake of a universal framework, a comparative review of the FDA, EMA, PhRMA, and Novo Nordisk frameworks was conducted with the goal of harmonizing the essential elements into an overarching framework.

The FDA framework (Table 4) was developed to accurately and concisely describe benefit-risk considerations to help assessors apply a structured approach in regulatory decision making.^10
–12 The context of the decision, an understanding of the condition treated, and the unmet medical need are important aspects in the use of this framework. Unlike other frameworks, it is not based on MCDA.

Table 4.

FDA approach to assessment of benefits and risks.

Consideration	Evidence and Uncertainties	Conclusions and Reasons
Analysis of condition	Summary of evidence	Conclusions (implications for decision)
Unmet medical need	Summary of evidence	Conclusions (implications for decision)
Clinical benefit	Summary of evidence	Conclusions (implications for decision)
Risk	Summary of evidence	Conclusions (implications for decision)
Risk management	Summary of evidence	Conclusions (implications for decision)

From T Mullin, presentation at June 16-17, 2011, Centre for Innovation in Regulatory Science workshop.⁹ Reprinted with the permission of the Centre for Innovation in Regulatory Science.

According to its set of guiding principles, the EMA initiates decision making for medicines by examining the decision to be made and the objectives of that decision and then considers the options, alternatives, and trade-offs before a decision or action would be decided.^13,14 Based on these principles and a generic framework for decision making developed by Hammond and colleagues,¹⁵ the EMA PrOACT-URL approach (problem, objectives, alternatives, consequences, trade-offs, uncertainty, risk tolerance, linked decisions) prompts the assessor to address the considerations required to develop the benefit-risk profile of a medicine (Table 5).

Table 5.

EMA 8-Step PrOACT-URL.

Steps	Actions
1. Problem	Determine the nature of the problem and its context. Frame the problem.
2. Objectives	Establish objectives that indicate the overall purposes to be achieved. Identify criteria of favorable and unfavorable effects.
3. Alternatives	Identify the options to be evaluated against the criteria.
4. Consequences	Describe how the alternative performs for each of the criteria—that is, the magnitudes of all effects and their desirability or severity and the incidence of all effects.
5. Trade-offs	Assess the balance between favorable and unfavorable effects.
6. Uncertainty	Assess the uncertainty associated with the favorable and unfavorable effects. Consider how the balance between favorable and unfavorable effects is affected by uncertainty.
7. Risk tolerance	Judge the relative importance of the decision makers’ risk attitude for this product and indicate how this affected the balance reported in step 5.
8. Linked decisions	Consider the consistency of this decision with similar past decisions, and assess whether taking this decision could affect future decisions.

PrOACT-URL, problem, objectives, alternatives, consequences, trade-offs, uncertainty, risk tolerance, linked decisions.

The BRAT, under the auspices of the PhRMA, developed a framework for benefit-risk assessment.^16
–18 The BRAT framework (Table 6) consists of 6 steps that produce representations of key trade-offs, with appropriate documentation of the rationale for decisions and the assumptions made in their development.

Table 6.

PhRMA BRAT framework.

Steps	Actions
1. Define the decision context	Define drug, dose, formulation, indication, patient population, comparator(s), time horizon for outcomes, perspective of the decision makers (regulator, sponsor, patient, or physician).
2. Identify outcomes	Select all important outcomes and create the initial value tree. Define a preliminary set of outcomes measures/endpoints for each outcome. Document rationale for outcomes included/excluded.
3. Identify and extract source data	Determine and document all data sources (eg, clinical trials, observational studies). Extract all relevant data for the data source table, including detailed references and any annotations, to help the subsequent interpretations create summary measures.
4. Customize the framework	Modify the value tree on the basis of further review of the data and clinical expertise. Refine the outcomes measures/endpoints. May include tuning of outcomes not considered relevant to a particular benefit-risk assessment or that vary in relevance by stakeholder.
5. Assess outcome importance	Apply or assess any ranking or weighting of outcome importance to decision makers or other stakeholders.
6. Display and interpret key benefit-risk metrics	Summarize source data in tabular and graphic displays to aid review and interpretation. Challenge summary metrics, review source data, and identify and fill any information gaps. Interpret summary information.

PhRMA BRAT, Pharmaceutical Research and Manufacturers of America Benefit-Risk Action Team.

BRAIN (Benefit-Risk Assessment in New and Old Drugs) is an 8-step interactive process that was developed by the pharmaceutical company Novo Nordisk A/S that can extract information from clinical trials that is otherwise not captured by direct statistical analyses.¹⁹ With the use of this tool in profiling the decision context, the aims, goals, expectations, and relevant information to support the benefit-risk assessment are identified.

The steps of these frameworks were tabulated and common process elements identified. It was found that at a higher level of categorization of the tasks involved, four common core elements were identified: framing the decision, identifying benefits and risks, assessing the benefits and risks, and interpretation and outcomes (Table 7). It was then possible to construct an overarching 8-step framework that incorporated all the elements included in each framework.

Table 7.

Comparisons of existing benefit-risk assessment frameworks by core elements.

Framing the Decision	Identifying Benefits and Risks	Assessing Benefits and Risks	Interpretation and Outcome
FDA
Analysis of conditions and unmet medical needs	Clinical benefits, risks	Evidence for benefits and risks	Evaluating uncertainty Conclusions and reasons, risk management plans
EMA PrOACT-URL
Nature and framing of the problem	Objectives, favorable and unfavorable effects	Alternatives regarding options to be evaluated and consequences Trade-offs and benefit-risk balance	Evaluating uncertainty Effects table and risk tolerance Consistency of decisions (linked decisions)
PhRMA BRAT
Define decision context	Identify outcomes, extract source data: build value tree Customize framework: refine value tree	Assess relative importance of different outcomes: weighting or ranking, other stakeholders	Evaluating uncertainty Display and interpret key benefit-risk metrics and validate results Decision and communication of benefit-risk assessment
Novo Nordisk BRAIN
Decision context	Disease profile Weighting	Scoring Evidence evaluation / weighted score	Evaluating uncertainty Presentation Overall conclusion
CIRS 7-step framework
Decision context	Building the value tree for all benefits and risks Rationale for which benefits and risks to be included for benefit-risk assessment	Weighting of benefits and risks Valuing or scoring of options	Visualization Expert judgment and risk management
Universal benefit-risk framework
Step 1: Decision context	Step 2: Building the value tree Step 3: Customizing the value tree	Step 4: Weighting of benefits and risks Step 5: Scoring the options	Step 6: Evaluating uncertainties Step 7: Concise presentation of results (visualization) Step 8: Expert judgment

BRAIN, Benefit-Risk Assessment in Old and New Drugs; CIRS, Centre for Innovation in Regulatory Science; PhRMA BRAT, Pharmaceutical Research and Manufacturers of America Benefit-Risk Action Team; PrOACT-URL, problem, objectives, alternatives, consequences, trade-offs, uncertainty, risk tolerance, linked decisions.

In a bid to promote the universal utility of the proposed framework, the CIRS 7-step framework was amended to reflect the core elements and provide a unified standardized framework that would meet the requirements of the FDA, EMA, PhRMA BRAT, and Novo Nordisk models, which had been used by the 2 major regulatory agencies and pharmaceutical companies. The final universal benefit-risk framework (Figure 1) consists of 8 steps. The processes were essentially unchanged, but step 6 (“evaluating uncertainty”; previously conducted as part of the overall process) was added as a specific step to ensure that the 3 key components of benefit-risk-harm evaluation are explicitly considered. This consolidated approach, termed the UMBRA framework, was subsequently endorsed by a group of regulatory agency and pharmaceutical industry leaders.²⁰

Figure 1.

The universal benefit-risk framework for the assessment of the benefits and risks of medicines.

The details of what should be considered in each step of the UMBRA 8-step framework are shown in Table 8.

Table 8.

UMBRA steps.

Step 1: Decision context	Define drug, dose, formulation, proposed indication, approved indication, patient population, comparator(s), time horizon for outcomes and perspective of the decision makers (regulator, sponsor, patient, physician, or payer), and other critical safety wording in the proposed product label, whether it meets an unmet medical need.
Step 2: Building the value tree	Identify and select all relevant outcomes and create an initial value tree (alternatively, list all benefits and risks/harms)
Step 3: Refining the value tree	Define a preliminary set of measures for each outcome included in the value tree. Document the rationale for the outcomes included in or excluded from the initial value tree. Modify the value tree based on further review of the data and clinical expertise.
Step 4: Relative importance of benefits and risks	If applicable, apply or assess ranking or weighting (relative importance) of the benefits and risks/harms to individual outcomes, according to the perspective of the decision makers or other stakeholders.
Step 5: Evaluating/scoring the options	Score/assess the performance/outcome of the benefits and risks/harms against options, placebo drug under investigation and/or comparator. Provide either qualitative or quantitative values for each benefit and risk/harms.
Step 6: Evaluating uncertainty	Conduct sensitivity analyses to assess the impact of uncertainty in data sources/evidence package. While uncertainties cannot be ruled out, the objective of this step is to make the analysis more transparent and increase confidence in the benefit-risk assessment process and the results. Describe the variability both within and between studies: uncertainty regarding whether or not the available information predicts the true relationship between the study drug and the range of benefit-risk outcomes. Consider how the balance between benefits and harms is affected by uncertainty.
Step 7: Concise presentation of results—visualization	Summarize data in tabular (ie, key benefit-risk summary tables) and, if possible, by graphical displays (eg, forest plot, waterfall diagram) to aid review and interpretation. Where possible, the quantitative measures of the criteria should be explicitly provided.
Step 8: Expert judgment and communication	Expert judgment provides the overall outcome/conclusion and whether the benefit-risk balance is positive or negative. Reference should be made to the pharmacovigilance and risk minimization plans. Describe any outstanding issues and how they will be addressed and need, if any, for further studies. Finally, a clear conclusion and recommendation should be provided. This judgment can be combined with the objective outcome measures to communicate the benefit-risk of the decision maker in a transparent way.

UMBRA, Universal Methodology for Benefit-Risk Assessment.

Discussion

Currently, there is a need for a systematic, standardized, structured approach to the benefit-risk assessment of new medicines that is broadly accepted and utilized by developers and regulators.¹⁸ In this study, the universal framework for the assessment of the benefits and risks of medicines was developed with input from regulators and pharmaceutical companies and the review of existing frameworks used by both these stakeholders. In addition, essential components of each framework were preserved through standardization to facilitate its global uptake and ensure its international relevance.

Additionally, while the incorporation of all stakeholder perspectives in benefit-risk assessment is ideal, there remain some challenges in collecting and integrating patient and health care professional perspectives into these frameworks. Furthermore, the UMBRA framework is not able to discern the intrinsic significance of the values provided and require that the regulators be able to apply clinical judgment at the conclusion of the assessment, given that such judgment is an accumulation of education and experience. Importantly, the UMBRA framework for the assessment of benefits and risks has yet to incorporate a quantitative methodology in the decision model.²¹ The framework allows for relative importance and values to be assigned to benefit and risk parameters, which can provide a more transparent perspective for decision making.²² However, the concept behind relative importance is not currently embraced by all stakeholders.²³

While we have observed a common alignment among the various frameworks, there is currently no consensus with regard to the specific methodologies that are considered appropriate or commonly applied in the evaluation of the benefit-risk assessment of medicines. However, this is not an issue, as long as they align to a universally accepted framework that accommodates the various existing methodologies used by companies and agencies. The importance of this is to enable other stakeholders, such as patients, physicians, and regulatory agencies, to be able to identify the key elements used to reach a benefit-risk decision and to assess the uncertainties surrounding the decision based on a common language. However, more research is required to further the understanding and application of weightings (relative importance) and identification and consensus of assessment methodologies.

Finally, the acceptance of a universal framework for companies and agencies requires further evaluation and implementation in practical situations. It is important that this work be continued to ensure the relevance and currency of the concept and tools as well their ability to meet the expectations of stakeholders. This will require the continuous and ongoing involvement of various stakeholders concerned with enhancing the science of decision making with respect to the benefit-risk assessment of medicines so that it may ultimately facilitate a universal system that provides improved patients’ access to medicines.

Conclusion

A universal framework should include the following characteristics: logical soundness, comprehensiveness, acceptability of results, practicality, specificity and sensitivity, presentation (visualization), and scope. This study demonstrated that the 5 key frameworks used by the pharmaceutical industry and regulatory agencies could be consolidated into a single framework—namely, the UMBRA. This represents the way forward by providing a structured, systematic approach for consistent, transparent, quality decision making. Testing the practical application and validity of the UMBRA framework through a prospective study will help to determine its place in global medicines development and assessment.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Center for Innovation in Regulatory Science. Measuring benefit and balancing risk: strategies for the benefit-risk assessment of new medicines in a risk-averse environment. http://www.cirsci.org/sites/default/files/June_2008.pdf. Published 2008. Accessed November 20, 2013.

Leong

McAuslane

Walker

Salek

. Is there a need for a universal benefit-risk assessment framework for medicines? Regulatory and industry perspectives. Pharmacoepidemiol Drug Saf. 2013;22:1004–1012.

European Medicines Agency. Reflection paper on benefit-risk assessment methods in the context of the evaluation of marketing authorization applications of medicinal products for human use. http://www.ema.europa.eu/docs/en_GB/document_library/Regulatory_and_procedural_guideline/2010/01/WC500069634.pdf. Published 2008. Accessed November 20, 2013.

European Medicines Agency. Benefit-risk methodology project: development and testing of tools and processes for balancing multiple benefits and risks as an aid to informed regulatory decisions about medicinal products. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2011/07/WC500109477.pdf. Published 2009. Accessed November 20, 2013.

European Medicines Agency. Benefit-risk methodology project: work package 2 report. Applicability of current tools and processes for regulatory benefit-risk assessment. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2010/10/WC500097750.pdf. Published 2010. Accessed November 20, 2013.

Guo

Pandey

Doyle

Bian

Lis

Raisch

. A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy: report of the ISPOR Risk-Benefit Management Working Group. Value Health. 2010;13:657–666.

Boyd

Singh

Varadhan

. Methods for Benefit and Harm Assessment in Systematic Reviews: Methods Research Report. Rockville, MD: Agency for Healthcare Research and Quality; 2012. AHRQ publication 12(13)-EHC150-EF.

Mussen

Salek

Walker

. Benefit-Risk Appraisal of Medicines: A Systematic Approach to Decision Making. Surrey, England: Wiley; 2009.

Dodgson

Spackman

Pearman

Phillips

. Multi-criteria Analysis: A Manual. London, England: Department for Communities and Local Government; 2009.

10.

Centre for Innovation in Regulatory Science. Visualising benefit-risk: the key to developing a framework that informs stakeholder perspective and clarity of decision making. http://www.cirsci.org/sites/default/files/CIRS_June_2011_Workshop.pdf. Published 2011. Accessed November 23, 2013.

11.

Fischhoff

. Good decision making requires good communication. Drug Saf. 2012;35:983–993.

12.

US Food and Drug Administration. PDUFA re-authorisation performance goals and procedures fiscal years 2013 through 2017. http://www.fda.gov/downloads/ForIndustry/UserFees/PrescriptionDrugUserFee/UCM270412.pdf. Published 2012. Accessed November 23, 2013.

13.

European Medicines Agency. Benefit-risk methodology project: work package 3 report. Field tests. http://www.emea.europa.eu/docs/en_GB/document_library/Report/2011/09/WC500112088.pdf. Published 2011. Accessed November 23, 2013.

14.

European Medicines Agency. Benefit-risk methodology project: work package 4 report. Benefit-risk tools and processes. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2012/03/WC500123819.pdf. Published 2012. Accessed November 23, 2013.

15.

Hammond

Keeney

Raiffa

. Smart Choices: A Practical Guide to Making Better Decisions. Boston, MA: Harvard University Press; 1999.

16.

Coplan

Noel

Levitan

Ferguson

Mussen

. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clin Pharmacol Ther. 2011;89:312–315.

17.

Levitan

Andrews

Gilsenan

. Application of the BRAT framework to case studies: observations and insights. Clin Pharmacol Ther. 2011;89:217–224.

18.

Noel

Hermann

Levitan

Watson

Van Goor

. Application of the Benefit-Risk Action Team (BRAT) framework in pharmaceutical R&D: results from a pilot program. Drug Info J. 2012;46:736–743.

19.

Centre for Innovation in Regulatory Science. Refining the benefit-risk framework for the assessment of medicines: valuing and weighting benefit and risk parameters. http://cirsci.org/sites/default/files/CIRS_June2010_Workshop_Report.pdf. Published 2010. Accessed November 23, 2013.

20.

Centre for Innovation in Regulatory Science. Building the benefit-risk toolbox: are there enough common elements across the different methodologies to enable a consensus on a scientifically acceptable framework for making benefit-risk decisions? http://www.cirsci.org/sites/default/files/CIRS%20June%202012%20Workshop%20Report%20low-res.pdf. Published 2012. Accessed November 23, 2013.

21.

Philips

Fasolo

Zafiropoulos

Beyer

. Is quantitative benefit-risk modeling of drugs desirable or possible? Drug Discov Today Technol. 2011;8:e3–e10.

22.

Walker

Liberti

McAuslane

. Refining the benefit-risk framework for the assessment of medicines: valuing and weighting benefit and risk parameters. Clin Pharmacol Ther. 2011;89:179–182.

23.

Mt-Isa

Tzoulaki

Callreus

Micaleff

Ashby

. Weighing benefit-risk of medicines: concepts and approaches. Drug Discov Today Technol. 2011;8:e29–e35.