Measuring the Impact of Patient Engagement and Patient Centricity in Clinical Research and Development

Abstract

Background:

Recently, drug development companies have sought out patient feedback to improve overall drug development. However, characterization of the overall impact and return on engaging with patients have not been determined.

Methods:

The Drug Information Association (DIA), the Tufts Center for the Study of Drug Development (Tufts CSDD), and 17 other stakeholder organizations collaborated on a study to (1) quantify and define patient-centric initiatives (PCIs) utilized in clinical research and development and (2) to define evidence-based metrics and performance indicators that demonstrate return on engagement (ROE) of specific PCIs. We conducted a literature review, industry surveys, and in-depth interviews to determine and measure the impact of adopted PCIs.

Results:

We identified and defined 30 PCIs used to engage with patients. We analyzed 121 case studies and created a comprehensive list of metrics assessing overall return to the organization and to patients. Advocacy Group Support and Involvement, Conducting Patient Advisory Panels, and Focus Groups were examples of PCIs with the lowest cost and largest impact with respect to quality, speed, and impact on the patient relative to other PCIs.

Conclusion:

The results from the literature review and use cases provide drug development teams with evidence and insights to help facilitate the adoption of specific PCIs within their organization and to help select those initiatives that would provide the highest impact to patients and development organizations. It is also hoped that the biopharmaceutical industry will apply the standardized metrics in the toolkit to systematically assess the overall return on engagement.

Keywords

patient engagement patient centricity patient-centric initiatives clinical trial design return on investment return on engagement

Introduction

Over the last 3 decades, there has been a proliferation of patient-centric initiatives (PCIs) across many different stakeholder groups in the health care product development and delivery continuum to better collaborate with patients.^{1

–13} Within drug development specifically, a patient-centric approach represents a major shift and a fundamental and profound change to a paternalistic 50+-year-old product-centric research and development (R&D) paradigm that is now plagued with delays, inefficiencies, and high failure rates.¹⁴ The shift toward patient-centric drug development has been encouraged and amplified by regulatory agency activities and a number of public, private, and academic initiatives.^11
–13

Regulatory agencies are increasingly looking toward feedback from patients to inform decision making. The Food and Drug Administration’s Patient-Focused Drug Development Initiative and resulting “Voice of the Patient” reports have signaled their aims to more systematically obtain the patient perspective on specific diseases and their treatments in order to help inform the Agency’s decisions and oversight during both drug development and review of a marketing application.^15,16

Public, private, and academic initiatives, though varied in focus and approach, are based on the common underlying principles of meaningful engagement of patients at all appropriate points within the medical product life cycle in order to ensure the best possible alignment between patient needs and products and services that improve health outcomes for patients. Frameworks such as that developed by the Parkinson’s Disease Foundation and adapted by the Clinical Trials Transformation Initiative (CTTI) and the Medical Device Innovation Consortium’s Catalog of Methods for Assessing Patient Preferences for Benefits and Harms of Medical Technologies, have helped to clearly define when to solicit the patient voice to influence drug development outcomes and implementation.^17
–19 Other examples from public, private, and academics include frameworks, rubrics, and strategies as outlined by groups such as the Patient-Centered Outcomes Research Institute (PCORI), the European Patients Academy for Therapeutic Innovation (EUPATI), a project of the EU Innovative Medicines Initiative (IMI), and the National Institute for Health Research (NIHR), a component of the UK National Health System.^20

–23

Frameworks such as these are valuable in helping to establish meaningful patient involvement in drug development, but they are conceptual. With the exception of the MDIC catalog of methods, they stop short of providing practical details about how initiatives within the frameworks can be implemented to best facilitate patient involvement that positively impacts study volunteer and stakeholder experience. Additionally, these frameworks do not provide details on how to adopt and incorporate PCIs into an organization. Operational approaches impact the effectiveness and sustainability of patient-centric practices. At this time there are limited data describing how organizations structure themselves to effectively use resources, both human and financial, to implement PCIs.

Lastly, while the proliferation of frameworks from public, private, and academic organizations signals growing industry-wide receptivity to patient-centric approaches, there are limited data demonstrating current levels of and barriers to adoption and the quantified impact of PCIs on drug development performance. Some PCIs may deliver sufficient and consistent impact to affirm their becoming standard practice in drug development, but others may prove costly with limited to no measurable benefit. Metrics, especially quantitative measures of the relative value of PCIs in specific applications, are needed to inform decision making around the PCIs that will have the most positive impact on outcomes for the patient and for the drug development process.

To address the limited data surrounding patient engagement, we conducted a study with 4 primary objectives:

To assess the current and future adoption rate of PCIs by biopharmaceutical organizations, that is, to assess the prevalence of PCIs;³

To identify organizational structures and mechanisms for managing and continuously improving PCIs;

To assess the availability of regulatory policy documents and guidelines for PCIs: what is available, what is coming, and what may be missing;²⁴

To define, collect, and measure the impact of these PCIs on drug development and patients via biopharmaceutical case studies and real examples of PCI use.

This paper focuses on the fourth objective. Results of our assessments addressing objectives 1 and 3 have been published elsewhere.^3,24 With regard to objective 2, insights on types of organizational structures and mechanisms for managing and continuously improving patient engagement will be addressed in a subsequent manuscript.

Methods

Cross-Industry Collaboration (Working Group) Members and Tasks

We created a cross-industry working group to assess patient engagement approaches. We identified 58 biopharmaceutical organizations, contract research organizations (CROs), and contract service organizations (CSOs) that were known to either research patient engagement or have an interest in patient-centered medical product development and invited them to participate in the study. Seventeen of the 58 accepted. These organizations were AstraZeneca, Bayer, Biogen, EMD Serono, HealthiVibe LLC, Horizon Pharma, ICON plc, INC Research, Janssen Pharmaceuticals, Merck, MK&A, Novartis Pharmaceuticals, Pfizer, Purdue Pharma, Quintiles IMS (now IQVIA), and UCB. The collaboration commenced on October 16, 2015, and was completed on October 31, 2016.

The working group was tasked with

deriving and defining a complete list of PCIs that have been adopted, piloted, considered, or not considered at their organizations;

cocreating an industry-wide survey assessing the importance and ease of collecting metrics measuring impact;

participating in the industry-wide survey;

codesigning a toolkit containing essential impact measures; and

providing case studies demonstrating the impact of PCIs used in clinical development.

Measuring Current Impact

We collected qualitative and quantitative impact measures via literature review, in-depth interviews with working group members, in-depth interviews with patients who participated in clinical trials, and in-depth interviews with individuals at organizations identified in the literature as having had significant experience with patient engagement in clinical research. The term case studies will be used to refer to the combination of information gathered from all of these sources. No research instrument was created or utilized to measure overall impact as case studies provided different metrics.

Seventeen total interviews with organization representatives were conducted, and 10 total interviews with patients were conducted. The Tufts University Institutional Review Board approved the in-depth interview guide and methodology for patient outreach. The Center for the Information and Study on Clinical Research Participation (CISCRP) reached out to patients on our behalf via their mailing list. Patient participants were offered a $50 amazon gift card for participating in the study. All manuscripts and articles identified during the literature review are cited in the Results section.

We designed a qualitative rubric summarizing data from case studies, as metrics captured in the case studies were not standard across cases and could not be averaged to report a single number. We began activity on January 7, 2016, and completed all analyses by September 7, 2016 (8 months total).

The qualitative rubric contains 3 domains:

The cost to conduct the specific PCI

The ease of conducting the PCI within the organization; for example, quick institutional review board (IRB) approval

The reported impact, which assesses changes in quality, speed, and impact on the patient.

Ratings are relative to each other. Citations from the literature review are provided in the results section. Citations for in-depth interviews are not provided as interviews are deidentified.

Identification and Classification of Impact Measures

Impact measures, that is, metrics measuring the success of PCIs, were assessed through literature review,^{4,10,25

–31} the data identified from the case studies when measuring current impact, and feedback from working group members. Nine total manuscripts were identified in the literature review. We used the following definitions to describe impact measures:

Impact: synonymous with return on engagement (ROE). Four components to measure: development speed, quality, and cost, and impact on study volunteer

Key performance indicators and metrics (KPI/Ms): measurable value that demonstrates how effectively companies are achieving patient-centric objectives

Patient advocacy groups (PAGs): synonymous with patient communities

Return on investment (ROI): financial and operational impact company has been realizing from utilizing PCIs

Return on engagement (ROE): ROI + impact on patient (ie, study volunteer [eg, increasing patient ease])

Impact measures were classified using the frameworks described by Getz,⁴ and Ashkenazy and Schneider.¹⁰ We also asked patients what they thought industry should collect and shared with them our framework to gather patient feedback.

PCI Definitions

The working group identified and defined 23 PCIs that organizations are using to engage patients.³ Seven more PCIs were identified through literature review. All PCIs were agreed by the working group as potentially yielding value to the company and to patients.

The 30 PCIs were classified into 4 categories: PCIs addressing protocol design (4 total), PCIs relying on technology (12 total), PCIs involving partnerships with patients and/or patient advocacy groups (7 total), and PCIs focusing on increasing ease of clinical trial participation (7 total). PCI classification is described in Table 1 and in Lamberti et al.³ Industry adoption rates (ie, the number of companies using the specific PCIs) are also provided in Lamberti et al for the 23 PCIs that were defined by organizations.³

Table 1.

Patient-Centric Initiatives Adopted, Piloted, Considered, or Not Considered in Biopharmaceutical Organizations.^a

Patient-Centric Initiative	Classification	Definition (Citation, If Not Defined by Working Group Members)	Adoption Rates provided in Lamberti et al³
Adaptive trial designs and adaptive licensing	Protocol design	Adaptive design: Preplanned adaptations—generated using trial simulations and scenario planning—of one or more specified clinical trial design elements that are modified and adjusted while the trial is underway based on analysis of interim data.¹¹⁹ Adaptive licensing: prospectively planned, adaptive approach to bringing drugs to market¹²⁰	Yes
Centralized/integrated clinical and electronic health records data	Technology	Utilization of electronic health records as a primary source of data for clinical research to enhance patient care and access to trials and to facilitate clinical research¹²¹	Yes
Community conversations	Partnerships	Joint physician and patient advisory boards	Yes
Digital medicine	Technology	Medicine that can be tracked using sensor-enabled technology¹²²	No
Direct-to-patient clinical trials	Technology	Allows drug therapies to be delivered and administered in the patient’s home and/or biological samples to be taken and uplifted from the patient’s home. Depending on the complexity of the trial protocols, patients may take medicines themselves, or rely on a home care nurse or caregiver for clinical support.¹²³	Yes
Electronic consent (eConsent)	Technology	Mobile-based informed consent method to deliver text, voice, and/or multimedia content¹²⁴	Yes
End of study surveys	Patient ease	End of study surveys assessing patient experience in the clinical trials	Yes
Gaming eConsent	Technology	Utilizing game mechanics (ie, constructs of rules or methods or rules that govern how a game works) to obtain clinical trial participants’ informed consent electronically^124,125	No
Gaming PRO	Technology	Utilizing game mechanics (ie, constructs of rules or methods or rules that govern how a game works) to improve patient reported outcome response rates¹²⁵	No
Home nursing networks	Patient ease	ie Ambulant care services; deliver clinical trial services in the home or alternate-site setting¹⁶	Yes
Human factor testing/simulation	Technology	Provide patients with a product or device before launching a trial to get feedback on study design	Yes
Innovative patient data analytics/collection	Technology	Identify drivers of disease for various subsets of people as a precursor to clinical trials	Yes
Lay-summary clinical trial results and/or risk management report	Patient ease	Clinical trial result summaries and reports written in nontechnical language and distributed to trial participants and other trial stakeholders	Yes
Logistics assistance	Patient ease	That is, global patient management organization; services such as individualized travel and housing logistics, payment/reimbursement systems and ongoing patient, family, and caregiver support	No
Medicine codevelopment partnerships with patient associations	Partnerships	The sharing of resources/funding/risk with patient advocacy organizations to develop new treatments	Yes
Open design and crowdsourcing	Protocol design	Method of gathering input from patients, physicians, and other stakeholders via the World Wide Web³	Yes
Patient advisory boards and focus groups	Partnerships	Patient-comprised panels dedicated to gathering direct input from patients on clinical trial plans and practices¹²⁶	Yes
Patient apps for clinical data collection	Technology	Use of electronic diary on a mobile platform to remotely record experience: eg, reminder of next dosage due, next appointment, emergency contacts	Yes
Patient community post trials	Patient ease	Inviting patients and patient advocates to participate in industry meetings to identify important outcome measures¹²⁷	Yes
Patient counseling and education	Patient ease	Comprehensive patient education programs to provide patients with scientific expertise about the disease and tools to address psychosocial challenges⁴⁵	No
Patient group support and involvement	Partnerships	Fostering relationships with and providing resources for disease-specific nonprofit advocacy groups that support patient communities¹²⁸	No
Patient involvement in study feasibility and study design	Protocol design	Engage with patient advocates and patients to help inform the effective design of clinical trial protocol, endpoints, and testing	Yes
Patient organization landscape analysis tool (disease area–specific)	Partnerships	To understand patients and patient organizations to target for a specific disease area	Yes
Patient trial community during trials	Patient ease	Inviting patients and patient advocates to meet and speak during a clinical trial¹²⁷	Yes
Patient wearable devices	Technology	Technology worn on the human body that captures clinical data; includes ePRO	Yes
Professional panels	Technology	Groups composed of individuals with relevant expertise to provide input on clinical research questions	Yes
Protocol feasibility review committees	Technology	Committees formed to assess clinical trial designs for ability to answer the research questions at hand and to realistically align with the needs of all stakeholders, including patients¹²⁹	Yes
Real world, practice-based clinical trials	Protocol design	ie, Pragmatic clinical trial; large RCTs designed to admit variations more representative of real-life conditions of care than traditional “explanatory” randomized control trials (RCTs) in which patient population characteristics, care setting, care administration, and follow-up are tightly controlled¹³⁰	Yes
Social media/online engagement	Technology	Social listening: aggregating social media data such as patient posts or comments to gather information on patient insights or to identify unmet needs during treatment³	No
Telemedicine	Technology	Use of electronic communication and remote technologies to treat patients	Yes

Abbreviation: PRO, patient-reported outcome; RCTs, randomized controlled trials.

^a Adoption rates, that is, the number of companies using the specific PCI, are provided in Lamberti et al.³

Results

Measuring Current Impact

We analyzed 121 case studies^{30,32

–118} containing qualitative and quantitative impact measures. Thirty-three of the 121 cases were gathered from in-depth interviews conducted in 2017. Case studies from the literature spanned studies from 1995 to 2017. From the 121 case studies, we found

15 cases with cost metrics,

118 cases with quantitative and qualitative impact measures (508 metrics total), and

58 cases with lessons learned and challenges (158 metrics total).

Of the 121 case studies identified, 35% provided data on patient group support and involvement, while 17% provided data on patient apps for clinical data collection. The remaining 48% provided data on 11 of the identified 30 PCIs described in Table 1. Sixty-two percent of all case studies were provided from industry, while 21% of case studies were provided by patient advocacy groups and 17% from academia. Figure 2 provides a breakdown of case studies by PCI and organization type.

Of the 666 metrics identified from the 121 case studies, 39% were quantitative in nature, while 34% were qualitative impact measures. The remaining 27% contained qualitative metrics on lessons learned and challenges. Figure 3 provides a breakdown of all metrics by PCI and metric type.

From the 666 metrics identified, we assessed the relative cost to conduct, ease of conducting, and reported impact of each of the 12 PCIs for which metrics were reported. We designed a qualitative rubric summarizing data from case studies, as metrics captured in the case studies were not standard across cases and could not be averaged to report a single number. Table 3 contains the qualitative rubric. The qualitative summaries were used to provide indicators of relative return on engagement among the 12 PCIs.

Within each PCI, there was a range of associated cost and ease of conducting the initiative. We found that the cost or ease of implementing a PCI did not always correlate with its impact. Some notable use cases include the following:

Patient group support and involvement showed a relatively high level of reported impact at a relatively low cost of time and effort. One organization noted that working with patient groups had saved the company millions of dollars that otherwise would have been spent on patient recruitment and retention. Other case studies have noted improvements in clinical trial planning and patient enrollment cycle times, regulatory agency and IRB approval cycle times, and fewer protocol amendments.

Patient counseling and education also showed a relatively high level of impact at a relatively low cost of time, effort and funds. Not only did patients report improvements in self-efficacy, but clinical data for patients undergoing health education also suggested better disease management for those patients undergoing health education programs.¹³¹

Social media/online engagement also showed a relatively high level of impact at a relatively low cost of time, effort, and funds. One organization noted that 50% of patients in the trial were recruited via social media; the cost of recruiting online is approximately $4000 per randomized patient as compared to approximately $8000 per randomized patient using traditional methods; and there was no extra time needed for IRB review and approval.

Open design and crowdsourcing clinical trial protocols showed a relatively low funding cost; however, the reported impact was not as high as other PCIs. Organizations noted more time required for planning and internal organization approval. One case study noted that crowdsourcing a clinical trial protocol for 6 weeks led to feedback from 102 patients and physicians, which resulted in 4 major and 5 minor protocol changes, all of which could have led to protocol amendments.³⁶

Patient apps for clinical data collection showed “some impact” relative to other PCIs. Furthermore, their development can be relatively expensive, with the cost varying by sophistication of the app and wearable device. A barebones app can cost about $30,000 to develop, plus $100 to $250 per wearable device, and positive patient feedback varies with the type of wearable device. The cost and complexity of conduct can thus have a dampening effect on the ROE of the medium impact PCI.

At present, digital medicine and gaming patient-reported outcomes (PROs) were viewed as costly with higher difficulty to conduct, resulting in a lower expected return on engagement. As costs come down and implementation experience accumulates, the ROE balance may shift, encouraging greater use of these PCIs.

Identification and Classification of Impact Measures

Based on the literature review and feedback from working group members, we derived a toolkit that industry and patient advocacy groups can use to assess the impact of patient engagement and PCIs. The toolkit identifies the following dimensions of medical product development:

Patient feedback: These metrics gauge whether patients in trials containing at least 1 PCI had a positive or negative experience; whether the PCI made a measurable difference to their experience; and how any patient feedback was incorporated to the protocol or program; that is, how has the patient been impacted by the PCI?

Long-term drug development portfolio and enterprise: These metrics assess the economics and impact around the PCI and its effect on the overall drug program; portfolio; that is, has the PCI affected business decisions and strategy at the corporate level?

Prevalence (ie, internal and external reach): Metrics that quantify the number of internal and external stakeholders involved in the PCI within a particular timeframe; for example, How many clinical trials have used a specific PCI? and How many patients were involved in the PCI?

Overall clinical trial performance: These metrics include traditional trial performance metrics so that trials with at least 1 PCI can be compared to similar trials with no PCIs; for example, do patient retention rates differ in trials with an implemented PCI versus similar trials that do not contain a PCI.

Metrics in the toolkit are stratified into the following 4 types of measures:

Measures assessing impact to the patient

Measures assessing cost

Measures assessing speed; both to implement and to decision making

Measures assessing quality

Table 2 lists all impact measures classified by dimension and measure type.

Table 2.

Impact Measures Industry Should Collect to Assess the Return on Patient Engagement.^a

Dimension	Measure Type	Metric (and Submetrics)
Long-term drug development Portfolio and enterprise (program and corporate level)	Impact to patient	Regulatory activity involving patients
	Cost measure	Cost of the PCI Overall cost of all PCIs Percentage of overall trial costs Cost per evaluable patient Cost per submission program
	Speed measure	Changes to development timelines; ie, time to go/no-go decision (compared to other programs)
	Quality measure	Program success rates relative to portfolio benchmark
Prevalence (ie, internal and external reach)	Impact to patient	Number of patients and/or patient advocacy groups involved in PCI Number of patient ambassadors Number of clinical trial alumni
Prevalence (ie, internal and external reach)	Cost measure	Total number of PCIs implemented (implemented; piloted; planned; not considering) Number of trials using PCI (planned vs completed) Overall Percent of total trials
Patient feedback	Impact to patient	Insignia Health’s Patient activation measure¹³⁰ (measure of self-efficacy) Patient-reported outcomes Quality of life measures Patient feedback to clinical trial sites; to companies Number of social mentions Social media sentiment Direct feedback
	Cost measure	Types and number of missed assessments (in clinical trial) Total number of changes proposed by patients to: Drug program (eg, program positioning) Clinical trial protocol (eg, endpoint selection) Communication
	Quality measure	Patient feedback to regulatory agencies Proportion of positive responses and feedback over total feedback provided Patient receptivity to protocol Description of how changes proposed by patients impacted overall program and/or study design
Trial performance metrics (typically captured for all clinical trials)	Impact to patient	Number of endpoints relevant to patients and patient advocacy groups Changes in protocol complexity and patient burden
	Cost measure	Cost of patient recruitment activity Cost of implementing PCI; eg, technology costs
	Speed measure	Clinical trial duration metrics Time from site activation to first patient in Time from first patient in to last patient in Time from last patient in to last patient out Time from institutional review board (IRB) submission to IRB approval Time to plan and implement PCI Whether clinical trial went into rescue (for recruitment)
	Quality measure	Number of protocol amendments and changes from protocol amendment IRB approval of PCI Recruitment rates; screen fail rates; randomization rates (for trial and changes to rates if PCI implemented midway through trial) Retention rates, excluding dropouts due to an adverse event (for trial and changes to rates if PCI implemented midway through trial) Adherence rates (for trial and changes to rates if PCI implemented midway through trial) To study drug To patient tool (eg, to activity monitor)

Abbreviation: IRB, institutional review board; PCI, patient-centric initiative.

^a Metrics can be analyzed by PCI; disease indication; study phase; and maturity of patient advocacy group, if working with a patient advocacy group.

Table 3.

Qualitative Rubric Assessing 12 of the 30 Identified Patient-Centric Initiatives Described in Table 1.^a

Patient-Centric Initiative	Classification	Cost to Conduct (Relative to Other Patient-Centric Initiatives)	Ease of Conducting (Relative to Other Patient-Centric Initiatives)	Reported Impact at Time of Study (Relative to Other Patient-Centric Initiatives)	Number of Collected Quantitative Metrics
Patient advisory boards and focus groups	Partnerships	Inexpensive	Easy for pharma to conduct	Major impact	40
Patient group support and involvement	Partnerships	Inexpensive	Easy for pharma to conduct	Major impact	42
Adaptive trial designs and adaptive licensing	Protocol design	Moderately expensive	Difficult for pharma to conduct	Major impact	9
Open design and crowdsourcing	Protocol design	Inexpensive	Moderately difficult for pharma to conduct	Some impact	11
Home nursing networks/logistics assistance	Patient ease	Moderately expensive	Moderately difficult for pharma to conduct	Some impact	6
Patient counseling and education	Patient ease	Inexpensive	Moderately difficult for pharma to conduct	Major impact	10
Digital medicine	Technology	Expensive	Difficult for pharma to conduct	Some impact	1
Direct-to-patient clinical trials/telemedicine	Technology	Moderately expensive	Easy for pharma to conduct	Some impact	55
Electronic consent (eConsent)	Technology	Moderately expensive	Moderately difficult for pharma to conduct	Limited impact	0
Gaming PRO	Technology	Expensive	Difficult for pharma to conduct	Limited impact	1
Patient apps for clinical data collection	Technology	Expensive	Difficult for pharma to conduct	Some impact	50
Social media/online engagement	Technology	Inexpensive	Easy for pharma to conduct	Major impact	33

Abbreviation: PRO, patient-reported outcome.

^a Reported impacts assess changes in study/program quality, study/program speed, and overall impact on the patient.

Of the 44 metrics identified in the table, we asked working group members to rank metrics in order of importance. Over 80% of working group members selected the following 8 metrics as most meaningful to answer key performance questions at their company:

Changes to development timelines

Number of patients reached by specific PCI

Overall patient satisfaction in a given study

Cost of the PCI implemented

The PCI used in a specific clinical trial protocol

Number of drop outs (not due to an adverse event)

Number of changes to the protocol due to an amendment (that could have been avoided)

Study conduct duration (first patient in to last patient out)

In addition to gathering feedback from working group members on which impact measures would be most meaningful to answer key performance questions, we also assessed the ease of collecting the top 8 metrics deemed most essential. Of the 8 selected metrics, only number of dropouts (not due to an adverse event) and study conduct duration were selected as extremely easy to collect by more than 50% of working group members. Figure 1 compares the number of respondents that rated metric as extremely valuable to the ease of collecting the metric.

The 10 patients interviewed overall indicated that since the discussion was focused on metrics measuring overall performance, they were not comfortable providing suggestions on metrics to collect. However, they were comfortable with our proposed toolkit.

Discussion

Because of a lack of common performance metrics collected by the biopharmaceutical industry, we have compiled standardized metrics within a toolkit that can be applied to assess the return on engagement of PCIs. We also collected and analyze results of real examples (case studies) for 12 of 30 identified PCIs with measurable benefit to drug development from patient involvement and engagement.

A few broad trends were observed, and some return assessments showed long-term drug development portfolio improvement, allowing companies to save millions with relatively small investment of resources. Overall, low cost PCIs generated the highest benefit: when companies engaged patients, particularly patient groups, drug development performance improved as evidenced by faster trial planning, IRB & regulatory agency approval, and study enrollment, and by fewer protocol amendments. Additionally, patient feedback was more positive, and patients reported improvements in self-efficacy of disease management via patient counseling and education programs.

The results from the literature review and use cases provide drug development teams with evidence and insights to facilitate the adoption of specific PCIs and to help with the selection of initiative that provide the highest impact to study volunteers and to drug development organizations. It is also hoped that the biopharmaceutical industry will use standardized metrics captured in the toolkit to assess return on engagement from various PCIs.

Though the study results are useful in establishing a baseline of patient-centric practice and measurement of its impact in drug development within the biopharmaceutical industry, it is important to note the limitations of this study. In the assessment of the PCI implementation landscape, the possibility of selection bias exists and must be considered before generalizing the observed levels of PCI adoption to the wider industry. The organizations invited to participate were known to have some interest in patient-centric drug development, and those who opted to participate were potentially farther along in their planning and execution of initiatives compared to those from organizations that did not opt to participate. In the assessment of metrics use and ROE determination, publication bias may have limited our access to negative results. Additionally, differences in the number of manuscripts identified for each PCI may bias results. Furthermore, cases identified were mainly from industry and not patient groups. Additionally, data gathered were not always from peer-reviewed sources, but rather from interviews, news stories, or trade press. Finally, many companies were still conducting studies using PCIs and are publishing the results now or in a few years. Our assessment is for a point in time, and results may change based on what later case studies and publications are reporting.

In phase II, participating companies will be engaged in the development of an approach to assessing organizational levels of preparedness, capability, and experience with patient-centric drug/medical product development. The assessment approach will be conducted and validated among participating companies, and the resulting data will be aggregated for overall findings. The final product will be a tool to assess organizational readiness to implement a program of patient-centered drug development and a metrics toolkit to guide planning and assess the impact of efforts.

Conclusion

The assessment of ROE is by nature complex, given the number of possible PCIs, the variability of circumstances in which they may be implemented, and the unique organizational culture and operating dynamics. Standardized metrics are useful in measuring the impact of PCIs and serve as critical input into management decision making. The results of this study provide retrospective evidence on the impact of specific patient-centered initiatives on drug development performance. The case examples identified also inform an initial set of standardized return-on-engagement metrics. We hope that these metrics—compiled within a toolkit—will become a routinely used reference resource.

Footnotes

Acknowledgments

The authors would like to thank Dr Mary Jo Lamberti, Elizabeth Lincoln, and Michael Wilkinson for their contributions to this manuscript.

Declaration of Conflicting Interests

No potential conflicts were declared.

Funding

This study was funded by an unrestricted organizational grants from AstraZeneca, Bayer, Biogen, EMD Serono, HealthiVibe LLC, Horizon Pharma, ICON Plc, INC Research, Janssen Pharmaceuticals, Merck, Merck KGAA, MK&A, Novartis Pharmaceuticals, Purdue Pharma, Quintiles, and UCB.

ORCID iD

Stella Stergiopoulos, MS, MPH

References

From aspiration to application: 5 years of patient-centricity. http://www.fastercures.org/assets/Uploads/Patients-Count-June-2017.pdf. Published 2017. Accessed May 11, 2018.

Stergiopoulos

. #PatientVoiceMatters: how social media is bringing patients and biopharmaceutical companies together to improve drug development. Clin Ther. 2017;39(11):2170–2172.

Lamberti

Awatin

. Mapping the landscape of patient-centric activities within clinical research. Clin Ther. 2017;39(11):2196–2202.

Getz

. Establishing return-on-investment expectations for patient-centric initiatives. Therapeutic Innovation & Regulatory Science. 2015;49(5):745–749.

Milne

C-P

. The use of social media in orphan drug development. Clin Ther. 2017;39(11):2173–2180.

Anand

Brandwood

Jameson Evans

. Improving patient involvement in the drug development process: case study of potential applications from an online peer support network. Clin Ther. 2017;39(11):2181–2188.

Housman

. I’m Home(screen)! Social media in health care has arrived. Clin Ther. 2017;39(11):2189–2195.

Getz

. Examining and enabling the role of health care providers as patient engagement facilitators in clinical trials. Clin Ther. 2017;39(11):2203–2213.

Ashkenazy

Peterson

. PAUSE: a patient-centric tool to support patient-provider engagement on menopause. Clin Med Insights Women’s Health. 2018;11:1179562x18757467.

10.

Ashkenazy

Schneider

. A patient centricity team tool to enable patient-focused drug development. Therapeutic Innovation & Regulatory Science. 2016;50(5):577–580.

11.

Gould

. AIDS and FDA drug-approval policy: an evolving controversy. J Health Soc Policy. 1990;2(2):39–46.

12.

Crimp

. Before occupy: how AIDS activists seized control of the FDA in 1988. https://www.theatlantic.com/health/archive/2011/12/before-occupy-how-aids-activists-seized-control-of-the-fda-in-1988/249302/. Published 2011. Accessed October 17, 2018.

13.

Evolution of patient engagement at FDA text alternative. https://www.fda.gov/AboutFDA/CentersOffices/OfficeofMedicalProductsandTobacco/ucm589527.htm. Published 2018. Accessed October 17, 2018.

14.

Getz

Kaitin

. Why does the industry need a change? In: Schueler

, ed. Re-Engineering Clinical Trials. 1st ed. Philadelphia, PA: Elsevier; 2014.

15.

US Food and Drug Administration. Learn about patient engagement. https://www.fda.gov/ForPatients/PatientEngagement/default.htm#_PRO__. Accessed May 11, 2018.

16.

US Food and Drug Administration. Externally-led patient-focused drug development meetings. https://www.fda.gov/forindustry/userfees/prescriptiondruguserfee/ucm453856.htm. Accessed May 11, 2018.

17.

Clinical Trials Transformation Initiative. CTTI recommendations: effective engagement with patient groups around clinical trials. https://www.ctti-clinicaltrials.org/files/pgctrecs.pdf. Accessed May 11, 2018.

18.

Framework of the CTTI/FDA Patient Engagement Collaborative. https://www.ctti-clinicaltrials.org/framework-cttifda-patient-engagement-collaborative. Published 2018. Accessed May 11, 2018.

19.

Medical Device Innovation Consortium (MDIC) Patient Centered Benefit-Risk Project Report. Arlington, VA: Medical Device Innovation Consortium; 2015.

20.

Anderson

Kimberly McCleary

. On the path to a science of patient input. Sci Transl Med. 2016;8(336):336ps11.

21.

Sheridan

Schrandt

Forsythe

Hilliard

Paez

. The PCORI engagement rubric: promising practices for partnering in research. Ann Fam Med. 2017;15(2):165–170.

22.

Daphnee

Pushparajah

, Niels Westergaard. EUPATI: collaboration between patients, academia and industry to champion the informed patient in the research and development of medicines. J Med Dev Sci. 2015;1(1):7.

23.

Ocloo

Matthews

. From tokenism to empowerment: progressing patient and public involvement in healthcare improvement. BMJ Qual Saf. 2016;25(8):626–632.

24.

Lincoln

. DIA’s Considerations Guide to Implementing Patient-Centric Initiatives in Health Care Product Development. Washington, DC: Drug Information Association; 2017.

25.

Kielstra

. Reinventing Biopharma: Strategies for an Evolving Marketplace, The Patient-Led R&D Strategy. London: Economist Intelligence Unit; 2012.

26.

Mullins

Vandigo

Zheng

Wicks

. Patient-centeredness in the design of clinical trials. Value Health. 2014;1 7(4):471–475.

27.

Patients and Public Involvement and Engagement Strategy: 2015-2020 . Delivery Research to Make Patients, and the NHS, Better. Leeds, England: National Institute for Health Research; 2015.

28.

Levitan

Getz

Eisenstein

. Assessing the financial value of patient engagement. Therapeutic Innovation & Regulatory Science. 2017;52(2):220–229.

29.

Croft

McLoughlin

. Managing performance in patient centricity. Kinapse. 2015:13.

30.

Crocker

Bostock

Locock

. Is it worth it? Patient and public views on the impact of their involvement in health research and its assessment: a UK-based qualitative interview study. Health Expectations. 2016:10.

31.

Parsons

Mullan-Jensen

Tham

Warner

Wever

. What do pharmaceutical industry professionals in Europe believe about involving patients and the public in research and development of medicines? A qualitative interview study. BMJ Open. 2016;6:11.

32.

Comstock

. Trialbee raises $5 M to bring mobile-enabled clinical trials business to US. http://www.mobihealthnews.com/43674/trialbee-raises-5m-to-bring-mobile-enabled-clinical-trials-business-to-us. Published 2015. Accessed June 11, 2018.

33.

Bain

. Patient centricity in clinical trials: applying process management to support site and patient engagement in clinical trials. Paper presented at: DIA Annual Conference; June 26-30, 2016; Philadelphia, PA.

34.

Watson

. Enhancing patient-centered research: combining technology with other direct-to-patient strategies. Paper presented at: DIA Annual Conference; June 26-30, 2016; Philadelphia, PA.

35.

Adinamis

. Innovative in-home clinical service model provides future direction to overcome patient recruitment and retention challenges. MONITOR. 2009:3.

36.

Leiter

Diefenbach

Foster

. Use of crowdsourcing for cancer clinical trial development. Brief Commun. 2014:3.

37.

Epstein

Franks

Shields

. Patient-centered communication and diagnostic testing. Ann Fam Med. 2005;3(5):415–421.

38.

Harter

Darden

McMenemy

McElvy

Hendrich

. Consent and enrollment process: achieving high enrollment rates for obstetric research. Appl Nurs Res. 2016;29:6.

39.

Kinmonth

Woodcock

Griffin

Spiegal

Campbell

. Randomised controlled trial of patient centred care of diabetes in general practice: impact on current wellbeing and future disease risk. BMJ. 1998;317:7.

40.

Quinn

Clough

Minor

Lender

Okafor

Gruber-Baldini

. WellDoc™ Mobile Diabetes Management Randomized Controlled Trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther. 2008;10(3):160–168.

41.

Jeffs

Vollam

Young

Horsington

Lynch

Watkinson

. Wearable monitors for patients following discharge from an intensive care unit: practical lessons learnt from an observational study. J Adv Nurs. 2016;72(8):1851–1862.

42.

Leong

Sirio

Rotondi

. eICU program favorably affects clinical and economic outcomes. Crit Care. 2005;9(5):E22.

43.

Patrick

Raab

Adams

. A text message–based intervention for weight loss: randomized controlled trial. J Med Internet Res. 2009;11(1):e1.

44.

Cortez

Chen

. Thousands have already signed up for Apple’s ResearchKit. http://www.bloomberg.com/news/articles/2015-03-11/apple-researchkit-sees-thousands-sign-up-amid-bias-criticism. Published 2015.

45.

Anderson

Funnell

Butler

Arnold

Fitzgerald

Feste

. Patient empowerment: results of a randomized controlled trial. Diabetes Care. 1995;18(7):943–949.

46.

Woolf

Zimmerman

Haley

Krist

. Authentic engagement of patients and communities can transform research, practice, and policy. Health Aff. 2016;35(4):590–594.

47.

Cate

Bhattacharya

Clark

Fordham

Holland

Broadway

. Improving adherence to glaucoma medication: a randomised controlled trial of a patient-centred intervention (The Norwich Adherence Glaucoma Study). BioMed Central. 2014;14:32.

48.

Darkins

Ryan

Kobb

. Care coordination/home telehealth: the systematic implementation of health informatics, home telehealth, and disease management to support the care of veteran patients with chronic conditions. Telemed E-health. 2008;14(10):10.

49.

Kidholm

Stafylas

Kotzeva

. REgioNs of Europe WorkINg toGether for HEALTH. RENEWING HEALTH. CORDIS, European Commission; 2014.

50.

Mapping the Advocacy Environment through LEAD. 2018; http://mkanda.com/mapping-the-advocacy-environment-through-lead/. Accessed June 11, 2018, 2018.

51.

Andrzejewski

Dowling

Stamler

. Wearable sensors in Huntington disease: a pilot study. J Huntingtons Dis. 2016;5(2):199–206.

52.

Dorsey

Yvonne Chan

McConnell

Shaw

Trister

Friend

. The use of smartphones for health research. Acad Med. 2017:92(2):157–160.

53.

The Aurora Project. Highlights from The Aurora Project. 2016.

54.

Dorsey

Venuto

Venkataraman

Harris

Kieburtz

. Novel methods and technologies for 21st-century clinical trials: a review. JAMA Neurol. 2015;72(5):582–588.

55.

McAlindon

Formica

Kabbara

LaValley

Lehmer

. Conducting clinical trials over the internet: feasibility study. BMJ. 2003;327(7413):484–487.

56.

Jacobs

Bent

Tice

Blackwell

Cummings

. An Internet-based randomized, placebo-controlled trial of kava and valerian for anxiety and insomnia. Medicine. 2005;84(4):197–207.

57.

Wicks

Vaughan

Massagli

Heywood

. Accelerated clinical discovery using self-reported patient data collected online and a patient-matching algorithm. Nat Biotechnol. 2011;29(5):411–414.

58.

Guetterman

Fetters

Legocki

. Reflections on the adaptive designs accelerating promising trials into treatments (ADAPT-IT) process—findings from a qualitative study. Clin Res Regul Aff. 2015;32(4):121–130.

59.

Carlisle

Warren

. A qualitative case study of telehealth for in-home monitoring to support the management of type 2 diabetes. J Telemed Telecare. 2013;19(7):372–375.

60.

Williams

Bird

Forbes

. Randomised controlled trial of an automated, interactive telephone intervention (TLC Diabetes) to improve type 2 diabetes management: baseline findings and six-month outcomes. BMC Public Health. 2012;12:602.

61.

Smith

Selig

Harker

. Patient engagement practices in clinical research among patient groups, industry, and academia in the united states: a survey. PLoS One. 2015;10(10):e0140232.

62.

SCORR Marketing and Applied Clinical Trials MHiCTSR. SCORR Marketing and Applied Clinical Trials. 2016.

63.

Palmer

. Assessing digital trials today and the future of the Internet of clinical things Paper presented at: mHealth for Clinical Trials; June 14, 2016; Boston, MA.

64.

Dorsey

. The future of clinical trials and the Clinical Trials Transformative Initiative Paper presented at: mHealth for Clinical Trials; June 14, 2016; Boston, MA.

65.

Spencer-Manzon

Hall

. The Yale Cardiomyopathy Index, Lessons From One of the First Study Apps Built with Apple ResearchKit. Paper presented at: mHealth for Clinical Trials; 6/13/2016; Boston, MA.

66.

Lauren

. Data Collection With Wearable Sensors in Parkinson’s Disease Research. New York: Michael J. Fox Foundation for Parkinson’s Research.

67.

Gabriel Vargas

. Introducing mHealth to Amgen’s Clinical Trials. Paper presented at: mHealth for Clinical Trials; June 14–15, 2016; Boston, MA.

68.

Anguera

Jordan

Castaneda

Gazzaley

Areán

. Conducting a fully mobile and randomised clinical trial for depression: access, engagement and expense. BMJ Innov. 2016;2(1):14–21.

69.

de Wit

Koelewijn-van Loon

Collins

Abma

Kirwan

. “If I wasn’t this robust”: patients’ expectations and experiences at the Outcome Measures in Rheumatology Conference 2010. Patient. 2013;6(3):179–187.

70.

Schorr

. Leveraging the digital age for better relationships with patients. Patient Power, July 21, 2016.

71.

Leventhal

. Best practices for PG engagement in clinical trials. Paper presented at: Clinical Trials Transformation Initiative; July 21, 2016.

72.

Pablo Lapuerta

. Strengthening Outsourcing Opportunities Through Patient Centered Support. Woodlands, TX: Lexicon Pharmaceuticals, Inc; 2015.

73.

Velazquez

. Treatment for Mexican hepatitis C patients: synergy with FUNDHEPA. Paper presented at: EUPATI Workshop: Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

74.

Uhlenhopp

. Advisory board of expert melanoma nurses and melanoma expert patient advocates. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

75.

Uhlenhopp

. Advisory Board of Expert GYN nurses and Ovarian Cancer patient group experts. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

76.

Novartis Pharma

. Advisory Board on Psoriasis Clinical Trials. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

77.

Pushparajah

. Bettering the Informed Consent Form by Using Patient Perspectives. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

78.

Matyushenko

. Charitable Foundation “Children with spinal muscular atrophy” promote phase I clinical trial in collaboration with academic institution in Ukraine. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

79.

Isabelle David

Corbier

Duchossoy

Lotz

. Collaboration with a French Patient Association in Oncology Trials: “ToujoursPlus Loin.” Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

80.

Duchenne Parent Project. Development of Exonskip technology for Duchenne MD. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led MEdicines R&D; July 23, 2014; Berlin.

81.

Janssen

. HIV patients actively involved by Janssen R&D. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

82.

Duchossoy

. Informed consent form: An innovative approach, GR56: Guide du Rédacteur (Writer’s guide)—5 Recommendations, 6 Rules. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industy-Led Medicines R&D; July 23, 2014; Berlin.

83.

Priolo

. Input of patients organizations into Cushing’s disease clinical trial design. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

84.

Priolo

. Input of patients organizations on a rare disease registry: TOSCA. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

85.

Houÿez

. Own experience as interface between sponsors of clinical trials and participants. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

86.

ECABotEATG. Patient advocacy for combination of two investigational compounds: DUET 1&2. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

87.

Geissler

. Patient feedback on a trial protocol of a paediatric CML study. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

88.

Kraus

. Patient friendly informed consent document. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

89.

Regnante

. Patient input forum. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

90.

Regnante

. Patient medication labeling(USA). Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

91.

Novo Nordisk A/S. Patient sounding board. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

92.

Warner

. Psoriasis. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

93.

Novo Nordisk. Using user research—training in the anthropological approach. Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

94.

Dillon

Paradis

; Genzyme. Working with a patient organisation and academia in the development of a treatment for an ultra-orphan disease (Pompe disease). Paper presented at: EUPATI Workshop, Meaningful Patient Involvement in Industry-Led Medicines R&D; July 23, 2014; Berlin.

95.

Brennan

. SONAR: blinded diagnostics, same day lab test results for pharmaceutical clinical trials. Paper presented at: SCOPE 2016; February 23–25, 2016; Miami, FL.

96.

Lawrence

. Updated: FDA rejects Otsuka, Proteus pill with medication adherence monitoring. http://www.fiercebiotech.com/medical-devices/updated-fda-rejects-otsuka-proteus-pill-medication-adherence-monitoring. Published 2016.

97.

Horizon Pharma. Horizon Pharma PLC Completes Target Enrollment of 90 Patients for Phase 3 Trial of ACTIMMUNE(R) (interferon gamma-1b) for the Treatment of People With Friedreich’s Ataxia. Deerfield, IL: Horizon Pharma; 5/5/2016.

98.

Ellis

Kass

. How are PCORI-funded researchers engaging patients in research and what are the ethical implications? AJOB Empirical Bioethics. 2017;8(1):1–10.

99.

Kato

Bradlyn

Pollock

. A video game improves behavioral outcomes in adolescents and young adults with cancer: a randomized trial. Pediatrics. 2008;122(2):15.

100.

Cristian Pop-Eleches

Habyarimana

James P.

Zivin

Joshua G.

Goldstein

Markus P.

Damien de

Walque

MacKeen

Leslie

Haberer

Jessica

Kimaiyo

Sylvester

Sidle

John

Ngare

Duncan

and Bangsberg

David R

. Mobile phone technologies improve adherence to antiretroviral treatment in a resource-limited setting: a randomized controlled trial of text message reminders. AIDS. 2011;25(6):18.

101.

Thakkar

Kurup

Laba

. Mobile telephone text messaging for medication adherence in chronic disease: a meta-analysis. JAMA Intern Med. 2016;176(3):340–349.

102.

Butler

. How “bring your own device” and other innovations are improving patient engagement. PharmaVoice. 2016:2.

103.

Gleason

. #WhyWeDoResearch Tweetfest – the world’s first research tweetfest. http://clinfield.com/2016/06/whywedoresearch-tweetfest-the-worlds-first-research-tweetfest/. Published 2016.

104.

de Wit

Abma

Koelewijn-van Loon

Collins

Kirwan

. Involving patient research partners has a significant impact on outcomes research: a responsive evaluation of the international OMERACT conferences. BMJ Open. 2013;3(5).

105.

I-SPY2 Advocate Update. Paper presented at: San Antonio Breast Cancer Symposium: I-SPY 2; December 11, 2009; San Antonio, TX.

106.

Selig

WKD

. Key Considerations for Developing and Integrating Patient Perspectives in Drug Development: Examination of the Duchenne Case Study. Biotechnology Innovation Organization, Parent Project Muscular Dystrophy; 2016.

107.

Johnson

. Astrazeneca outperform clinical trial standards, but increasing the use of patients’ data: data quality & technology in clinical trials. EyeForPharma. 2016.

108.

Bartek

. Foundation-industry relationships—a new business model joint-venture philanthropy in therapy development. Curr Topics Med Chem. 2014;14(3):313–318.

109.

Patient-centric profitability: pharma’s global survey & analysis. A discussion of results from the Aurora Project. EyeForPharma. 2016.

110.

Neumann

. Transforming the Clinical Trial Enterprise. Rutgers Business School; 2014.

111.

Nayak

Bolognese

. Adaptive Clinical Trials (Whitepaper). Cytel.

112.

Ownby

Zoratti

. Improving efficiency and reducing costs: design of an adaptive, seamless, and enriched pragmatic efficacy trial of an online asthma management program. Contemp Clin Trials. 2014;38(1):19–27.

113.

Senchaudhuri

. Seamless Adaptive Clinical Trials: now that we get the statistics, what’s really at stake? Cytel, May 20, 2015.

114.

Mehta

. Group Sequential and Adaptive Designs: Part II Adaptive Designs. Cytel Corporation and Harvard TH Chan; May 2, 2015.

115.

Korieth

. Pfizer’s “virtual” clinical trial could have far-reaching implications for future research productivity. CenterWatch News Online 2011. https://www.centerwatch.com/news-online/2011/06/20/pfizers-virtual-clinical-trial-could-have-far-reaching-implications-for-future-research-productivity/. Accessed June 12, 2018.

116.

Langel

. Remote trial methodology and the role of the virtual site. ClinPal, 2016.

117.

eClinicalHealth announces successful results for an entirely remote online clinical trial. ClinPal, 2016.

118.

Rosenberg

. Early inroads for wearable devices in clinical trials. https://www.centerwatch.com/news-online/2015/08/15/early-inroads-for-wearable-devices-in-clinical-trials/. Published 2015. Accessed June 12, 2018.

119.

Orloff

Douglas

Pinheiro

. The future of drug development: advancing clinical trial design. Nature reviews Drug discovery. 2009;8(12):949–957.

120.

Hemmings

. EMA Adaptive licensing: a concept for accelerated access to innovative medicines? European Medicines Agency. 2015.

121.

Cowie

Blomster

Curtis

. Electronic health records to facilitate clinical research. Clinical Research in Cardiology. 2017;106(1):1–9.

122.

Steinhubl

Topol

. Digital medicine, on its way to being just plain medicine. npj Digital Medicine. 2018;1(1).

123.

Sweeney

. The Direct-to-Patient Model: Your Top 10 Q&A. 2016; https://www.worldcourier.com/insights/the-direct-to-patient--model. Accessed May 16, 2018.

124.

McNair

. Electronic Informed Consent: A New Industry Standard. WIRB Copernicus Group. 2014.

125.

Adelman

. PRO Powered Game Mechanics. 2016. https://www.ratchethealth.com/pro-powered-game-mechanics/. Accessed May 16, 2018.

126.

Anderson

Getz

. Insights and Best Practices for Planning and Implementing Patient Advisory Boards. Therapeutic Innovation & Regulatory Science. 2017;0(0):2168479017720475.

127.

Wicks

. Could digital patient communities be the launch pad for patient-centric trial design? Trials. 2014;15(1):172.

128.

Nader

Hart

. Patient Advocacy: An emerging role within pharma and biotechnology. BioNJ and Merrill DataSite. 2015.

129.

Rajadhyaksha

. Conducting Feasibilities in Clinical Trials: An Investment to Ensure a Good Study. Perspectives in clinical research. 2010;1(3):106–109.

130.

Messner

Moloney

Warriner

Wright

Foster

Saag

. Understanding practice-based research participation: The differing motivations of engaged vs. non-engaged clinicians in pragmatic clinical trials. Contemp Clin Trials Commun. 2016;4:136–140.

131.

Norris

Lau

Smith

Schmid

Engelgau

. Self-management education for adults with type 2 diabetes: a meta-analysis of the effect on glycemic control. Diabetes Care. 2002;25(7):1159–1171.