Abstract
In any clinical trial, it is essential to monitor the accumulating data to be sure that the trial continues to be safe for participants and that the trial is being conducted properly. Data monitoring committees, independent expert panels who undertake regular reviews of the data as the trial progresses, serve an important role in safeguarding the interests of research participants and ensuring trial integrity in many trials. Many pragmatic clinical trials, which aim to inform healthcare decisions by comparing alternate interventions in heterogeneous healthcare delivery settings, will warrant review by an independent data monitoring committee due to their potential impact on clinical practice. However, the very features that make a trial “pragmatic” may pose challenges in terms of which aspects of a trial to monitor and when it is appropriate for a data monitoring committee to intervene. Using the Pragmatic-Explanatory Continuum Indicator Summary tool that draws distinctions between pragmatic and explanatory clinical trials, we review characteristics of pragmatic clinical trials that may have implications for data monitoring committees and interim monitoring plans. These include broad eligibility criteria, a focus on subjective patient-centered outcomes, and in some cases a lack of standardized follow-up procedures across study sites. Additionally, protocol adherence is often purposefully not addressed in pragmatic trials in order to accurately represent the clinical practice setting and maintain practicability of implementation; there are differing viewpoints as to whether adherence should be assessed and acted upon by data monitoring committees in these trials. Some other issues not specifically related to the Pragmatic-Explanatory Continuum Indicator Summary criteria may also merit special consideration in pragmatic trials. Thresholds for early termination of a pragmatic clinical trial might be controversial. The distinguishing features of pragmatic clinical trials require careful consideration when developing interim data monitoring plans, and trial sponsors, investigators, and data monitoring committees should agree on a plan before trial inception. Finally, special expertise, such as an informatics, may be helpful on data monitoring committees for some pragmatic clinical trials. Patient representatives may provide particularly valuable insights in the monitoring process.
Keywords
Introduction
All clinical trials require regular monitoring to identify emerging safety concerns, as well as problems with trial conduct that might undermine its ability to generate reliable data. In many trials, particularly those evaluating treatments for serious diseases and/or with important implications for public health, an independent data monitoring committee (DMC) is established to perform regular review of the accumulating data and advise the sponsor and investigators of any need for modification of the trial design or procedures.
The process of clinical trial data monitoring and the operation of DMCs have been discussed in detail.1–4 The primary objectives of a DMC are to safeguard the interests of study participants and preserve trial integrity and credibility. Special issues in the monitoring of certain types of trials have also been addressed.6,7
In recent years, there has been growing interest in “pragmatic” clinical trials (PCTs), although the concept dates back to the 1960s. 8 The “large, simple trials” in cardiovascular disease conducted in the 1980s and 1990s fit this term well,9–12 as do some trials done more recently.13,14 Tunis et al. 15 described four criteria that a PCT must meet: (1) clinically relevant alternative interventions to compare, (2) a diverse population of study participants, (3) heterogeneous practice settings, and (4) a broad range of health outcomes. Thorpe and colleagues developed the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) framework to identify individual dimensions of trial design that characterize a trial as more pragmatic or explanatory (Table 1). 5
Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) dimensions of trial design. 5
The general principles of clinical trial data monitoring will apply to PCTs, and most aspects of interim monitoring of PCTs will probably not differ from those in more traditional trials, but some features of PCTs raise special issues potentially relevant to their DMCs. In this article, we discuss four aspects of trial design included in the PRECIS criteria and relevant to data monitoring for PCTs: outcomes, protocol adherence, patient eligibility, and intensity of follow-up. Broad considerations such as need for an independent DMC, analytical issues, and composition of a DMC for PCTs are also addressed.
What PCTs should have an independent DMC?
All clinical trials require some level of monitoring, but not all trials need an independent DMC. DMCs are most valuable when interim review of comparative data is needed to ensure participant safety; independent review allows the sponsor and study investigators to remain blinded to emerging data patterns to avoid potential influence of interim data on trial conduct.16–18 PCTs, being primarily intended to define optimal treatment practices, will likely utilize DMCs as interim data reviews will be important from both ethical and logistical perspectives. Problems with quality of trial conduct will be just as important to identify in PCTs as in other trials; some quality issues, such as completeness and timeliness of follow-up of trial participants, may require evaluation of data by treatment arm. Early identification of emerging safety issues will also be warranted in most PCTs.
Nevertheless, an independent DMC may not be necessary for every PCT. For example, consider a study of obesity in children that evaluates the effect of meeting in community groups to encourage weight loss. In this trial with a minimal risk, reversible intervention, study conduct issues could be adequately monitored without comparing treatment arms. Other trials in which there is no ethical need to perform interim comparisons of effectiveness among treatment arms—for example, short-term trials comparing treatments to relieve minor symptomatic conditions—may only require monitoring for emerging safety events, with effectiveness reviewed only when safety issues require risk-to-benefit considerations for decisions about trial continuation or termination.
What will a DMC monitor in a PCT?
Outcomes to be monitored
PCTs are partly characterized by their focus on patient-centered outcomes that often incorporate multiple measures of quality of life. 19 Although some explanatory trials include quality of life measures, their primary focus is more typically on objective clinical or biological measures such as mortality, progression of disease, change in well-established biomarkers, or highly specific aspects of quality of life such as number of depression episodes. In PCTs, a DMC may need to be comfortable monitoring more subjective outcomes than are typically encountered in traditional trials.
Even when major outcomes, such as disease progression, are assessed, they may be assessed and reported only by the treating clinicians, or derived from electronic health record (EHR) or claims databases, rather than by a central adjudication group. This may introduce more variability into the outcome summary, generated at least in part by differences among sites in reporting and data systems. Additionally, sites for PCTs may be more heterogeneous than in a traditional trial in terms of patient characteristics and treatment approaches of practitioners. Thus, DMCs may need to pay close attention to site-specific data to determine whether an emerging result may be attributable to one or two sites and is perhaps not widely generalizable.
Although data collection in PCTs should generally be limited to key study outcomes, if prior information about biological mechanisms suggests that the intervention may affect patient safety (presumably this would be a primary justification for a DMC), the committee should also have access to clinical and biological outcomes. For example, it may be necessary for the DMC to monitor changes in markers such as serum creatinine levels for possible renal toxicity, as would be done in usual clinical practice. When safety is an issue, it will be optimal to have standardized measurement and timely reporting of these outcomes, as well as the usual safety outcomes such as mortality rates and standard adverse event tabulations. This may prove difficult in some PCTs that seek to assess the effectiveness of an intervention in the absence of rigorous study protocols that tend to modify investigators’ usual operating practice. The DMC, in collaboration with the study sponsor, should decide during protocol development whether and to what extent standardized follow-up reporting should be required relative to the known or anticipated safety profile of the interventions.
Protocol adherence
The intensity of measuring participants’ adherence with the prescribed intervention and consideration of adherence-improving strategies are subjects for consideration in PCTs. If adherence to protocol and/or other quality parameters are not being monitored, a failure to find a difference in treatment effects will be difficult to interpret, as this may be due to true equivalence of treatments or to widespread failure to adhere to the assigned interventions. In most clinical trials, a DMC monitors adherence in addition to health-related outcomes to ensure that the quality of study conduct is sufficient to yield reliable data. In PCTs, however, the objective is to evaluate the effects of treatment in a “clinical care” context—the way the treatment would actually be used. Thus, many would argue that protocol adherence is irrelevant in PCTs, since the level of nonadherence should reflect treatment use in everyday practice. Others might counter that even in a PCT, adherence is relevant, since extensive nonadherence will render the data on treatment effects uninterpretable. Another view is that information regarding adherence to a treatment is a worthy finding in its own right. 20
When poor adherence is observed in a PCT, there may be opportunities for a DMC to suggest opportunities for improvement without undermining the pragmatic nature of the trial. For example, those administering the treatment may not have understood that specific treatments had been assigned to specific patients. Re-explaining the study to participating investigators would not necessarily be inconsistent with the principles of PCTs. However, if participants fail to adhere to a protocol because they find aspects of the trial and their assigned treatment (cost, comfort, convenience, etc.) to be problematic, that may well reflect the way the product would be used outside the trial.
Patient eligibility
DMCs typically review the quality of the accruing data. One aspect of data quality is the adherence to eligibility criteria; the discovery that a substantial number of randomized participants did not meet the trial’s eligibility criteria would raise concerns about the quality of trial conduct. In contrast with most traditional trials, a PCT generally tries to keep eligibility criteria as minimal as possible to achieve a maximally generalizable result. In fact, an approach advocated for large, simple trials is the “uncertainty principle”: if the clinician and the potential participant are both uncertain about whether the treatment under investigation would be useful for that participant, that determination would be sufficient to establish eligibility. 21 (Of course, if allergy, genetic profile, prior history, etc., would make a treatment dangerous for some individuals, appropriate exclusion criteria could be added, consistent with what would happen in the clinical care setting.)
Assessment of ineligibility rates is particularly important in “open-label” trials in which both participants and clinicians know the assigned treatments. In such trials, a difference in ineligibility rates could suggest that some “ineligible” individuals might really have just rejected their assigned treatment. While ineligible subjects are often excluded from analyses, those who reject treatment should remain in, to be consistent with the intention-to-treat principle. Such exclusions have the potential to bias results in a PCT, as they would in any randomized trial. Because many PCTs are conducted in an open-label fashion (to get “real-world” answers, to limit burden to participants, to minimize costs), it may be particularly important for DMCs to monitor ineligibility rates by treatment arm.
Follow-up of trial participants
As noted previously, a defining characteristic of many PCTs is the focus on assessing effectiveness in the absence of adherence to rigorous study protocols that modify the standard practices of study investigators and/or institutions. However, for a DMC to effectively monitor a trial for safety and/or efficacy, it is essential that data be of high quality (i.e. sufficiently accurate to support reliable conclusions), up-to-date, and ideally, consistently measured across study sites and participants. This may appear to conflict with the guiding principle behind PCTs, which is to assess the effects of treatments just as they are used in practice. If quality problems emerge that threaten the ability of the trial to provide reliable results, however, it would be worthwhile to identify these problems at an early stage to determine whether they can be addressed, or whether the trial should be terminated as unlikely to yield a reliable result.
As with other clinical trials monitored by a DMC, follow-up intensity and frequency of monitoring the interim data will depend upon the clinical setting, particularly the known or presumed safety profile of the study interventions, and should be agreed upon by the DMC and the study sponsor at trial inception. If the DMC is charged with monitoring interim efficacy and safety, follow-up of trial outcomes should ideally be standardized across study sites, which may require some sites to modify their standard follow-up practices. In cluster-randomized trials where hospitals or other healthcare settings represent the sampling units, treatment effects may be confounded by differential follow-up frequencies at participating sites unless these varying practices are accounted for in the randomization process. Furthermore, when EHRs or complicated database manipulations are used for obtaining trial outcomes, it may be necessary to have more intense follow-up at early stages to assess data quality and ensure credibility and validity of trial results going forward.
Analytical issues
Early termination
Analyses of data from PCTs will largely proceed as in traditional trials, but there are a few areas of particular concern. One important issue is the appropriate criterion for early termination. PCTs typically address questions of optimal practice so that all interventions being compared are considered appropriate according to available medical evidence and are likely in wide use. In such trials, the level of evidence required to influence provider and patient decision-making may be even greater than what is generally required for regulatory approval of an investigational treatment.
Some would argue that the criteria for early termination with a conclusion that one treatment is superior should be more stringent for PCTs because they may have a major impact on clinical practice, and that substantial thought should be given to whether early termination for effectiveness should be considered at all. However, others assert that it is not possible to estimate the strength of evidence that would be required to change clinical practice, and the primary responsibility should be to avoid exposing trial participants to unnecessary risk. Study sponsors, investigators, and DMC members should reach consensus prior to initiating the trial regarding the criteria for consideration of early termination, keeping in mind that while these criteria provide statistical guidelines to ensure protection of type I error, many other factors will contribute to a DMC’s decision about continuing or terminating a trial.
Early termination for “futility” may also require special considerations in PCTs. Futility assessments are common in trials testing experimental treatments against placebo or “standard of care.” Early termination of a trial that is very unlikely to change treatment practices or introduce new care options is desirable, both from the perspective of trial participants (who would avoid the burdens of research participation) and trial organizers (who could put the resources to other uses). In trials comparing treatments in common use, however, detection of small differences in outcomes may be particularly important, as even small advantages will benefit many people.
Some PCTs may assess the value of a widely used but insufficiently studied treatment; for these trials, early termination on futility grounds may yield insufficient data to persuade providers and/or patients of lack of effectiveness. Furthermore, an understanding of the potential for less common but serious adverse effects of widely used treatments is important to uncover in both circumstances. As an example, a placebo-controlled trial of digitalis, a drug that had been a mainstay of treatment of heart disease for many decades, was undertaken in the 1990s because it had never been studied in a large trial that would collect enough serious outcomes to permit an adequate benefit–risk assessment. 22
Trial leadership and DMC members must agree on the extent and type of evidence that could lead to consideration of early termination, whether for effectiveness or futility. Early termination could be an area of particular interest to the patient community and underscores the value of a patient representative on a DMC. Diverse perspectives in DMC deliberations could make it more difficult to achieve consensus on early termination, but may clarify the need for additional evidence to ensure wide acceptance of study findings.
Optimizing precision of estimation
Because PCTs typically include heterogeneous populations, analytical methods that can limit random error will improve precision of estimation and power to detect differences in treatment effects. While study investigators determine the approaches to randomization and primary analysis, a DMC might ask to see analyses that stratify or otherwise adjust for important known prognostic factors.
De-centralized analysis
For multicenter PCTs in which privacy concerns preclude having all EHR data available at a central statistical site, analyses may need to be performed at each site behind a firewall and the results delivered to the central statistical group to be combined. This may result in less timely data, and delays in providing a DMC with any requested follow-up analyses, due to steps necessary for data quality assurance and aggregation of results.23,24
Cluster designs
PCTs commonly use cluster designs; 25 that is, groups of participants are randomized as a unit, rather than individually. These units are typically healthcare delivery sites such as clinics or hospitals, but they might be family groups, classrooms, or entire communities. The design of cluster-randomized trials requires estimating the intracluster correlation coefficient (ICC)—the parameter that indicates the expected correlation of treatment effect estimates within each cluster. 24 Such estimates are usually very difficult to make, 26 so it is worthwhile to monitor this parameter as study data accumulate. If early calculations indicate that the actual ICC is likely to be much larger than originally estimated, it may be necessary to increase the number of clusters to maintain desired power.25,27
The consistency of measurement is another important issue in cluster-randomized trials, as noted earlier. Analyses of consistency in data collection and reporting practices across sites (sampling units in cluster-randomized trials) should be performed throughout the trial and presented to the DMC. If clusters assigned to one treatment have on average different follow-up practices from the other clusters, monitoring outcomes will be challenging.
Composition of a DMC for PCTs
Finally, we consider whether the composition of a DMC should differ from those for traditional trials. A DMC for any trial requires members with varying expertise to address the complex issues that come with trial monitoring. The goal of a PCT is to develop information to support decisions about healthcare made by patients, clinicians, and others involved in making healthcare decisions. 28 The types of outcomes considered in PCTs, as well as the manner in which trial data are obtained, may necessitate additional expertise beyond what is required for DMCs in the traditional setting, which typically comprise clinicians, statisticians, and in some cases bioethicists.
The DMC for a PCT, as for any clinical trial, should include experts in the medical setting being investigated. For example, in a trial considering an intervention to lower the rate of hospital-related infection, the DMC may include a surgeon, a hospitalist, and an infectious disease expert. An experienced statistician is also needed, as the presentation of interim data is generally complex and easily misinterpreted; statisticians help ensure that the proper analyses are performed and correctly interpreted in order for the DMC to make informed recommendations. Complexities of PCTs, such as the use of cluster designs and the reliance on EHR data, make statistical expertise especially important for DMC decision-making.
PCTs may also require other types of expertise. Because they may involve the use of EHRs, natural language processing, unusual data sources (e.g. biosensors or activity monitors), or complicated database manipulation or linkage for obtaining trial outcomes, the assessment of data quality may necessitate the expertise of a biomedical informatician. This individual could play a critical role in assessing and interpreting the reliability and timeliness of data elements in a DMC report, as well as investigating whether differences in data management procedures across study sites may confound safety and efficacy signals.
We must also consider the potential role of patient representatives. Patient representatives have been included on DMCs for many trials and are required, for example, in most Phase III and many Phase II trials of the US cancer cooperative groups funded by the National Cancer Institute. 7 Given the focus on patient-centered outcomes in PCTs, the role of a patient representative may be particularly important. PCTs funded by the Patient-Centered Outcomes Research Institute (PCORI) 29 require inclusion of patient partners as research team members involved in study design and conduct. Including patient representatives in the interim monitoring process would be in keeping with the involvement of patients on research teams.
Patient representatives on the DMC can provide a patient’s perspective regarding the relative benefits, burdens, and potential harms of the interventions under study. They may identify issues regarding the trial outcomes and treatments that may not be recognized by other DMC members, and offer valuable perspectives regarding emerging benefits and risks.30–33 Initial experiences in PCORI-funded studies have demonstrated that patient partners bring critical insights to study design and conduct questions. 34 Another possible function of a patient representative on the DMC is to provide insight into optimal ways that information can be shared with participants and relevant patient groups, who have long called for greater access to research results.35,36
In order to meaningfully participate on the DMC, patient representatives will need to understand PCT methodology and feel comfortable with the scientific concepts being discussed. Simple tactics like avoiding jargon, refraining from the use of acronyms, and encouraging committee members to ask questions when a concept is unclear can equalize participation and benefit all DMC members, not only the patient representative. All DMC members, as in any clinical trial, will need to appreciate the need for confidentiality of interim results.
An important question is how to identify appropriate candidates to serve as patient representatives on DMCs. A patient representative might be a clinical or statistical scientist with the disease under study, someone with little or no background in clinical research, a community leader, 37 or a leader of a patient advocacy organization. The perspectives offered by such individuals might be quite different.38–40 Best practices for selection are unclear; optimal choices might depend on the specifics of a particular study. It will be important to share experiences to provide clearer guidance in the future.
Conclusion
Although DMCs for PCTs will operate according to well-accepted principles for clinical trial data monitoring, PCTs have features that may warrant special attention in developing monitoring plans and establishing DMCs. In particular, when the DMC, sponsor, and investigators are considering what parameters to monitor and what outcomes might cause concern, examination of the PRECIS criteria as a guide for such issues could be a valuable exercise. Many issues surrounding DMCs in PCTs are evolving; because of increasing attention to PCTs by PCORI, the National Institutes of Health, and industry, more experience with PCTs will be gained, and additional considerations for DMCs may be identified.
Footnotes
Acknowledgements
We thank Gina Uhlenbrauck for editorial assistance in the preparation of this manuscript. Gina Uhlenbrauck is an employee of the Duke Clinical Research Institute, Durham, North Carolina, and received no compensation for her work on this article other than her usual salary.
The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health or the Patient-Centered Outcomes Research Institute (PCORI), its Board of Governors or Methodology Committee, or other participants in the National Patient-Centered Clinical Research Network (PCORnet).
Declaration of conflicting interests
Suzanne Schrandt is Deputy Director for Patient Engagement at the Patient-Centered Outcomes Research Institute (PCORI), and Maryan Zirkle is a Program Officer for the CER Methods and Infrastructure Program at PCORI. All other authors have no conflicting interests to disclose.
Funding
This work is supported by the National Institutes of Health (NIH) Common Fund, through a cooperative agreement (U54 AT007748) from the Office of Strategic Coordination within the Office of the NIH Director. Dr Culbertson’s work was supported in part by 1 U54 GM104940 from the National Institute of General Medical Sciences of the National Institutes of Health, which funds the Louisiana Clinical and Translational Science Center. Dr Ellenberg’s work was supported in part by UH2-AT007797 from the National Institutes of Health.