How to construct an optimal interim report: What the data monitoring committee does and doesn’t need to know

Introduction

Independent data monitoring committees (DMCs) have the important role of monitoring accumulating safety and efficacy data in a clinical trial to assess risk and benefit. Since the concept of having an independent advisory committee review the conduct of multi-center trials was proposed in the “Greenberg Report,” commissioned by the National Heart Institute, ¹ operating procedures for DMCs have evolved. Case studies, editorials, and meeting summaries describe the composition and operations of DMCs,^2–9 text books describe DMC membership and functions,^10–12 and policy and guidance documents prepared by the National Institutes of Health (NIH), ¹³ regulatory authorities,^14,15 and other organizations discuss how DMCs should function.^16,17 A number of papers describe best practices,^18,19 and efforts to identify and train DMC members have been emphasized as a priority.^18,20

While variations in how DMCs function persist, some operating procedures have become commonplace and almost “standard.” For example, it has become usual practice to have the independent (of the investigators conducting the trial) DMC review the protocol, data analysis plan, and DMC charter before beginning a trial. It is important during this initial review that the DMC be informed of the data collection protocol, including what data are to be collected at scheduled visits and what data are collected as soon as an event occurs. The following questions should be addressed in the initial review of the protocol and data collection plan, data analysis plan, and DMC charter: (1) What are the most important outcomes to assess participant safety and efficacy? (2) Have anticipated adverse events (AEs) been defined? (3) Are checklists being used to collect AEs or are AEs being collected using an open-ended format? (4) What happens if a participant discontinues study treatment—will they continue to be followed for safety, efficacy, and other outcomes? (5) Will primary or other outcomes be adjudicated? (6) How soon will primary event data be available? and (7) Are there monitoring guidelines for early termination due to benefit/harm or futility (if it is to be assessed) and are they clear and reasonable?

It is also common for DMC meetings (or teleconferences) to include open and closed sessions. For most trials, only the DMC members and the unblinded statistician(s) are unblinded to interim treatment results. Trial investigators and sponsor representatives remain blinded until the end of the study. However, as noted in a 2013 report from the Office of the Inspector General, attendance by NIH and Center staff during closed sessions varies. ²¹ The absence of sponsor representatives during the closed session of DMCs may be more common in industry-sponsored trials because of the Food and Drug Administration (FDA) guidance on this. ¹⁴

During the open and closed sessions, the open and closed reports are reviewed. At the end of each meeting, DMCs recommend to continue the study as planned, modify it, or terminate it. The recommendation is given to the study sponsor and lead investigators who have the responsibility of sharing it with the rest of the study team, ethics committees, and in some cases, regulatory authorities.

The deliberations and recommendations of the DMC are based on the information they receive in open and closed reports. Reports can vary substantially in quality across trials. Reports that are diffuse and ill-organized make it difficult for the DMC to identify emerging problems, or make it difficult to react in a timely manner. The key is to provide all of the information the DMC needs without burying the DMC under an avalanche of hundreds of pages of computer output. Customizing the reports to the changing needs of the DMC through the course of a trial requires experience and some background knowledge of the interventions and disease condition under study by the unblinded statistician(s) who prepare the closed reports for the DMC.

The purpose of this article is to outline the key components of open and closed reports that will enable the DMC to carry out their responsibility of safeguarding the interest of study participants and ensuring the integrity of the trial. We also discuss the important role the unblinded statistician plays in preparing closed reports for the DMC.

Open DMC report

The choice of information to include in open reports and the text accompanying tables and figures should be the responsibility of the protocol chair(s) and blinded protocol statistician (the statistician responsible for the initial design and amendments to the design). In many cases, the unblinded statistician(s) prepare the summary tables and figures for the open report that are requested by the trial leaders managing the study. However, the unblinded statisticians(s) should not participate in discussions of study design amendments once unblinded to interim data. The open report should be distributed to the DMC with the closed report at least 1 week before the meeting. Following DMC meetings, it is useful to share the open report with investigators in the trial.

Open reports do not include analyses by treatment group. They should be concise and provide information on the current status of the trial that can be shared with the investigators. Open reports should begin with a response to previous DMC recommendations and provide a brief summary of the study design, the data collection plan, and protocol history, for example, amendments.

Trial investigators should use the open report and discussions during the open session to indicate how they have responded to previous DMC recommendations, to inform the DMC of protocol amendments since the last meeting, and of results of sample size re-estimation, if performed, that can be carried out on the blinded data (e.g. using pooled event rates). The report should summarize enrollment progress (or reasons for lack of progress); data timeliness issues, especially for major endpoints; protocol deviations; event adjudication backlogs; major safety outcomes known to the investigators (e.g. AEs meeting expedited reporting criteria ²² ); any unblinding of study treatments for participants due to safety concerns or other reasons; and unanticipated problems. In general, the trial investigators should use the open report to share information that allows the DMC to judge both the strengths and limitations of the interim data.

Some open reports include detailed tables of AE frequencies for the pooled treatment groups with events categorized according to Medical Dictionary for Regulatory Activities (MedDRA) terms. This is usually not necessary since the DMC will see these summaries by treatment group in the closed report. Combining data across treatment groups could mask differences between treatment groups. The DMC focus should be on the difference in AEs between the randomized treatment groups (the data in the closed report), which if collected properly, will provide an unbiased estimate of AEs due to the study treatment. Because of this, in some trials, for regulatory purposes in a clinical development program, the DMC may also be asked by the sponsor to assess whether aggregate data by treatment group indicate that certain AEs (e.g. those related to the underlying disease condition) are occurring more frequently in the treatment as compared to the control group.^22,23

A few tables and figures summarizing baseline characteristics should be included in the open report. It is not necessary to include many pages of baseline data. The trial investigators can review baseline data in more detail and prepare reports of baseline cross-sectional analyses while the trial is ongoing. The focus of the baseline tables should be to describe the demographic diversity of the population being enrolled and key prognostic variables.

It is important that no information about emerging trends in the treatment results can be derived from the open report or be discussed during the open session of the DMC meeting. For some studies, the total number of primary events (all treatment groups combined) is included in the open report. This may be problematic, however, because it could lead to guesses in trends based on historical data. If pooled event counts are needed for sample size re-estimation or fiscal planning, it is usually best to restrict this information to the study leadership responsible for these tasks.

The open report should also be used to inform the DMC of new information from other studies that could impact trial conduct. This external information could be new safety data for the treatment under study from another trial, or the results of another trial of a similar, or the same, treatment. The external information could also be the findings from an observational study, or even pre-clinical studies. For example, recently, scientists have debated whether data from mice which indicated no benefit of stem cells should have been shared with patients enrolling in a clinical trial.^24–28

As part of the informed consent process, trial participants must be told that they will be informed of significant new findings that develop during the course of their trial participation that may relate to their willingness to continue in the trial (Code of US Federal Regulations Part 46, Subpart A, Section 46). While the DMC should be informed of new information because it may lead to the request of additional analyses of the unblinded statistician, the responsibility for determining the response to the external information is typically with the sponsor and investigators. In part, this is due to the fact that the DMC reviews the unblinded treatment data, and their response to external information could reveal trends in blinded information. This caution is mentioned in the FDA guidance on DMCs. ¹⁴

Some examples of trials that had to react to external information are described below.

Closed DMC report

Closed reports are prepared by the unblinded statistician(s) for the trial. For major trials, this may involve more than one statistician. Closed reports for the DMC are usually by coded treatment group (A and B for two groups) and the DMC is told what A and B are. The codes are provided to the DMC under separate cover. Codes are used to avoid unblinding someone who inadvertently sees the closed report. Typically, the closed report contains a core set of tables and figures that are based on the data analysis plan. These tables and figures may be expanded upon by the unblinded statisticians as the trial matures and in response to questions from the DMC.

Closed reports should begin with a brief summary of the monitoring guidelines. The closed report should include a written summary with references to key tables and figures. This summary should also specify the file creation or “freeze” date and cut-off date used for censoring in time-to-event analyses. The cut-off date may precede the file creation date by several weeks to ensure the inclusion of most reported events. For example, data from previous trials or from the current trial as it matures can be used to determine the needed lag time between the cut-off and freeze dates using the distribution of the length of time between when a study endpoint occurs (e.g. date of initial medical workup) and when it is available for analysis. In the case of emerging trends in the treatment difference, it may be important to update the event counts for each treatment group at the time of the DMC review. This may be challenging if the unblinded statistician(s) do not have direct access to the database.

The closed report should include tables and figures that assess whether treatment groups are balanced with respect to key baseline characteristics, the completeness of follow-up, adherence to study treatment or strategy (e.g. sequence of treatments), completeness of event adjudication (if relevant), major safety and efficacy outcomes, and findings for key subgroups to assess the consistency of benefits and harms. If the primary outcome is a composite endpoint, components of the composite should be summarized.

When the primary outcomes are clinical events, it is helpful to not only summarize the current event counts and the treatment difference but also the monitoring history—the number of primary events reported at previous DMC meetings by treatment group, as well as the number of events in the current database that occurred before the cut-off dates used for previous DMC reports. Large differences in these numbers may reflect a lag in reporting or adjudication, and this variability may prove useful to the DMC to judge the stability of event counts at an interim analysis when early termination is being considered.

Tables and figures should be clearly labeled and annotated to state statistical methods used, and what the numbers mean (e.g. whether the counts are numbers of events, or numbers of participants with at least one event). Point estimates should be accompanied by estimates of variability. For example, when reporting treatment differences in clinical events, in addition to event counts and denominators, rates, hazard ratios, and p-values, accumulated person-years (denominator for rate estimates) as well as confidence intervals for the hazard ratios should be provided. Similarly, when summarizing time-to-event data by Kaplan–Meier curves for cumulative probabilities of events over time, the size of the risk sets should be indicated, as well as pointwise confidence intervals at key time points (or confidence bands) for the estimated curves.

When constructing summary tables, it is important to distinguish between information that is not yet available and information that is missing and will not be recovered. For example, if the focus of an analysis is on missed follow-up visits, and the number and percentage of participants who did not attend the visit is presented, the correct denominator for the percentage would be the number of participants who should have attended the visit by the cut-off date of the report, not all randomized participants. If the focus is the proportion of participants who experienced an event by a given time point, the denominator should be the number of participants at risk for the event at the time of the report (e.g. those who reported the event or reached the time point without an event), not the number of participants randomized. In each case, the denominator should be provided along with the numerator and percentage.

Often safety summaries are given according to MedDRA preferred terms. There are 22,000 preferred terms. Such reports might be given for all AEs irrespective of severity, AEs that led to treatment discontinuation, serious AEs, and AEs related to treatment. Many preferred terms will have zero or small counts of patients. Often, it is more helpful to summarize higher level MedDRA terms and provide more detailed lower level terms when event numbers increase, or a clear signal begins to emerge. It is easy to “miss the forest for the trees” if the presentation of safety summaries is not carefully considered.

Safety analyses should be planned at the design stage. Following are some recommendations to consider: (1) identify targeted, clinically relevant AEs and consider using checklists to collect them. If open-ended questions are used and AEs are coded using MedDRA, consider using higher level terms or standardized MedDRA queries which combine different preferred terms; (2) define an event hierarchy to evaluate safety: (a) death; (b) death or serious AE; (c) death, serious AE, or AE resulting in discontinuation of study treatment, and show time to event curves for each of these outcomes; (3) avoid tabulations based on the investigator’s assessment of relationship to treatment because it is often unreliable;^39–41 and (4) give p-values, not for making categorical decisions (other factors have to be considered), but to allow quick screening of many lengthy tables for differences that should be re-examined in future DMC reports. The screening of many AE differences in a closed report can also be facilitated using fonts or color to highlight differences.

In some trials, major, clinically relevant, safety outcomes are pre-specified (e.g. bleeding events for studies of anticoagulant therapy). For these outcomes, treatment differences should be summarized in a manner which is parallel to that for major efficacy outcomes (number of patients with an event, rates, hazard ratios, confidence intervals, p-values, time-to-event curves, and subgroup analyses) to assess risk/benefit.

In some cases (e.g. when a standardized MedDRA query does not exist), it may be necessary to have the protocol team or a medical monitor who are blinded to treatment group combine MedDRA preferred terms for event categorization.

It is often helpful to include listings of selected events (e.g. events that required expedited reporting) in an appendix to the closed report. The events and information to include in the listing should be planned at the beginning of the study and discussed with the DMC.

If laboratory data are being collected at scheduled follow-up visits to monitor patient safety (e.g. increase in creatinine or liver function tests), the raw laboratory data should be summarized in addition to toxicity grades or MedDRA codes for laboratory abnormalities that are classified as AEs based on the judgment of the investigator.

For laboratory data collected at scheduled follow-up visits, both summaries of categorical shifts based on reference ranges or severity grades and summaries of quantitative changes should be provided. Reference ranges are often used in the protocol to define entry criteria, criteria for stopping study treatment, and as evidence of severe risk of organ injury (e.g. drug-induced liver disease and Hy’s law ⁴² ). Thus, increases in grade of severity (categorical summaries) can be useful even though they result in some loss of information compared to quantitative summaries. Another limitation of the categorical summaries is that for some populations, normal ranges may not be available or not be applicable. In studies that use multiple laboratories, normal ranges may vary among laboratories.

For some studies (particularly smaller trials), graphical summaries for each patient or spaghetti plots depicting data for all the patients can be informative in illustrating whether laboratory changes are persistent, time-limited, or worsen with time.

Key role of the unblinded statistician(s)

A number of authors have addressed the independence and role of the statistician(s) who prepares closed DMC reports.^6,43–47 No matter whether the unblinded statistician works for the sponsor, works in the same academic unit as the blinded protocol statistician, works for a contract research organization, or works in an independent data analysis center in an academic unit, they should be knowledgeable about the protocol, the data collection plan, and steps being taken in the trial to ensure complete, high-quality data. Too often unblinded statisticians cannot address questions raised by the DMC because they are either not familiar with the protocol or they are simply generating pre-programmed, “validated” tables and figures for which the computer code was written by someone else.

Ideally, the unblinded statistician has subject knowledge relevant to the trial beyond the statistical methodology, to better anticipate the needs of the DMC when safety or efficacy signals arise, and to pro-actively carry out additional analyses. This is important in order to ensure a timely reaction by the DMC to address safety concerns. At the very least, the unblinded statistician must be capable of writing a brief description of methods used and of annotating tables and figures in the closed report, and be able to carry out analyses the DMC requests in a timely manner. It should be possible for the unblinded statistician to perform analyses without the knowledge of the study investigators and sponsor.

Summary

Open reports should be concise and informative. They should address issues concerning trial conduct. If enrollment is lagging, data collection of major events is not timely, or follow-up is poor, the DMC will want to know what is being done about it. Open reports and the discussions during the open session should also be used to inform the DMC of relevant external data. No data in the open report or discussions during the open session should reveal trends in treatment differences.

Closed reports should include written summaries that point the DMC to key analyses. Methods for collecting major outcome data should be summarized and the tables and figures should be accompanied with annotations. The closed report should allow the DMC to assess the risk/benefit of the study treatments as well as the integrity of the data, including completeness and timeliness, used in the interim analyses. Prior to beginning the trial, it is important for the DMC to discuss the nature and timing of efficacy and safety data that the DMC will be provided in closed reports with the investigators and sponsor. In some cases, sponsors are reluctant to provide interim data on efficacy for fear that regulators will require a statistical penalty. It is not possible for the DMC, however, to weigh benefits and risks at interim analyses without summaries of both efficacy and safety data. Thus, if this is a concern, it should be addressed in the data analysis plan (e.g. with methods for controlling type 1 error, see Chapter 8 of Ellenberg et al.10) and DMC charter.

The unblinded statistician has a key role in ensuring that the closed reports for the DMC are complete, timely, and easily understood. This does not usually require that reports are hundreds of pages long. The unblinded statistician must understand the protocol and data collection plan, be capable of anticipating questions from the DMC, and responding to requests for additional analyses.