Abstract
Background:
Organizations presenting reports to independent data monitoring committees (IDMCs) should present data in a way that facilitates the ability of the IDMC to make informed judgments about the trial.
Methods:
This paper reviews reports to IDMCs and suggests approaches an independent statistical reporting group (ISRG) might take to prepare clear, complete, and comprehensible reports.
Results:
Sensible reporting by an ISRG and informed decision making by an IDMC require a productive partnership between the quantitative and clinical disciplines involved in a clinical trial. IDMC reports differ in structure and purpose from clinical study reports that summarize data at the end of a trial. The ISRG must have intellectual independence, recognizing that although the sponsor may be paying the bills, the ISRG is responsible to the IDMC. Ideally, it should have access to all data from the trial and should be capable of responding to requests from the IDMC without the sponsor’s specific permission. The ISRG and sponsor must understand the differences between clean data at the end of the trial and data collected during the trial. To perform its role most effectively, the ISRG must collaborate with sponsor and IDMC clinicians to become conversant with the disease area, the product’s mechanism of action, and the clinical relevance of important outcome measures.
Conclusions:
An IDMC is best served by an independent ISRG that will prepare clear, complete, and comprehensible reports. Given the complexities of interim data and IDMC requirements, the ISRG must be an active and informed participant in the monitoring process.
Introduction
The explosive growth of independent data monitoring committees (IDMCs) for clinical trials from IDMCs’ early inception 1 to the relevant regulatory guidances 2 to texts on the subject 3,4 to today, we contend, has overshot the expertise of many organizations who prepare and present reports to IDMC members. We call such a reporting organization an independent statistical reporting group (ISRG). This paper discusses how an ISRG can most effectively present data to an IDMC in a manner that allows the IDMC’s members to make informed judgments about an ongoing trial. This paper does not discuss detailed recommendations concerning the types of presentations or the specific statistical analyses to be used. Though these are important considerations, we have chosen to focus more generally on the need for interpretable reports, which require the ISRG to understand the clinical context of the study and the nature of the data it is summarizing.
To set the stage for the rest of this paper, we briefly describe the roles and functions of the ISRG and IDMC in a randomized clinical trial. While we focus here on industry-sponsored phase 3 trials, our thoughts apply in somewhat modified form to other types of sponsors and to other kinds of trials.
Independent Statistical Reporting Group
In our ideal model, the ISRG operates independently of the trial sponsor and of the contract research organization (CRO) or the academic research organization (ARO) managing the trial and the data. Other authors have discussed the advantages and importance of such independent operation. 5 –8 Of course, no ISRG is truly “independent”; it depends on the sponsor both for financial support and for the data. Moreover, it has frequent interactions with clinical and quantitative teams involved in the trial. In some trials, a sponsor, a CRO, or an ARO may use an appropriately firewalled internal group in place of a separate ISRG to provide reports to the IDMC. In any case, the independence comes from an intellectual and moral conviction that the presenting group must be beholden primarily to the safety of the study participants and to the integrity of the trial. The ISRG answers primarily to the IDMC, not to the sponsor or the investigators. Consequently, the ISRG must attempt to respond to the IDMC’s questions and requests. An ISRG’s answer to a reasonable IDMC request must not be “That is not in our contract.”
The ISRG’s role, which is fundamentally cross-disciplinary, lies at the intersection of quantitative and clinical sciences. In our experience, an ISRG team is most effective when it includes individuals with a mixture of statistical and programming expertise who work in close collaboration with each other and who are willing to develop a basic understanding of the biology of the trial about which they are reporting; doing so will often require productive interaction between the ISRG, IDMC clinicians, and the sponsor’s quantitative and clinical scientists involved in the trial.
Role and Function of an IDMC
Others describe the operation of IDMCs in detail. 3,4,8 –11 We use the term IDMC, but these committees may go by other names (eg, data monitoring committees, data safety monitoring boards or committees). Most of this paper considers trials where the interval between randomization of the first participant to the last visit of the last participant in the trial is at least 2 years. For such a trial, the IDMC is responsible for reviewing unmasked (usually called “unblinded”) data periodically during the trial to ensure that the participants are not experiencing unacceptable levels of harm and to confirm that the trial is proceeding in a manner that is likely to lead to interpretable results. 12,13 The IDMC may recommend stopping or modifying the trial if the data are showing harm or unequivocal evidence of benefit (or if the trial is highly unlikely to meet its scientific objectives). In order to discharge its duties, the IDMC needs reports that are clear, complete, and comprehensible.
The typical IDMC meeting has 3 types of sessions: open, closed, and executive. In the open session, the IDMC and the ISRG meet with the sponsor and often the academic study chair who summarize recruitment, adherence, and follow-up. The ISRG may prepare an “open report” with selected presentations pooled across all treatment arms to maintain the masking of treatment assignment. During the closed session, attended by the IDMC and the ISRG only, the IDMC reviews the “closed report” the ISRG has prepared. This report, which presents analyses on safety (and often on efficacy) by treatment group, forms the basis for the recommendations the IDMC will make to the sponsor and Executive Steering Committee. Some IDMCs also hold executive sessions attended only by members of the IDMC.
Consequences of Badly Prepared Reports
The closed report is the most important source of information the IDMC has. Unfortunately, many reports are unclear and unfocused. They are frequently disorganized and replete with errors, internal inconsistencies, and invalid (or badly constructed) statistics. A poorly prepared IDMC report may have dire consequences for the trial. For reasons of confidentiality, we are not divulging the names of the 3 trials described below. In one multiyear trial with several thousand participants, the IDMC received reports that throughout the trial incorrectly used dummy, rather than actual, treatment codes in the by-treatment group presentations! The IDMC observed no evidence of benefit (of course, because the reported treatment codes were random). Noting no safety concern (again, because the treatment codes were random) and being interested in whether the study would eventually show benefit of the treatment, the IDMC did not recommend stopping the trial for futility. Fortunately, at the end of the study, the final analysis, using the correct treatment codes, established both the benefit and safety of the experimental drug. In a large double-blind trial, the IDMC was dismayed to receive a closed report of over 25,000 pages. The members, stating they could not responsibly monitor the trial under these conditions, threatened to resign en masse unless the sponsor changed the ISRG. The new ISRG produced a report of roughly 100 pages that clearly and comprehensively summarized the issues important to the IDMC. An IDMC that was monitoring a serious disease was concerned that the number of reported deaths differed from table to table; some tables showed more deaths in the active group than in the control group and other tables showed the opposite. After several meetings with closed reports containing ongoing inconsistencies and with no clarification or explanation from the ISRG, the IDMC recommended the sponsor stop the trial because it was impossible to determine if the trial participants were experiencing an unacceptably high level of risk.
General Principles Guiding Production of IDMC Reports
Differences Between IDMC and Clinical Study Reports
A clinical study report (CSR) summarizes the final results of a trial; an IDMC report is designed to give the IDMC tools for making recommendations during the course of the trial. In its capacity of ensuring participant safety and trial integrity in the context of incomplete and inconsistent data, the IDMC asks different clinical questions from those asked at the end of the trial. Therefore, the IDMC’s reports must differ from reports to investigators summarizing the results of a trial.
Customized Programs for IDMC Reports
At the end of the trial, analysis programs use final, locked data sets; IDMC reports are based on interim data that are incomplete and in flux. Programs designed for producing final study reports may crash if applied to interim data because the programs do not consider certain scenarios, or they may exclude or misrepresent data because records have not yet been cleaned. Final presentations are fully prespecified; IDMC reports also have prespecified presentations, but the planned presentations often change over the course of a trial (sometimes considerably) depending on the IDMC’s evolving concerns as well as the emerging available data. For these and other reasons, IDMC reports require customized computer programs for analysis and display.
Statisticians’ Roles in Interim and Final Reports
The ISRG’s role in interim reporting differs from the study statisticians’ role in final study reporting. The study statisticians are typically most involved in the trial before it starts—collaborating with other quantitative and clinical scientists to design the study, define outcome measures, and determine the appropriate statistical methods and analyses—and again at the end when the analyses of the final data begin. The main activity of the ISRG, on the other hand, occurs during the study. The ISRG conducts planned analyses but also designs and performs statistically appropriate and relevant unplanned analyses based on IDMC requests. 2 The ISRG’s contract with the sponsor must allow access to all relevant data and modification of reports during the trial without the requirement to tell the sponsor what the ISRG is doing. Such an arrangement requires the ISRG and the sponsor to trust each other. 14 It also requires the ISRG to have access to adequate statistical and programming expertise, to develop an understanding from the outset of the biological basis and relevant clinical issues for the trial, and to engage in an ongoing collaboration with IDMC members to meet the IDMC’s evolving reporting needs.
Scientific Collaboration Between ISRG, IDMC, and Sponsor
To prepare a clinically relevant and useful report, an ISRG must understand important clinical aspects of the trial on which it is reporting. This will require the ISRG to review relevant clinical and quantitative documentation (such as the study protocol, investigator’s brochure, annotated CRF forms, and the statistical analysis plan) and productive interactions with other quantitative and clinical sponsor scientists involved in the trial. Ideally, this includes extensive interactions prior to the ISRG’s receiving unmasked data, when discussion can proceed most freely. Ongoing interactions between the ISRG and sponsor scientists to address important scientific issues should be encouraged, provided the ISRG is cautious in those interactions to protect the masking and integrity of the trial. In addition, regular communication between the ISRG and the clinical members of the IDMC throughout the trial is invaluable. This is particularly true so that the ISRG can modify its presentations in response to an emerging safety concern with clinical insight and understanding.
Facilitating the Review of IDMC Reports
Characteristics of an Ideal IDMC Report
An IDMC report must be comprehensive and comprehensible. It must include all potentially relevant information so that the IDMC has full access to what it needs to monitor the trial. However, a dogged pursuit of comprehensiveness must not come at the cost of comprehensibility. Many ISRGs include so much information in so disorganized a manner that the IDMC is overwhelmed with unnecessary and irrelevant detail; even well-organized minutiae can jeopardize comprehensibility if high-level summaries are lacking. The IDMC report must facilitate efficient review of comprehensive data through a well-designed report structure and thoughtful organization of analyses.
Structure of the Report
To ensure efficient review, the ISRG should structure the report sensibly, highlighting the most relevant material without biasing the interpretation of the data. The report should use a “top-down” approach combining summaries that the IDMC reviews carefully with additional detail available for consultation only on an as-needed basis. For example, presentations for many types of data (in particular, adverse event [AE] and laboratory data) can begin with a high-level overview and provide successively more detailed summaries, such as figures or tables, and then listings. If these detailed summaries present similar data in similar formats (eg, a standard presentation format used consistently for each laboratory analyte), the IDMC can review many analyses quickly (see Figure 1). Appendices can include lengthy tables and listings so the IDMC can navigate the report without getting lost or too bored to read the important material. We have provided a high-level overview of suggested organization and content for typical reports along with some specific questions the presentations should seek to answer (see Table 1).
Sample laboratory figure. A standard, 1-page graphical presentation for a single laboratory analyte (simulated data). The presentation provides detailed by-treatment comparisons of measurements at scheduled visits (top panel), change from baseline (bottom panel), and percentages of participants with abnormal values (middle panel). Once an IDMC member becomes familiar with this presentation, analyses of dozens of analytes can be quickly reviewed. IDMC, independent data monitoring committee.
Sample Content and Organization of Open and Closed IDMC Reports.
Abbreviations: AE, adverse event; ECG, electrocardiogram. Note: Open reports do not present any data by treatment group.
The overall document structure of the report is important. Some ISRGs send the IDMC an electronic zipped file containing thousands of pages of data distributed across hundreds of files with uninformative names (eg, “table 1a.rtf,” “figure 1.rtf”; yes, we have seen this all too often!). The ISRG should combine the presentations into a single document with a table of contents, bookmarks or tabs, page numbers (believe it or not, many reports lack page numbers, literally preventing the IDMC from being “on the same page” when they discuss the report), and relevant cross-references in the text.
Contextual Information
Providing the IDMC with a brief summary of the protocol at the beginning of each report helps IDMC members quickly remind themselves of the study design and objectives. Making available the trial’s model informed consent document allows the IDMC members to confirm that participants have been adequately warned about potential harms.
The report should include text that identifies clearly what the report includes and that explains analysis conventions and other nonobvious issues. The text should neither provide interpretations of the data nor guide the IDMC to conclusions. An executive summary table at the beginning of the report can provide the IDMC a “bird’s-eye view” of the report’s contents (see Table 2). 15 Including the summary from the previous meeting reminds the IDMC of the data it has previously reviewed.
Sample Executive Summary.
Abbreviations: CV, cardiovascular; MACE, major adverse cardiovascular event; MI, myocardial infarction; SAE, serious adverse event.
Identification of Treatment Groups
Our view, which is shared by regulators 2 and many experts, 12,13 is that both the IDMC and the ISRG should have unfettered access to the treatment codes and should know the actual treatment assignments in their deliberations about the data. Only then can the IDMC make rational recommendations. (In many closed reports, randomization allocation ratio, patterns of AEs, or laboratory data may unmask the IDMC anyhow.) We recommend that the ISRG use semimasked codes, such as A and B, in closed reports and provide the actual unmasked correspondence between codes and treatment group assignments in a separate document. Some of us have found that semimasked mnemonics rather than letter codes are more memorable for reviews over time and less confusing during discussion. For example, the IDMC reports for monitoring the EXPEDITION trial used “Lewis” and “Clark” to designate the treatment groups. 13,16
Use of Appropriate Displays
Reports will generally contain a mixture of graphs and tables, balancing the communicative value of the former against the detailed information of the latter. For the first few reports, especially when recruitment is slow, listings may be more informative than tables or figures. Later in the life of the trial, the reports may transition to figures and tables, and the listings they replace may be moved to an appendix or removed entirely. Tables replete with missing data or multiple rows of zeros are distracting; graphs that hide what is missing can be deceptive.
Sponsors and ISRGs often produce presentations that follow a standard strict template. While such a template can lead to efficient programming, the ISRG must be flexible so as not to hinder the IDMC’s ability to review the document. For example, if an AE table cannot include all treatment groups and toxicity grade columns on one page, the table should be created thoughtfully (eg, not break pages within a treatment group requiring the reviewer to flip back and forth across multiple pages to interpret the data). The ISRG should be open to consider different approaches (eg, figures vs tables) depending on IDMC preferences and/or accumulating data.
Avoidance of Unnecessary Detail
A particularly insidious source of excess detail can arise when the desire to be maximally responsive to the IDMC is combined with an insistence that the full contents of the IDMC report be specified in advance. From time to time, the IDMC may find it necessary to review certain analyses by baseline subgroup (such as age, gender, race, relevant medical history, and so on), by clinical site, or even at the level of an individual patient in order to evaluate more clearly an emerging, unanticipated safety concern. Such supplementary analyses may be critical in informing the IDMC’s decision, but the analytical requirements are driven by emerging data and cannot generally be anticipated in advance. The only way to provide such analyses in the context of a report with fully prespecified content is by including a tremendous number of additional analyses and listings in the hope that the answer to any conceivable IDMC question will be found, like the proverbial needle in a haystack of worthless and unnecessary detail. Potential technical solutions, like “dynamic documents” or clinician-friendly data visualization systems (whether standalone or web-based), offer an intriguing approach to this problem, but not one that has been extensively explored in the context of IDMC reporting. As long as IDMC reports remain largely static documents, the unintended consequence of including all possible analysis may be a huge report that tries to serve all purposes and ends up serving none.
Effective and efficient responsiveness to IDMC reporting needs is best addressed by an ISRG that combines expertise in statistics and programming, collaborates with sponsor and IDMC clinicians to develop an adequate clinical understanding of the trial, and maintains flexibility to modify the IDMC report content on an ongoing basis to respond to requests from the IDMC as they arise. In particular, an effective ISRG can bring its quantitative expertise and its clinical understanding to bear in preparing highly effective displays that provide detailed, clinically relevant information on individual patients far more effectively than lengthy listings (see Figure 2).
Example of a patient-profile plot of liver function tests. An example of a highly effective, targeted display to replace a lengthy detailed listing (mock data). Based on its understanding of the clinical hallmarks of potential drug-induced liver injury and its experience presenting analyses of liver function tests (LFTs) to IDMC clinicians, an experienced ISRG will recognize the value of selecting those patients experiencing simultaneous elevations of AST/ALT and bilirubin and presenting, for each such patient, a detailed, longitudinal display of LFT data normalized to a “multiples of the upper limit of normal” scale, supplemented by dosing, adverse event (particularly hepatic event), and concomitant medication information (not shown). More generally, similar displays can be constructed as more effective alternatives to detailed listings to address trial-specific clinical concerns, guided by the clinical insight of the IDMC members. Developing such displays requires collaborative statistical and programming expertise within the ISRG. Abbreviations: ALT, alanine transferase; AST, aspartate transferase; IDMC, independent data monitoring committee; ISRG, independent statistical reporting group.
Working with Interim Data
Interim vs Final Analyses
To understand the unique challenges posed by interim data, consider the nature of a typical final analysis. Data used for final analysis are usually housed in a single, locked database. Database structure and checks have been finalized; the database is complete and fully populated. All queries have been resolved, AEs reconciled, and endpoints adjudicated. Neither data elements nor database structure will change from one run of the computer programs to the next, and if a particular program fails to anticipate an unexpected condition or an anomalous data item that is not actually present in the final data, no harm is done. Interim data and their analysis present a rather different set of problems, requiring analysts to pay careful attention to such issues as the quality, quantity, and completeness of data.
Acquisition of Data
Interim data in clinical trials typically come from multiple sources. In a masked trial, the clinical database snapshot and the randomization list often come from distinct groups. Frequently, a safety (or pharmacovigilance) department provides timely information on serious adverse events (SAEs). Central laboratories may provide laboratory and electrocardiogram data that will be merged into the database at the end of the trial. (This will certainly be the case for a subset of important laboratory parameters if they have been identified as potentially unmasking.) Many trials include other sources of data, such as imaging data or nonstandard laboratory assays available only through direct transfer from the vendor while the trial is ongoing.
Irrespective of the number of data sources, interim data will also by their nature involve multiple snapshots transferred over time. Problems with data come and go; the data structure may change in later transfers; inconsistent or erroneous data items may be repeatedly introduced and then resolved. When multiple transfers combine with multiple sources, inconsistencies related to data vintage are unavoidable (eg, an SAE reported by the Safety Department with no corresponding event record in the clinical trial database; postbaseline laboratory data available on participants with no record of randomization).
In many trials, the IDMC’s first formal data review meeting is scheduled when some significant portion of data is expected to be available (eg, “data through week 8 for the first 30 randomized participants”). In such a case, providing the IDMC with 1 or more highly abbreviated “safety report” in the months prior to that first meeting can facilitate later review. Such a report might include only basic accrual and AE information on a handful of participants. Providing this type of concrete deliverable to the IDMC allows thorough testing of many aspects of the data transfer mechanics in a manner far preferable to testing in the midst of preparing a full report for a scheduled IDMC meeting.
The ISRG should receive frequent transfers of the entire database, rather than data transfers just before meetings (even when preceded by an additional “test transfer”). Monthly transfers provide a stream of “tests” using real data, allowing timely identification of problems with the data. Regular transfers of the entire database have the added benefit that relatively up-to-date data are always available to support ad hoc requests from the IDMC. Such requests might be made on the basis of an externally identified safety concern raised by the sponsor or one raised—in confidence—by the IDMC in closed session. Providing the reporting statistical group with maximum flexibility in responding to such requests using timely data protects participant safety while maintaining the integrity of the trial.
Ideally, transfers should include all available data rather than data limited to the analyses in the original plan for the closed report. We recommend including data related to efficacy endpoints, regardless of whether or not formal efficacy analyses are planned for the upcoming review. When a study has emerging safety issues, the IDMC may need to weigh the risk and benefits of the study treatment in deciding whether to recommend stopping the trial. 12 Concerns related to control of overall type I error rate may be addressed using several well-known standard approaches. 17 –19
Challenges Regarding Availability of Data
Interim data confront the ISRG with a trade-off between quantity and quality. Interim trial data are fundamentally incomplete because the trial is still in progress: data that have not yet been collected will not appear in an interim database snapshot. This self-evident statement has some less evident consequences. First, quantification of data availability and completeness—usually a secondary issue in the final analysis—becomes a central question for the IDMC in interpreting interim analyses. In addition, important definitions and analyses specified in the final analysis plan (eg, definitions of “full treatment adherence”) might make sense in the context of evaluating final trial data but be inadequate or nonsensical when applied to interim data. Moreover, certain crucial aspects of preparing data, such as coding AEs or adjudicating endpoints, may lag behind initial investigator reports, but the latter cannot simply be ignored if the IDMC is to discharge its responsibility to review all relevant data. The ISRG’s combined statistical and programming expertise and its ongoing collaboration with clinical scientists to fully understand the relevant clinical considerations is important in identifying and addressing these issues related to interim analysis.
Our general approach is to analyze everything of importance currently in the database while clearly indicating what data are not available. The report should include explicit description of availability of data. For a given category of data, the report should provide information on the proportion of randomized participants (1) who have data in the database; (2) whose data are not present for a legitimate reason (eg, they have withdrawn or died); and (3) who do not have data but ought to, given expectations about timing of data collection. Uncoded and unadjudicated data should be analyzed and displayed, either in separate categories (eg, a separate “uncoded” AE category alongside the coded system organ classes on a summary page) or incorporated into combined analyses (eg, a “best available” or “unrefuted” analysis of investigator-reported events combining positively adjudicated events with potential events not yet adjudicated). If nothing else, the IDMC can act on uncoded or inadequately adjudicated data by recommending redoubled efforts to code and adjudicate.
IDMCs often struggle with the tension between the currency and correctness of the data. Many sponsors send the IDMC fully clean interim data at the expense of timely information. An easy way to aggravate an IDMC is to present data that are unacceptably “stale,” for that forces the IDMC to make recommendations on the basis of trends that may have already changed. The problem is exacerbated when a clinical endpoint committee is adjudicating the endpoints. In that case, the endpoint data may be delayed for many months. Often, the delay is longest for complicated cases. We prefer to use current data coupled with defensive programming to deal sensibly with potentially dirty data.
Challenges Regarding Quality of Data
Using “dirty” data, that is, data that have not been fully queried and confirmed as correct, means dealing with incorrect and inconsistent data items (eg, heights of 150 inches, dead individuals attending follow-up visits from beyond the grave, a potassium of 140 mEq/L coupled with a sodium of 4.5 mEq/L). On the other hand, using only confirmed “clean data” risks removing potentially important or useful information from the analysis. We prefer to receive all available data, clean and dirty alike, and handle dirty data with defensive computer programs.
Dealing with dirty data requires an interdisciplinary approach that combines programming and statistical expertise with adequate consideration of clinical issues. We recommend adopting methods that are clinically sensible, easily defined, and simple to implement. Ease of definition means the ISRG can effectively communicate its conventions to the IDMC so that members may assess the implications to the analysis they are reviewing. Simple implementation avoids wasted analyst effort in addressing an issue that the next month’s data transfer may adequately resolve. General approaches to dirty data include removing or assuming an appropriate unit conversion for implausible or impossible data, or—for potentially contradictory sources of information on important measures—using a sequential or hierarchical definition, such as calculating a death date based on the adjudicated date if available, falling back on dates of potential death endpoints, death dates from the SAE data set, or AEs in the clinical database coded as fatal. Robust statistical techniques, such as nonparametric tests or trimmed summary statistics, may produce more interpretable summaries than methods designed for fully clean data. Presentations should avoid paying undue attention to minima, maxima, or other outliers, particularly for data items like baseline characteristics, where an outlier is unlikely to represent a potential safety issue. Programs can be written so that people are not resurrected after they die, and sodium and potassium records are switched when they are clearly reversed.
In the specific case of presenting results for the “same” data taken from multiple sources (eg, SAE data reported from the Safety Department and an AE marked “serious” in the clinical database), experienced ISRGs and IDMCs understand that the final analysis will reconcile these sources into a single definitive data set. For interim analyses, one obvious approach is to attempt a similar reconciliation, though dirty data can make this extremely challenging, generally requiring additional attention with each new transfer. A streamlined approach is to choose the most definitive source and ignore the others. Alternatively, particularly if the sources appear to differ materially (eg, records in one source not appearing in another), it may be best to report both sources separately with clear identification. If the method of presentation is similar for each source, the ISRG can quickly review the additional analyses and judge which one presents the clearest and most comprehensive picture of the actual available data.
Another challenge in working with interim data is that data collection instruments may be amended during the course of the trial. For example, one trial initially had 2 age categories: 60–74 years and ≥ 75 years. After a protocol amendment, the case report forms (CRFs) added a category of 50–59 years to reflect the trial’s modified inclusion criteria. Given that changes to the data collection can occur, programs should be coded defensively to identify new values that may not have been present in the data when the program was originally written. The following SAS code fragment—while correct for the initial CRF values—would produce incorrect results with the addition of the new age category: if AGECAT = ‘60-74’ then AGELT75FL = ‘Y’; else AGELT75FL = ‘N’;
Better, more defensive, code would be something such as the following: if AGECAT = ‘60-74’ then AGELT75FL = ‘Y’; else if AGECAT = ‘>=75’ then AGELT75FL = ‘N’; else put ‘NOTE: New value of AGECAT encountered in data. Amend code as appropriate:’ AGECAT=;
Programming and Validation of an IDMC Report
Who Does the Programming?
There are several ways to assign responsibility for programming of IDMC presentations.
At one extreme, the sponsor or CRO does all the programming; the ISRG attaches the treatment code to the data sets and presses the proverbial button. This method, at least initially, promises to be the least expensive and the quickest; however, a serious disadvantage is the degree to which it may compromise the independence of the ISRG (and the accuracy of the analysis!). Moreover, we have already discussed the inadequacy of fully prespecified programs to address the IDMC’s reporting needs. There is a further danger that lack of familiarity with the programs someone else has written will leave the ISRG unable to respond to even the most basic questions from the IDMC. In the end, if the ISRG must expend a significant level of effort understanding, validating, and modifying sponsor programs or even writing new ones, the cost and time efficiencies can be lost.
At the other extreme, the ISRG may write programs from scratch. This requires the ISRG to take an active role in combining statistical and programming expertise with an understanding of the clinical background of the trial, but it ensures the ISRG will understand both the underlying data and the analyses performed and will be fully prepared to answer the IDMC’s questions. While this approach has the potential to lead to duplicated effort and increased expense, it is the approach we prefer. Our experience working at both extremes has convinced us that the potential cost savings associated with sponsor-written programs is often only fully realized if the ISRG takes a completely hands-off approach to the analysis, with a corresponding negative effect on report quality.
When faced with 2 extremes, one frequently seeks a middle ground. The sponsor might provide the analysis data sets and the ISRG programs the presentations. The danger remains that the ISRG may not fully understand the data conventions used to create the datasets, but this can be mitigated by also supplying the raw data sets so the ISRG may “trust, but verify” important analysis data sets. Alternatively, the sponsor might provide reference implementations of important analyses, with the expectation that the ISRG will program them independently and use the sponsor’s programs for validation purposes, as further discussed below.
When the ISRG does all or part of the programming, the division of labor and form of communication between statisticians and programmers must be considered. All too often, specifications the statisticians write describing the analysis are passed off to a programming group with little or no ongoing interaction between the 2 parties. Statisticians expend considerable effort trying to specify exactly what they want; programmers try to produce output quickly and efficiently according to their potentially imperfect understanding of those specifications. Many important questions are never asked, resulting in misunderstandings and mistakes. The most effective ISRGs combine statistical and programming expertise in a collaborative environment, providing ample opportunity for interaction and discussion.
Validation of the Report
Although it is unrealistic to expect an IDMC report to be completely free of errors, reports must be accurate enough for the IDMC to make informed recommendations. The randomized treatment assignments and crucial analyses must be as correct as possible given the limitations of the data. The actual treatment codes are essential to the ability of an IDMC to interpret data from the study. In merging treatment codes with the database, redundancy and compulsivity are assets. The ISRG should program checks of the file with treatment codes against the master schedule; it should confirm that the merged treatment codes agree with the randomization schedule and drug kit list. The ISRG should, insofar as possible, check that the treatment is consistent with pharmacokinetic data, the known toxicity profile of the drug, development of antidrug antibodies, or unmasked narratives of SAEs. Ideally, the ISRG should audit the implementation of the randomization at least once early during recruitment to ensure that the randomization scheme is being implemented correctly.
In validating programs for an IDMC report, the ISRG should adopt a risk-based approach. All results are important, but some are more important than others. SAEs and the primary outcome should be prioritized over demographics. Complex analyses (such as a formal interim analysis or reconciliation of adjudicated and unadjudicated event data) warrant additional attention. An ISRG should have the technical ability to ensure rigorous statistical analysis.
Many sponsors and CROs permit only 1 designated ISRG statistician access to the unmasked data and the closed report. This limitation invites disaster. It is far better to have an entire ISRG team unmasked. As long as each member of the team understands and respects the importance of confidentiality, more heads are better than one. A page-by-page walk through of the draft report by members of the ISRG prior to sending it to the IDMC, preferably with an ISRG staff member who has not been intimately involved in the preparation of the report (or at least not with the particular section of the report being reviewed), gives the team the opportunity to question whether the presentations make sense, identify inconsistencies across presentations, and anticipate questions the IDMC may have. In the week between the sending of the report and the meeting itself, the ISRG may prepare answers to these anticipated questions.
Timelines for Preparing the Report
The sponsor must provide adequate time between data transfer and distribution of the report. Many sponsors suppose that loading the database and running prewritten and pretested programs should take no more than a day or two. This is certainly true, provided one does not care about the quality or correctness of the resulting analyses! Interim analysis is complex with great potential for error, and it does not easily lend itself to precise prespecification of analyses or reliance solely on pre-tested programs. Providing a high-quality, reliable closed report that adequately supports the IDMC’s decision making takes time. For small studies with simple databases where the trial has been running smoothly for many months, it may be reasonable to expect distribution of a report based on data received 1 or 2 weeks prior to sending the report. For large studies, or for studies of any size with complex databases or many data sources, and for meetings with an interim analysis of efficacy, a more reasonable expectation is to allow 3 to 5 weeks between data transfer and distribution of the report. Such a deliberate approach to producing a report should minimize mistakes and rework, which can be costly in terms of time, resources, and reasonable decision making.
Where there is concern about use of unacceptably “stale” data, the ISRG can identify several data tiers. The main transfer may be used for most presentations, supplemented by more timely transfers of important data, such as up-to-date SAE data sets. Other updates (eg, deaths and additional SAEs) can be refreshed just before the IDMC meeting and presented at the meeting itself.
Conclusions
In discharging its responsibilities to ensure an acceptable balance of risk and benefit for trial participants and to monitor the scientific integrity of the trial, an IDMC is best served by an ISRG operating independently of the trial sponsor and data management CRO/ARO. The ISRG must provide the IDMC with reports that are clear, comprehensive, and comprehensible. The nature of interim data means that reports based on fully prespecified presentations produced by fully prewritten analysis programs are unlikely to serve the IDMC well. Instead, the sponsor must trust the ISRG enough to provide it with all relevant safety and efficacy data and with the flexibility to respond to evolving data and the IDMC’s requirements. The ISRG, for its part, must be an active, informed, and effective participant in the monitoring process. To do this, it must combine the efforts of programmers and statisticians, and it must forge a partnership with the sponsor and IDMC clinical scientists involved in the trial.
Footnotes
Acknowledgment
We thank Ms Hannah Kalvin for her expert editorial assistance.
Declaration of Conflicting Interests
No potential conflicts were declared.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Kevin Buhr and Robin Bechhofer work in an academic statistical reporting group funded by contracts with multiple pharmaceutical companies for DMC-monitored clinical trials. Matthew Downs, Janelle Rhorer, and Janet Wittes work in a company that has contracts to serve as the Independent Statistical Reporting Group for DMCs for multiple pharmaceutical companies.