Abstract
Background/aims
In clinical trials, it is not unusual for errors to occur during the process of recruiting, randomising and providing treatment to participants. For example, an ineligible participant may inadvertently be randomised, a participant may be randomised in the incorrect stratum, a participant may be randomised multiple times when only a single randomisation is permitted or the incorrect treatment may inadvertently be issued to a participant at randomisation. Such errors have the potential to introduce bias into treatment effect estimates and affect the validity of the trial, yet there is little motivation for researchers to report these errors and it is unclear how often they occur. The aim of this study is to assess the prevalence of recruitment, randomisation and treatment errors and review current approaches for reporting these errors in trials published in leading medical journals.
Methods
We conducted a systematic review of individually randomised, phase III, randomised controlled trials published in New England Journal of Medicine, Lancet, Journal of the American Medical Association, Annals of Internal Medicine and British Medical Journal from January to March 2015. The number and type of recruitment, randomisation and treatment errors that were reported and how they were handled were recorded. The corresponding authors were contacted for a random sample of trials included in the review and asked to provide details on unreported errors that occurred during their trial.
Results
We identified 241 potentially eligible articles, of which 82 met the inclusion criteria and were included in the review. These trials involved a median of 24 centres and 650 participants, and 87% involved two treatment arms. Recruitment, randomisation or treatment errors were reported in 32 in 82 trials (39%) that had a median of eight errors. The most commonly reported error was ineligible participants inadvertently being randomised. No mention of recruitment, randomisation or treatment errors was found in the remaining 50 of 82 trials (61%). Based on responses from 9 of the 15 corresponding authors who were contacted regarding recruitment, randomisation and treatment errors, between 1% and 100% of the errors that occurred in their trials were reported in the trial publications.
Conclusion
Recruitment, randomisation and treatment errors are common in individually randomised, phase III trials published in leading medical journals, but reporting practices are inadequate and reporting standards are needed. We recommend researchers report all such errors that occurred during the trial and describe how they were handled in trial publications to improve transparency in reporting of clinical trials.
Keywords
Introduction
Randomisation is the unique and most critical feature of randomised controlled trials (RCTs), enabling the formation of similar treatment groups at baseline and an unbiased estimate of the treatment effect. In practice, the randomisation process typically involves a series of steps, from assessing eligibility and obtaining consent, through to performing the randomisation and providing the assigned treatment. Even in well-conducted trials, it is not uncommon for errors to occur during this process of recruiting, randomising and providing treatment to participants. For example, an ineligible participant may inadvertently be randomised, a participant may be randomised in the incorrect stratum, a participant may be randomised multiple times when only a single randomisation is permitted or the incorrect treatment may inadvertently be issued to a participant at randomisation. Such errors have the potential to undermine the benefits of randomisation, affect the validity of the trial and introduce bias into treatment effect estimates if handled incorrectly, yet detailed discussion of these specific errors in the literature remains limited.1,2
When recruitment, randomisation or treatment errors occur, it is important to report their existence and describe how they were handled in trial publications, as this allows readers to assess the extent of the problem and whether the errors were dealt with appropriately. Guidance on handling common recruitment, randomisation and treatment errors has been provided previously.1,2 However, there is little motivation for researchers to report these errors, as they are not a requirement in the widely adopted Consolidated Standards of Reporting Trials (CONSORT) statement.3,4 It is unclear how often recruitment, randomisation and treatment errors occur in trials and how they are reported in trial publications. The aim of this study is to assess the prevalence of recruitment, randomisation and treatment errors and review current approaches for reporting these errors in individually randomised, phase III RCTs published in leading medical journals. We focus on the adequacy of reporting practices in relation to the errors, rather than the adequacy of the trial conduct that may have enabled such errors to occur.
Methods
A systematic review was conducted according to a pre-specified protocol. The review was restricted to the five leading medical journals according to their 2014 impact factor (New England Journal of Medicine, Lancet, Journal of the American Medical Association, Annals of Internal Medicine, British Medical Journal), as reporting quality is expected to be high in these journals. All journals publish one issue each week, except for Annals of Internal Medicine which is published fortnightly. The search was conducted in PubMed on 1 December 2015 using the search terms (“Annals of internal medicine”[Journal] OR “BMJ”[Journal] OR “Lancet”[Journal] OR “JAMA”[Journal] OR “The New England journal of medicine”[Journal]) AND (randomized controlled trial[pt] OR controlled clinical trial[pt] OR randomized[tiab] OR randomised[tiab] OR placebo[tiab] OR clinical trials as topic [mesh: noexp] OR randomly[tiab] OR trial[ti]) AND (“2015/01/01”[PDat] : “2015/03/31”[PDat]) NOT (animals[mh] NOT humans[mh]). These search terms are identical to those specified in the sensitivity and precision-maximising version of the Cochrane highly sensitive search strategy for identifying randomised trials in MEDLINE (PubMed format) 5 with several additions. First, the search term ‘randomised[tiab]’ was added, as the Cochrane strategy only includes the American spelling ‘randomized’ and some included journals use the British spelling. Second, search terms were added for the journals of interest and the 3-month time period (January–March 2015) that was chosen both for feasibility and to allow time for the articles to be Medical Subject Heading indexed prior to conducting the search.
Articles were eligible for inclusion in the review if they reported the results of an individually randomised, phase III, RCT conducted in humans. Articles were excluded if they indicated they reported the results of a pilot, feasibility, phase I, phase II or cluster randomised trial, as these trials may substantively differ from individually randomised, phase III trials in their conduct, protocol and data monitoring procedures and may therefore have qualitatively different error rates. Short reports or articles presenting the results of multiple trials together were also excluded, as these have less space to provide details about any errors that occurred. To ensure that each trial was included only once in the review, only the first article reporting the primary trial results was eligible.
Titles and abstracts of all articles identified in the search were independently assessed for potential eligibility by two reviewers (L.N.Y. and E.D.). The full text of all potentially eligible articles was then examined by L.N.Y., with a 20% random sample independently examined by a second reviewer (B.C.K. or J.A.C.), to confirm eligibility and extract trial details for eligible trials using a purpose-specific data extraction form. The published articles and supplementary results files were included in the review process but not the trial protocols for feasibility. An initial version of the data extraction form was piloted by two reviewers (L.N.Y. and B.C.K.) on one article from each journal of interest published in 2014. Final changes were made to the form based on discussion prior to conducting the full text review. Any disagreements between reviewers were resolved by discussion.
For each eligible article, basic trial characteristics were collected, including the number of participants, centres and treatment arms, the methods used to generate the randomisation sequence and perform the randomisation and whether participants were allowed to be randomised multiple times. In extracting details of the recruitment, randomisation and treatment errors that were reported, four specific types of errors that have been discussed in detail previously 2 were assessed: ineligible participant randomised, participant randomised using incorrect baseline information (e.g. participant randomised in incorrect stratum), participant incorrectly randomised multiple times and participant received incorrect treatment. For the latter error, all instances of participants receiving the incorrect treatment were documented. Further details were recorded, where available, to enable the reviewer to determine whether any instances were due to errors rather than deliberate departures from the assigned treatment (non-compliance), as we were only interested in identifying errors. Free text fields were used to document any other errors that occurred during the process of recruiting, randomising and providing treatment to participants. Details on each type of error were recorded, including the number of errors that occurred, where the errors were reported and any information provided on how they were handled during the conduct of the trial or in the analysis. The number and percentage of trials reporting each type of error were calculated. The total number and prevalence of reported errors were summarised across trials by the median and range. Trials that did not mention any errors were excluded from these summaries, as it was unclear whether they had zero errors or failed to report the errors that had occurred. Prevalence was expressed as the number of errors reported per 10,000 participants randomised, since each participant provides an opportunity for an error to occur and there was considerable variation in sample size across trials.
As trial publications may not provide full details of all recruitment, randomisation and treatment errors that occurred, a random sample of corresponding authors were contacted to seek clarification regarding these errors for their trial. Specifically, we asked (a) whether all errors that occurred in the trial were correctly identified by our review and (b) for details on the number and type of errors not reported in the trial publication and hence not identified by our review. Where no response was received, a single follow-up email was sent several weeks later. Only the random sample of trials selected for duplicate review that were subsequently found to be eligible were included in this process for feasibility. The corresponding authors were advised that responses would be presented in summary format only and not associated with their individual trial.
Results
The search strategy identified 241 articles. Based on the title and abstract review, both reviewers agreed on whether the article should proceed to full text review or not for 230 of 241 articles (95%). There were 89 articles that underwent a full text review, of which 82 articles (92%) met the inclusion criteria and were included in the review (see Supplementary Material). There were 18 of 89 articles (20%) randomly selected for duplicate full text review. Both reviewers agreed on whether the article was eligible or not for 16 of 18 articles (89%), and 15 of 18 articles (83%) met the inclusion criteria and were included in the review (Figure 1).
Flow diagram.
The characteristics of the 82 trials included in the review are presented in Table 1. The majority of trials were published in either Lancet (39%) or New England Journal of Medicine (37%). The trials varied greatly in size, with a median of 24 centres and 650 participants. Most trials involved two treatment arms (87%) and had a publicly available protocol (77%). Many trials failed to clearly describe the methods used to generate the randomisation sequence (29%) or how the randomisation was performed (34%). Where randomisation was clearly described, it was most commonly performed using a web-based system (27%), and half the trials used stratified permuted blocks to generate the randomisation sequence. Only 12 of 82 trials (15%) explicitly stated that repeat randomisations were not permitted as part of the inclusion/exclusion criteria.
Characteristics of included trials.
Values are number (percentage) unless otherwise indicated.
Excludes five trials where the number of centres was not stated.
Includes three trials with technology-based randomisation processes (pager, fax, email) and three trials with physical randomisation processes (opening next available treatment pack).
Includes two trials using biased coin randomisation and four trials using a combination of stratification and minimisation.
Includes 18 trials where randomisation was stratified, but blocking was not mentioned.
One or more recruitment, randomisation or treatment errors were reported in 32 of 82 trials (39%) that had a median of eight errors reported (range 1–176) and 906 randomised participants per trial. The number of errors reported per 10,000 participants randomised varied greatly across these 32 trials, ranging from 2 to 934 (median 71). For the remaining 50 of 82 trials (61%) that included a median of 505 randomised participants, we found no mention of any errors and were unable to determine whether no errors had occurred or errors had occurred but not been reported.
The specific types of errors reported are summarised in Table 2. There were 23 of 82 trials (28%) reporting that ineligible participants had been inadvertently randomised, with a median of 12 ineligible participants randomised per trial (range 1–80). This information was typically reported in the results section or flow diagram but could also be found in a table or online supplement. While few trials explicitly stated how the ineligible participants were handled, it was possible to infer that they were excluded from the primary analysis in 12 of 23 trials (52%), and included in the primary analysis in 11 of 23 trials (48%), based on the flow diagram and statistical methods section. Only 2 of 82 trials (2%) reported that any participants were randomised using incorrect baseline information, both of which involved the use of stratified permuted blocks. In one of these trials, four errors were reported in the flow diagram and the affected participants were excluded from the primary analysis. In the other trial, eight errors were reported in the online supplementary material, and the affected participants were included in the analysis, although it was unclear whether the correct or incorrect baseline information was used in the adjusted analysis. No trial reported that a participant was randomised multiple times in error. Most of the 82 trials included in the review (n = 64, 78%) reported that one or more participants failed to receive the allocated treatment (median seven participants per trial), and this information was usually included in the flow diagram. However, only 42% (27/64) of these trials provided sufficient detail to determine whether any instances were due to errors (n = 4 trials) or other reasons, such as participant withdrawal, participant non-compliance or treatment provider recommendation (n = 23 trials). The number of participants who failed to receive the allocated treatment in the four trials where it was clearly due to an error ranged from two to eight, and in each trial the affected participants were included in their randomised groups in an intention-to-treat analysis. There were 9 of 82 trials (11%) that reported other types of errors, and these were typically identified from the flow diagram or online supplement. The most common other type of error was failure to randomise people who were eligible for the trial and had provided consent to participate.
Reporting of recruitment, randomisation and treatment errors in trial publications.
aCalculated across trials where error was reported. Range not presented for errors reported in only one trial.
bBoth trials used stratified permuted blocks.
cExcludes cases where participant received incorrect treatment due to a deliberate departure from the assigned treatment or where the reason was unclear.
Of the 15 of 82 trials (18%) included in the review where the corresponding authors were contacted regarding errors for their trial, responses were received from 9 of 15 trials (60%) that randomised a median of 640 participants. Between 1% and 100% of the errors that had occurred in these trials were reported in the trial publications. Two trials provided information on an additional 301 errors that were not reported in the trial publications: 286 ineligible participants were inadvertently randomised, 3 participants were randomised following improper consent procedures, 4 participants were possibly given the incorrect treatment at randomisation, 6 participants were definitely given the incorrect treatment at randomisation and 2 participants were randomised twice in error. No additional errors were reported for the remaining seven trials.
Discussion
Recruitment, randomisation and treatment errors were common in individually randomised, phase III RCTs published in leading medical journals in 2015, with 39% (32/82) of trials reporting at least one error. While the number of errors reported was fairly small relative to the number of participants randomised (median 71 errors per 10,000 participants randomised), we believe these errors are both common and important enough to warrant careful consideration regarding how they might be prevented, and how they should be handled during the conduct of the trial and in the analysis when they do occur. Recruitment errors may be reduced by use of a dummy enrolment run-in phase. 6 Randomisation errors may be minimised by thoroughly checking the randomisation sequence and testing the randomisation system prior to commencing the trial. 1 Methods for handling common recruitment, randomisation and treatment errors that maintain the goals of the intention-to-treat principle have been discussed in detail elsewhere. 2 Such methods should ideally be pre-specified in the trial protocol and this could be incorporated in a future SPIRIT statement.7,8
Estimating the true prevalence of recruitment, randomisation and treatment errors based on errors reported in trial publications is challenging. If the 61% (50/82) of trials included in our review that did not mention any errors in fact had no errors, then we have overestimated the median number of errors per trial. In contrast, if reporting of errors was incomplete, then we may have underestimated the median. The latter scenario is more likely, given our finding that as little as 1% of errors that occur in a trial may be reported. It is difficult to estimate the true rate of under-reporting based on this finding due to the small number of corresponding authors contacted (15), the moderate response rate (60%) and the possibility that this process did not identify all unreported errors. Despite these limitations, our survey of the corresponding authors remains valuable as it confirms previous speculation that errors occur more often than they are reported 1 and highlights the inadequacy of current reporting practices. More accurate estimates of the prevalence of recruitment, randomisation and treatment errors could be obtained from future trial publications if reporting practices improve.
We are aware of one previous review that assessed the prevalence and reporting of recruitment, randomisation and treatment errors as part of a broader review of protocol violations. 9 Among 80 cluster or individually randomised trials published in leading medical journals in 2009, 13% of trials reported ‘enrolment protocol violations’, with a median of 0.8% of participants found to be ineligible (or 80 per 10,000 randomised). This error was more common in our review, where 28% of trials reported that a median of 91 participants per 10,000 randomised were ineligible. ‘Randomisation protocol violations’ were reported in 9% of trials included in the previous review, while only 2% of trials in our review reported that participants were randomised using incorrect baseline information. ‘Study intervention protocol violations’ were reported in 21% of trials in the previous review, with a median of 1.3% of participants (or 130 per 10,000 randomised) experiencing ‘a dosing, timing or delivery error in the study intervention attributable to members of the research team’. Treatment errors were less common in our review, where only 5% of trials reported errors affecting a median of 41 participants per 10,000 randomised. Differences in findings between the reviews may reflect changes in reporting practices over time or differences in the definitions and methodology used. Our review adds to this study by considering any errors that occurred during the process of recruiting, randomising and providing treatment to participants, and providing evidence of under-reporting of these errors by contacting corresponding authors.
The potential for recruitment, randomisation and treatment errors to impact on the trial results depends on the type of error, how often it occurs and how it is handled during the conduct of the trial and in the analysis. By far the most commonly reported error found in this review was ineligible participants inadvertently being randomised. This is a serious concern, since the trial treatments under investigation may be inappropriate or harmful for ineligible participants, 2 and care must be taken if they are to be excluded from the analysis to avoid potential bias.10–12 Unfortunately, such errors are unavoidable in some scenarios, as participants may appear to be eligible based on the information available at the time of randomisation and only later be identified as ineligible. Another common but far less concerning reported error was failure to randomise people who were eligible for the trial and had provided consent to participate. These missed opportunities will lead to increased time required to meet recruitment targets and could impact on the generalisability of the trial results but will not bias treatment group comparisons. Contrary to our expectations, no trial reported randomising the same participant twice in error in the trial publication, although two occurrences were identified through contacting authors and we have experienced this error in our own trials. These repeat randomisations introduce the possibility of choosing the preferred randomised treatment for the participant, which could introduce bias. While few instances of using incorrect baseline information to perform the randomisation were reported, the implications of these errors can be substantial, particularly for dynamic allocation methods such as minimisation, as even one error will alter the probability of receiving the treatment for future participants. 1 Given that balancing variables should be controlled for in the analysis, 13 these errors raise the question of whether an adjusted analysis should be performed using the correct baseline variables or the incorrect versions used to perform the randomisation, and this issue is being investigated separately.
Our review highlights several inadequacies in current reporting practices relating to recruitment, randomisation and treatment errors. First, many trial reports failed to clearly describe the methods used to generate the randomisation sequence and perform the randomisation, making it difficult to understand the types of randomisation errors that were possible. This finding is surprising, since it is recommended that both the ‘sequence generation’ and ‘allocation concealment mechanism’ be fully described in the current CONSORT statement.3,4 The lack of detail on randomisation methods may be explained by the common practice of making trial protocols publicly available combined with the strict word limits applied by the journals. Second, few trials indicated whether repeat participation was permitted in the inclusion/exclusion criteria, and thus it was often unclear whether a repeat randomisation would be considered an error or not. Third, recruitment, randomisation and treatment errors were often poorly reported, making it difficult to identify how many errors had occurred (if any). For instance, many trials reported that some participants failed to receive the allocated treatment, as recommended in the CONSORT flow diagram,3,4 but insufficient detail was provided to determine whether any cases resulted from an error. Fourth, some trials did not report all recruitment, randomisation and treatment errors that occurred in the trial publication and hence under-reporting is a concern. Finally, it was often challenging to determine how any errors that were reported were handled during the conduct of the trial and in the analysis. This is problematic, given the potential for bias to be introduced into treatment effect estimates if certain errors are handled incorrectly. Reporting standards are needed to address these inadequacies.
Reporting recommendations
We make several recommendations for reporting recruitment, randomisation and treatment errors in trial publications that could be incorporated into formal reporting standards in future. Our recommendations are based on the findings of this review and the collective trial experience of the authorship team that has been obtained from serving as trial statisticians, investigators and members of independent data monitoring committees involved mainly with publicly funded randomised trials in health-related disciplines. Consistent with the CONSORT statement,3,4 our recommendations focus on promoting transparency in reporting, rather than how errors should be handled. Specifically, we recommend the following:
Fully describe the method of randomisation by following the reporting recommendations in the CONSORT statement.3,4
Indicate whether repeat participation in the trial was permitted in the inclusion/exclusion criteria.
Report the number and type of recruitment, randomisation and treatment errors that occurred during the trial by treatment group (where applicable), or state that no errors were identified.
Provide details on how the errors were handled during the conduct of the trial and in the analysis.
The best location in a trial manuscript to report on recruitment, randomisation and treatment errors is open to debate. For trials with few errors that are easily described, these could be included in the participant flow diagram with additional details provided in footnotes. For trials where a range of errors occurred, these may best be reported in a table. Given the strict word limits applied by many journals and the increasing use of online supplements, we suggest including such a table in a supplementary results file. An example of such a table is given for a hypothetical trial in Table 3. While the approaches used to handle the errors during the conduct of the trial and in the analysis in this hypothetical trial were chosen to minimise the potential for bias to be introduced into the treatment group comparisons, alternative approaches may also be appropriate.1,2
Example report of recruitment, randomisation and treatment errors for hypothetical trial.
Strengths and limitations
A key strength of this review is that to our knowledge, it provides the first specific assessment of the prevalence of recruitment, randomisation and treatment errors and how they are reported in trial publications. By contacting a random sample of the corresponding authors, we also provide the first formal evidence of under-reporting of these errors in trial publications. A limitation is that we did not review additional trial resources such as trial registrations, protocols or other publications reporting trial results, both for feasibility and since many readers would only access the main trial publication. Review of such additional documents may have provided further details on the randomisation methods, inclusion/exclusion criteria and any errors that occurred. A further limitation is that our assessment of the prevalence of errors and current reporting practices was based on 82 individually randomised, phase III RCTs published in leading medical journals, which are unlikely to be representative of all published RCTs. Likewise, our evidence of under-reporting of errors was based on responses received from authors of only a small sample of trials. Although the contacted trials were chosen at random and should therefore be representative of all trials included in the review, a larger study would provide a more accurate estimate of under-reporting rates.
Conclusion
Recruitment, randomisation and treatment errors are an important and common problem that has largely been overlooked in the clinical trials literature. Researchers should be aware of the possibility of these errors when designing their trials and give careful consideration to how these errors can be minimised and how they should be handled during the conduct of the trial, in the analysis and at the reporting stage. Our review indicates that reporting of recruitment, randomisation and treatment errors is currently inadequate in individually randomised, phase III RCTs published in leading medical journals and reporting standards are needed. Our recommendations for reporting these errors could be incorporated in a future update of the CONSORT statement to encourage better planning for and clearer and more complete reporting of these errors in reports of clinical trial results.
Supplemental Material
761627_supp_mat – Supplemental material for Prevalence and reporting of recruitment, randomisation and treatment errors in clinical trials: A systematic review
Supplemental material, 761627_supp_mat for Prevalence and reporting of recruitment, randomisation and treatment errors in clinical trials: A systematic review by Lisa N Yelland, Brennan C Kahan, Elsa Dent, Katherine J Lee, Merryn Voysey, Andrew B Forbes and Jonathan A Cook in Clinical Trials
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Funding
L.N.Y., E.D. and K.J.L. were supported by Australian National Health and Medical Research Council Fellowships (Early Career Fellowship IDs 1052388 and 1112672 for L.N.Y. and E.D., respectively; Career Development Fellowship ID 1053609 for K.J.L.). J.A.C. was supported by a Medical Research Council UK Methodology Research Fellowship (ID G1002292).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
