Influential methods reports for group-randomized trials and related designs

Abstract

Background. This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial. Methods. PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article. Results. In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were “first reports” on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design. Conclusions. Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

Keywords

Group-randomized trial cluster-randomized trial individually randomized group treatment trial stepped wedge group-randomized trial stepped wedge cluster-randomized trial

Introduction

Murray et al.¹ recently reviewed the development of methods for the design and analysis of group- or cluster-randomized trials, individually randomized group treatment trials, and stepped wedge group- or cluster-randomized trials based on a review of reports published through 2018. This article provides an update to that article, reviewing reports published through 2020. After describing the key features for these three types of randomized trials, the most influential reports for the development of these methods are identified and described.

Key features

Parallel group- or cluster-randomized trials

A group-randomized trial randomizes groups rather than individuals to study conditions and outcomes are measured for participants from each group.^1–7 In the parallel group-randomized trial considered here, there is no crossover of groups to a different study condition during the trial. The parallel group-randomized trial is the best comparative design available when there is a good reason for randomization of groups rather than individuals. The usual reasons are (1) concern for contamination across conditions if delivered within the same group or (2) the use of a group-based intervention.

The key feature of the group-randomized trial is the randomization of groups to study conditions. Outcomes on participants from the same group are expected to be positively correlated as a result of common exposures, shared experience, or participant interaction.⁸ This correlation violates the assumption of independence of errors that underlies the familiar analytic methods for individually randomized controlled trials.^1–7 This correlation is often measured by the intraclass correlation.

Murray et al.⁹ recently characterized the design features for group-randomized trials involving cancer or cancer-related outcomes. Most group-randomized trials compared two study conditions using a pretest–posttest design. Some used a posttest-only design, while others included multiple pretest and/or posttest measures. Most employed a cohort design observing the same participants from each group at each measurement occasion. Others employed a cross-sectional design observing different participants from each group at each measurement occasion. Still others included both a cohort design and a cross-sectional design in the same study. Most employed some form of restricted randomization, such as stratification, matching, or constrained randomization.

Despite the availability of many books and hundreds of papers on the design and analytic methods for group-randomized trials, reviews have regularly shown that a large proportion of published group-randomized trials fail to account for the intraclass correlation in either sample size calculations or in data analysis.^9–20 This is problematic because undersized studies will have insufficient power for a valid analysis and because invalid analyses will have an inflated type 1 error rate.^{9,12–19,21–37}

Stepped wedge group-randomized trials

Stepped wedge group-randomized trials have increased in popularity over the last 15 years and are now commonly used to evaluate interventions that test new approaches to delivery of care.³⁸ Stepped wedge group-randomized trials are more complex than either parallel group-randomized trials or individually randomized group treatment trials and they also face greater risks for bias.^39,40

The key feature of the stepped wedge group-randomized trial is the crossover of each group from the control to the intervention condition in a random order and on a staggered schedule.⁴¹ Observations from participants from the same group will be positively correlated as in parallel group-randomized trials; however, the impact of the intraclass correlation is reduced in the stepped wedge group-randomized trial because the groups are crossed with study conditions rather than being nested within study conditions. Unlike parallel group-randomized trials, the intervention effect is confounded with calendar time.^41,42 Moreover, the effect of the intervention may vary depending on how much time has passed since the intervention was introduced;^42,43 that is important in a stepped wedge group-randomized trial because they are often longer than a parallel group-randomized trial. Finally, the pattern of correlation over time can be complex because stepped wedge group-randomized trials involve repeated measurements on the same groups and sometimes on the same participants, often for a prolonged period.^44–46

The most common design for a stepped wedge group-randomized trial is a complete design; here, data are collected when all groups are in the control condition, again in each group when one or more groups cross over to the intervention condition, and usually again after all groups are in the intervention condition. The incomplete design is less common; here, data are not collected from all groups at all steps.⁴⁷ Stepped wedge group-randomized trials vary in the number of groups that crosses over in each step, the number of steps, and in the time between steps. They may employ restricted randomization, as described above, to balance groups in the sets to be randomized to the steps. In the continuous recruitment short exposure design, participants are recruited continuously and exposed for only a short period; participants may be measured only once or repeatedly. In the closed cohort design, participants are identified at the beginning of the study, participate throughout the study, and are measured repeatedly. In the open cohort design, many participants are identified at the beginning of the study, but some may leave while other participants are recruited over time; participants may be measured only once or at multiple occasions.

The literature on the design and analytic methods for stepped wedge group-randomized trials has developed rapidly over the last 15 years. Reviews have noted deficiencies in reporting of study design, sample size, analytic methods, and ethical conduct.^48–57

Individually randomized group treatment trials

An individually randomized group treatment trial differs from an individually randomized controlled trial in that the method by which the intervention is delivered creates a level of intraclass correlation that otherwise would not exist.³⁴ Correlated observations can result if participants receive at least some of their intervention in a group format (e.g. attend the same weight-loss class), if participants share the same interventionist (e.g. have the same instructor, therapist, or surgeon), or if participants interact with one another in some other way that is related to the method in which the intervention is delivered (e.g. through a virtual chat room created for participants in the same study condition). The individually randomized group treatment trial is the best comparative design available if randomization of individuals is possible but it is necessary or more efficient to deliver at least some of the intervention in a group format or through a shared interventionist.

The key feature of the individually randomized group treatment trial is that the method by which the intervention is delivered generates some level of correlation among outcomes taken on groups of participants within the same study condition, creating the same type of intraclass correlation seen in parallel group-randomized trials. Investigators must account for the intraclass correlation in the sample size to avoid low power and in the analysis to avoid type I errors.^34,58–62 If the method of intervention delivery creates multiple overlapping groups or if the group structure changes over time, the situation is even more complicated, further increasing the risk of an inflated type I error rate if the investigators do not account for the complex pattern of correlation.^63–66

The methods literature for individually randomized group treatment trials is much more limited than for the other designs considered here and the issues are not widely recognized. Reviews suggest that most investigators who employ the individually randomized group treatment trial design are not aware of it and do not use appropriate methods for sample size or analysis.^34,66–69

Influential methods reports for these designs

Methods

The primary objective of this article was to identify the most influential reports in the development of methods for group-randomized trials and related designs through 2020. In the development of an earlier paper,¹ PubMed was searched to identify all methods papers related to these designs through 2018. Search terms used in the initial PubMed search were (“Randomized Controlled Trials as Topic” (MeSH Terms) AND “Cluster analysis” (MeSH Terms) AND (“sample size” (MeSH Terms) OR “computer simulation” (MeSH Terms) OR “research design” (MeSH Terms) OR “Analysis of Variance” (MeSH Terms) OR “Epidemiologic Research Design” (MeSH Terms) OR “Random Allocation” (MeSH Terms)) AND (“1975/01/01” (PDAT): “2018/12/31” (PDAT)) OR ((cluster *(Title) OR group*(Title) OR community(Title)) AND (random*(Title) OR RCT(Title)) AND (analysis* (Title) OR design(Title) OR method(Title) OR sampl* (Title)) AND (“1975/01/01” (PDAT): “2018/12/31” (PDAT)). Additional searches were conducted for all papers published by each of the first authors identified in the initial search.

The results were augmented by articles, books, chapters, and reports known to the authors of the earlier report. PubMed was then searched for other papers by any of the authors of the identified reports, yielding a total of 4514 candidates. The author reviewed each report and any that were not focused on design or analytic methods for the designs of interest were excluded, leaving 924 reports; 797 focused on parallel group-randomized trials, 49 on individually randomized group treatment trials, 74 on stepped wedge group-randomized trials, and 4 addressed all three designs.

Using the same methods, another 119 reports were identified through 2020, bringing the total to 1043 reports through 2020. That total included 873 reports focused on group-randomized trials, 55 focused on individually randomized group treatment trials, 105 focused on stepped wedge group-randomized trials, and 10 that addressed two or more of these designs.

Citation counts and the relative citation ratio (RCR) were used to assess the influence of each report, accessing data on 27 April 2021. The RCR uses relative citation rates to measure influence at the article level, standardized across areas of science defined by the relative citation network (the articles cited by other articles that also cite the reference article).⁷⁰ In brief, the citation rate of the reference article is compared to the citation rates of the relative citation network. RCRs change over time as the relative citation network grows, particularly in the first few years after a report is published; as a result, the RCR for some of the reports included in the earlier paper had changed by the time the data were accessed for this article 2 years later.

The RCR was available for 898 reports; for the remaining 145 reports, the Web of Science, Scopus, and Google Scholar were used to obtain citation counts. Group-randomized trial reports (N = 32) and reports addressing two or more of the study designs (N = 0) with an RCR ≥ 7.98 (99th percentile for all PubMed entries) or without an RCR but with ≥ 200 citations were retained. Individually randomized group treatment trial reports (N = 7) with an RCR ≥ 3.45 (95th percentile) or without an RCR but with ≥ 100 citations were retained. Stepped wedge group-randomized trial reports (N = 16) with an RCR ≥ 4.91 (97.5th percentile) or without an RCR but with > 150 citations were retained. The citation count thresholds generally discriminated between the reports that fell above and below the RCR thresholds; the sliding scale reflected the number of reports for each design with many more for group-randomized trials and many fewer for individually randomized group treatment trials. This provided 55 methods reports related to these designs that were deemed most influential (Tables 1 –3). These reports do not necessarily represent the current state of the science; recent summaries of the state of the science are available elsewhere.^1,71–73

Table 1.

Influential reports for group-randomized trial methods sorted by RCR and citation count with explanation for selecting items; RCR and count data were retrieved on 27 April 2021.

Citation	RCR	Count	Explanation
Campbell ert al.⁷⁴	47.84	849	High RCR and citation count
Campbell et al.⁷⁵.	37.90	1042	High RCR and citation count
Donner et al.⁷⁶.	23.15	411	High RCR and citation count
Bland and Kerry⁷⁷	20.98	397	High RCR and citation count
Gulliford et al.⁷⁸	19.89	427	High RCR and citation count
Hayes and Bennett⁷⁹	18.16	512	High RCR and citation count
Rao and Scott⁸⁰	15.64	316	High RCR and citation count
Grant et al.⁸¹	14.51	219	High RCR and citation count
Donner et al.⁸²	14.22	352	High RCR and citation count
Zou and Donner⁸³	13.73	253	High RCR and citation count
Krull and MacKinnon⁸⁴	13.49	359	High RCR and citation count
Hedeker and Gibbons⁸⁵	12.99	221	High RCR and citation count
Murray et al.⁸⁶	12.45	341	High RCR and citation count
Eccles et al.⁸⁷	11.40	321	High RCR and citation count
Murray and Hannan⁸⁸	11.33	133	High RCR
Adams et al.⁸⁹	11.05	303	High RCR and citation count
Eldridge et al.⁹⁰	9.56	260	High RCR and citation count
Hedeker and Gibbons⁹¹	9.33	180	High RCR
Feldman⁹²	9.09	138	High RCR
Handley et al.⁹³	9.01	49	High RCR
Emsley et al.⁹⁴	8.46	190	High RCR
Rutterford et al.⁹⁵	8.41	105	High RCR
Austin⁹⁶	8.11	82	High RCR
Donner and Klar³		2120	High citation count
Murray²		1832	High citation count
Hayes and Moulton⁴		1033	High citation count
Hayes and Moulton⁷		1033	High citation count
Hedges and Hedberg⁹⁷		414	High citation count
Raudenbush⁹⁸		383	High citation count
Eldridge and Kerry⁵		295	High citation count
Cornfield⁹⁹		289	High citation count
Hedges¹⁰⁰		230	High citation count

Reports in bold type were not included as influential reports in the earlier paper (1).

Table 2.

Influential reports for individually randomized group treatment trial methods sorted by RCR and citation count with explanation for selecting items; RCR and count data were retrieved on 27 April 2021.

Citation	RCR	Count	Explanation
Boutron et al.⁶⁰	59.92	1360	High RCR and citation count
Boutron et al.¹⁰¹	41.80	321	High RCR and citation count
Crits-Christoph and Mintz¹⁰²	8.35	126	High RCR and citation count
Whiting-O’Keefe et al.¹⁰³	5.85	90	High RCR
Roberts and Roberts⁵⁸	4.82	128	High RCR and citation count
Baldwin et al.⁶⁸	3.57	88	High RCR
Nye et al.¹⁰⁴		2465	High citation count

Reports in bold type were not included as influential reports in the earlier paper (1).

Table 3.

Influential reports for stepped wedge group-randomized trial methods sorted by RCR and citation count with explanation for selecting items; RCR and count data were retrieved on 27 April 2021.

Citation	RCR	Count	Explanation
Hemming et al.⁴²	31.13	416	High RCR and citation count
Hussey and Hughes⁴¹	19.48	557	High RCR and citation count
Brown and Lilford⁴⁸	17.69	497	High RCR and citation count
Mdege et al.⁴⁹	12.51	252	High RCR and citation count
Kasza et al.⁴⁴	11.42	35	High RCR
Hemming et al.¹⁰⁵	10.90	67	High RCR
Woertman et al.¹⁰⁶	9.22	148	High RCR
Hemming et al.¹⁰⁷	7.85	98	High RCR
Copas et al.⁴⁷	7.35	87	High RCR
Hemming and Taljaard¹⁰⁸	6.67	70	High RCR
Hooper et al.¹⁰⁹	6.29	71	High RCR
Beard et al.⁵⁰	5.86	68	High RCR
Baio et al.¹¹⁰	5.70	71	High RCR
Hemming et al.¹¹¹	5.37	45	High RCR
Barker et al.⁵³	5.33	54	High RCR
Girling and Hemming¹¹²	4.93	51	High RCR

Reports in bold type were not included as influential reports in the earlier paper (1).

The next three sections identify the influential reports for the three designs. The earliest reports are mentioned first in each section and the remaining reports are grouped by theme.

Parallel group-randomized trials

In 1978, Cornfield published the first methods paper for trials involving group randomization in the biomedical literature. He identified two penalties associated with group randomization: extra variation attributable to the group and limited degrees of freedom for the test of the intervention effect.⁹⁹ Both penalties must be addressed in studies that randomize groups rather than individuals.

Several influential papers addressed general issues for parallel group-randomized trials. A 1997 commentary drew attention to the methodological issues inherent in these trials.⁷⁷ A 1997 paper presented features for the optimal design of a parallel group-randomized trial.⁹⁸ A 2003 paper reported on the use of parallel group-randomized trials for evaluating the effectiveness of change and improvement strategies.⁸⁷ A 2013 paper reported on methods for process evaluation.⁸¹ A 2018 paper reported on the use of parallel group-randomized trials in dissemination and implementation research.⁹³

Others presented methods for sample size calculations based either on the intraclass correlation^{76,78,88,89,97} or on methods based on the coefficient of variation.⁷⁹ Eldridge et al.⁹⁰ and Rutterford et al.⁹⁵ addressed the question of unequal sample size at the group or cluster level. Rutterford et al.⁹⁵ also addressed the effect of attrition, non-compliance, adjustment for baseline covariates, and repeated measures on sample size estimation.

Others focused on specific analytic methods. Those included methods for random coefficient or growth curve models,⁹² binary data,⁸⁰ ordinal data,^85,91 Poisson regression,⁸³ meta-analysis,^82,100 mediation analysis,^84,94 and survival analysis.⁹⁶

The first textbook on the design and analysis of parallel group-randomized trials was published in 1998² followed in 2000 by the second.³ Three subsequent textbooks also met the criteria to be judged an influential report.^4,7,113

The first CONSORT statement on parallel group-randomized trials was published in 2004⁷⁴ and provided a checklist to identify the methodological information to include in trial reports. An update was published in 2012.⁷⁵

Murray et al.⁸⁶ published a review of methodological issues in parallel group-randomized trials in 2004, summarizing work on both design and analytic methods.

Stepped wedge group-randomized trials

Though the concept of the stepped wedge design was introduced in 1987,¹¹⁴ Hussey and Hughes⁴¹ published the first analytic methods for that design in 2007. Copas et al.⁴⁷ later delineated three types of stepped wedge group-randomized trials and discussed the number and length of the steps, incomplete and complete designs, and randomization methods, including restricted randomization methods. Hemming et al.⁴² provided guidance on the rationale, design, analysis, and reporting for this design. They noted that the stepped wedge group-randomized trial is particularly well suited for evaluations of health service delivery interventions. Barker et al.⁵³ reviewed the statistical methods used in stepped wedge group-randomized trials. Hemming et al.¹¹¹ summarized methods appropriate for stepped wedge group-randomized trials with repeated cross-sectional samples.

Several papers addressed sample size methods. Woertman et al. noted that the sample size will depend on group size, the intraclass correlation, the number of steps, the number of baseline measurements, and the number of measurements between steps;¹⁰⁶ a subsequent letter¹¹⁵ corrected an important error in that paper which was accepted by the authors.¹¹⁶ Baio et al.¹¹⁰ presented simulation methods for sample size estimation. Hemming et al.¹⁰⁷ considered power for several different stepped wedge group-randomized trial designs, including incomplete cross-sectional designs, designs with multiple levels of nesting, and complete designs. Girling and Hemming¹¹² proposed an algorithm to optimize the design of stepped wedge group-randomized trials and showed that for large studies, the best design may be a hybrid of parallel group-randomized trial and stepped wedge group-randomized trial design components. Hemming and Taljaard¹⁰⁸ compared power for parallel group-randomized trials and stepped wedge group-randomized trials when the number of groups is fixed and reported that the parallel group-randomized trial tends to be more efficient when the intraclass correlation is small and that the stepped wedge group-randomized trial tends to be more efficient when the intraclass correlation is large, dependent on group size. Hooper et al.¹⁰⁹ presented formulaic methods for sample size estimation based on the intraclass correlation and the cluster and individual autocorrelations. Kasza et al.⁴⁴ reported on the effect of decaying correlation over time on sample size and power.

Several state-of-the-practice reviews of stepped wedge group-randomized trials have been published,^48–50 reporting wide variation in data analytic methods and reporting standards. The recent CONSORT statement for this design¹⁰⁵ was written in part to try to reduce that variation.

Individually randomized group treatment trials

Several early reports addressed the risk of type 1 error in studies in which individually randomized participants receive their intervention in a group format or from a shared interventionist. These papers appeared in the biomedical,¹⁰³ psychological,^68,102 and educational literature.¹⁰⁴

Roberts and Roberts⁵⁸ presented the first report to address the analytic challenges specific to individually randomized group treatment trials in 2005. Consistent with guidance for parallel group-randomized trials, they noted that mixed models specifying the groups as levels of a random effect provided an appropriate analysis.

In 2008, Boutron et al.⁶⁰ extended the CONSORT statement to non-pharmacologic interventions which include many individually randomized group treatment trials. They pointed to the need to provide details on how the intervention was delivered (e.g. individually, in groups, through a common interventionist) and to address the implications for the analysis of having correlated observations within one or more study conditions. Boutron et al.¹⁰¹ published an update to that CONSORT statement in 2017.

Summary

This article identifies the 55 most influential reports contributing to the methods for the evaluation of group- or cluster-randomized trials, individually randomized group treatment trials, and stepped wedge group- or cluster-randomized trials through 2020, adding 2 years of data to an earlier paper.¹ There was substantial overlap with the 50 reports listed in the earlier paper, but there was also turnover in the list of influential reports, based on changes to the RCRs and citation counts as often happens with the passage of time. There were six new parallel group-randomized trial reports, one new individually randomized group treatment trial report, and five new stepped wedge group-randomized trial reports included here. Of the original 50 reports, 3 individually randomized group treatment trial reports, 2 stepped wedge group-randomized trial reports, and 2 reports that addressed more than two designs no longer met the inclusion criteria and so were not included in this report. Reports that had been included previously but failed to meet the inclusion criteria for this update generally had low but qualifying RCRs for the original report and those values declined with the additional follow-up time. RCRs are generally stable after a few years and most of the reports for which the ratio changed were relatively new in the original report.

The RCR and citation counts were used to gauge the influence of a report on the work published later. Influence does not equate to quality, so there is no claim that the most influential reports are also the highest quality reports.

Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbook treatments that covered a wide range of issues for these designs. Others were “first reports” on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-sectional). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the most influential reports were CONSORT statements which provide guidance on how to present the methods and results from a study based on its design. Collectively, they address topics of great interest to investigators who might consider conducting a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group or cluster-randomized trial and need information to guide their planning for design, analysis, and sample size. As a set, they make an excellent reading list for anyone interested in learning about the methods used in these designs and their appropriate applications in public health and medicine.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

David M Murray

References

Murray

Taljaard

Turner

et al. Essential ingredients and innovations in the design and analysis of group-randomized trials. Annu Rev Public Health 2020; 41: 1–19.

Murray

DM.

Design and analysis of group-randomized trials. New York: Oxford University Press, 1998, p. 467.

Donner

Klar

Design and analysis of cluster randomization trials in health research. London: Arnold, 2000, p. 178.

Hayes

Moulton

LH.

Cluster randomised trials. Boca Raton, FL: CRC Press, 2009.

Eldridge

Kerry

A practical guide to cluster randomised trials in health services research. London: Arnold, 2012.

Campbell

Walters

SJ.

How to design, analyse and report cluster randomised trials in medicine and health related research. Chichester: John Wiley & Sons, 2014.

Hayes

Moulton

LH.

Cluster randomised trials. 2nd ed. Boca Raton, FL: CRC Press, 2017.

Kish

Survey sampling. New York: John Wiley & Sons, 1965, p. 643.

Murray

Pals

George

et al. Design and analysis of group-randomized trials in cancer: a review of current practices. Prev Med 2018; 111: 241–247.

10.

Brown

Bohan Brown

et al. Best (but oft-forgotten) practices: designing, analyzing, and reporting cluster randomized controlled trials. Am J Clin Nutr 2015; 102(2): 241–248.

11.

Crespi

Maxwell

Cluster randomized trials of cancer screening interventions: are appropriate statistical methods being used. Contemp Clin Trials 2011; 32(4): 477–484.

12.

Diaz-Ordaz

Froud

Sheehan

et al. A systematic review of cluster randomised trials in residential facilities for older people suggests how to improve quality. BMC Med Res Methodol 2013; 13: 127.

13.

Donner

Brown

Brasher

A methodological review of non-therapeutic intervention trials employing cluster randomization, 1979–1989. Int J Epidemiol 1990; 19(4): 795–800.

14.

Eldridge

Ashby

Bennett

et al. Internal and external validity of cluster randomised trials: systematic review of recent trials. BMJ 2008; 336: 876–880.

15.

Ivers

Taljaard

Dixon

et al. Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials 2000–8. BMJ 2011; 343: d5886.

16.

Murray

Pals

Blitstein

et al. Design and analysis of group-randomized trials in cancer: a review of current practices. J Natl Cancer Inst 2008; 100: 483–491.

17.

Rutterford

Taljaard

Dixon

et al. Reporting and methodological quality of sample size calculations in cluster randomized trials could be improved: a review. J Clin Epidemiol 2015; 68(6): 716–723.

18.

Simpson

Klar

Donnor

Accounting for cluster randomization: a review of primary prevention trials, 1990 through 1993. Am J Public Health 1995; 85(10): 1378–1383.

19.

Varnell

Murray

Janega

et al. Design and analysis of group-randomized trials: a review of recent practices. Am J Public Health 2004; 94(3): 393–399.

20.

Cook

Rutherford

Scales

et al. Rationale, methodological quality, and reporting of cluster-randomized controlled trials in critical care medicine: a systematic review. Crit Care Med 2021; 49: 977–987.

21.

Puffer

Torgerson

Watson

Evidence for risk of bias in cluster randomised trials: review of recent trials published in three general medical journals. BMJ 2003; 327: 785–789.

22.

Eldridge

Ashby

Feder

et al. Lessons for cluster randomized trials in the twenty-first century: a systematic review of trials in primary care. Clin Trials 2004; 1(1): 80–90.

23.

Bowater

Abdelmalik

Lilford

RJ.

The methodological quality of cluster randomised controlled trials for managing tropical parasitic disease: a review of trials published from 1998 to 2007. Trans R Soc Trop Med Hyg 2009; 103(5): 429–436.

24.

Froud

Eldridge

Diaz Ordaz

et al. Quality of cluster randomized controlled trials in oral health: a systematic review of reports published between 2005 and 2009. Community Dent Oral Epidemiol 2012; 40Suppl1: 3–14.

25.

Ivers

Halperin

Barnsley

et al. Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials 2012; 13: 120.

26.

Diaz-Ordaz

Kenward

Cohen

et al. Are missing data adequately handled in cluster randomised trials? A systematic review and guidelines. Clin Trials 2014; 11(5): 590–600.

27.

Mdege

Brabyn

Hewitt

et al. The 2 x 2 cluster randomized controlled factorial trial design is mainly used for efficiency and to explore intervention interactions: a systematic review. J Clin Epidemiol 2014; 67: 1083–1092.

28.

Wright

Ivers

Eldridge

et al. A review of the use of covariates in cluster randomized trials uncovers marked discrepancies between guidance and practice. J Clin Epidemiol 2015; 68(6): 603–609.

29.

Fiero

Huang

Oren

et al. Statistical analysis and handling of missing data in cluster randomized trials: a systematic review. Trials 2016; 17: 72.

30.

Richardson

Garner

Donegan

Cluster randomised trials in cochrane reviews: evaluation of methodological and reporting practice. PLoS ONE 2016; 11(3): e0151818.

31.

Huang

Fiero

Bell

ML.

Generalized estimating equations in cluster randomized trials with a small number of clusters: review of practice and simulation study. Clin Trials 2016; 13(4): 445–449.

32.

Agbla

DiazOrdaz

Reporting non-adherence in cluster randomised trials: a systematic review. Clin Trials 2018; 15(3): 294–304.

33.

Heo

Nair

Wylie-Rosett

et al. Trial characteristics and appropriateness of statistical methods applied for design and analysis of randomized school-based studies addressing weight-related issues: a literature review. J Obes 2018; 2018: 8767315.

34.

Pals

Murray

Alfano

et al. Individually randomized group treatment trials: a critical appraisal of frequently used design and analytic approaches. Am J Public Health 2008; 98(8): 1418–1424.

35.

El Alili

van Dongen

Goldfeld

et al. Taking the analysis of trial-based economic evaluations to the next level: the importance of accounting for clustering. Pharmacoeconomics 2020; 38(11): 1247–1261.

36.

Golzarri-Arroyo

Oakes

Brown

et al. Incorrect analyses of cluster-randomized trials that do not take clustering and nesting into account likely lead to p-values that are too small. Child Obes 2020; 16: 65–66.

37.

O’Hara

Blanco

Leekha

et al. Design, implementation, and analysis considerations for cluster-randomized trials in infection control and hospital epidemiology: a systematic review. Infect Contr Hosp Epidemiol 2019; 40(6): 686–692.

38.

Hemming

Taljaard

Grimshaw

Introducing the new CONSORT extension for stepped-wedge cluster randomised trials. Trials 2019; 20: 68.

39.

Thompson

Fielding

Davey

et al. Bias and inference from misspecified mixed-effect models in stepped wedge trial analysis. Stat Med 2017; 36: 3670–3682.

40.

Hemming

Taljaard

Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice?

Int J Epidemiol 2020; 49: 1043–1052.

41.

Hussey

Hughes

JP.

Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007; 28: 182–191.

42.

Hemming

Haines

Chilton

et al. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ 2015; 350: h391.

43.

Hughes

Granston

Heagerty

PJ.

Current issues in the design and analysis of stepped wedge trials. Contemp Clin Trials 2015; 45(Pt. A): 55–60.

44.

Kasza

Hemming

Hooper

et al. Impact of non-uniform correlation structure on sample size and power in multiple-period cluster randomised trials. Stat Methods Med Res 2019; 28(3): 703–716.

45.

Design and analysis considerations for cohort stepped wedge cluster randomized trials with a decay correlation structure. Stat Med 2020; 39: 438–455.

46.

Kasza

Forbes

AB.

Inference for the treatment effect in multiple-period cluster randomised trials when random effect correlation structure is misspecified. Stat Methods Med Res 2019; 28(10–11): 3112–3122.

47.

Copas

Lewis

Thompson

et al. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation approaches. Trials 2015; 16: 352.

48.

Brown

Lilford

RJ.

The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006; 6: 54.

49.

Mdege

Man

Taylor Nee Brown

et al. Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation. J Clin Epidemiol 2011; 64(9): 936–948.

50.

Beard

Lewis

Copas

et al. Stepped wedge randomised controlled trials: systematic review of studies published between 2010 and 2014. Trials 2015; 16: 353.

51.

Davey

Hargreaves

Thompson

et al. Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014. Trials 2015; 16: 358.

52.

Martin

Taljaard

Girling

et al. Systematic review finds major deficiencies in sample size methodology and reporting for stepped-wedge cluster randomised trials. BMJ Open 2016; 6: e010166.

53.

Barker

McElduff

D'Este

et al. Stepped wedge cluster randomised trials: a review of the statistical methodology used and available. BMC Med Res Methodol 2016; 16: 69.

54.

Grayling

Wason

Mander

AP.

Stepped wedge cluster randomized controlled trial designs: a review of reporting quality and design features. Trials 2017; 18: 33.

55.

Taljaard

Hemming

Shah

et al. Inadequacy of ethical conduct and reporting of stepped wedge cluster randomized trials: results from a systematic review. Clin Trials 2017; 14(4): 333–341.

56.

Kristunas

Morris

Gray

Unequal cluster sizes in stepped-wedge cluster randomised trials: a systematic review. BMJ Open 2017; 7: e017151.

57.

Eichner

Groenwold

RHH

Grobbee

et al. Systematic review showed that stepped-wedge cluster randomized trials often did not reach their planned sample size. J Clin Epidemiol 2019; 107: 89–100.

58.

Roberts

SA.

Design and analysis of clinical trials with clustering effects due to treatment. Clin Trials 2005; 2(2): 152–162.

59.

Baldwin

Bauer

Stice

et al. Evaluating models for partially clustered designs. Psychol Methods 2011; 16(2): 149–165.

60.

Boutron

Moher

Altman

et al. Extending the CONSORT statement to randomized trials of nonpharmacologic treatment: explanation and elaboration. Ann Intern Med 2008; 148: 295–309.

61.

Walwyn

Roberts

Therapist variation within randomised trials of psychotherapy: implications for precision, internal and external validity. Stat Methods Med Res 2010; 19(3): 291–315.

62.

Kahan

Morris

TP.

Assessing potential sources of clustering in individually randomised trials. BMC Med Res Methodol 2013; 13: 58.

63.

Roberts

Walwyn

Design and analysis of non-pharmacological treatment trials with multiple therapists per patient. Stat Med 2013; 32: 81–98.

64.

Andridge

Shoben

Muller

et al. Analytic methods for individually randomized group treatment trials and group-randomized trials when subjects belong to multiple groups. Stat Med 2014; 33: 2178–2190.

65.

Bauer

Gottfredson

Dean

et al. Analyzing repeated measures data on individuals nested within groups: accounting for dynamic group effects. Psychol Methods 2013; 18(1): 1–14.

66.

Conroy

Rosala-Hallas

Blazeby

et al. Randomized trials involving surgery did not routinely report considerations of learning and clustering effects. J Clin Epidemiol 2019; 107: 27–35.

67.

Pals

Wiegand

Murray

DM.

Ignoring the group in group-level HIV/AIDS intervention trials: a review of reported design and analytic methods. Aids 2011; 25: 989–996.

68.

Baldwin

Murray

Shadish

WR.

Empirically supported treatments or type I errors? Problems with the analysis of data from group-administered treatments. J Consult Clin Psychol 2005; 73: 924–935.

69.

Lee

Thompson

SG.

Clustering by health professional in individually randomised trials. BMJ 2005; 330: 142–144.

70.

Hutchins

Yuan

Anderson

et al. Relative citation ratio (RCR): a new metric that uses citation rates to measure influence at the article level. Plos Biol 2016; 14(9): e1002541.

71.

Turner

Gallis

et al. Review of recent methodological developments in group-randomized trials: part 1 – design. Am J Public Health 2017; 107(6): 907–915.

72.

Turner

Prague

Gallis

et al. Review of recent methodological developments in group-randomized trials: part 2 – analysis. Am J Public Health 2017; 107(7): 1078–1086.

73.

Hughes

Hemming

et al. Mixed-effects models for the design and analysis of stepped wedge cluster randomized trials: an overview. Stat Methods Med Res 2021; 30(2): 612–639.

74.

Campbell

Elbourne

Altman

et al. CONSORT statement: extension to cluster randomised trials. BMJ 2004; 328: 702–708.

75.

Campbell

Piaggio

Elbourne

et al. CONSORT 2010 statement: extension to cluster randomised trials. BMJ 2012; 345: e5661.

76.

Donner

Birkett

Buck

Randomization by cluster: sample size requirements and analysis. Am J Epidemiol 1981; 114(6): 906–914.

77.

Bland

Kerry

SM.

Statistics notes. Trials Randomised in Clusters. BMJ 1997; 315: 600.

78.

Gulliford

Ukoumunne

Chinn

Components of variance and intraclass correlations for the design of community-based surveys and intervention studies: data from the Health Survey for England 1994. Am J Epidemiol 1999; 149: 876–883.

79.

Hayes

Bennett

Simple sample size calculation for cluster-randomized trials. Int J Epidemiol 1999; 28(2): 319–326.

80.

Rao

Scott

AJ.

A simple method for the analysis of clustered binary data. Biometrics 1992; 48(2): 577–585.

81.

Grant

Treweek

Dreischulte

et al. Process evaluations for cluster-randomised trials of complex interventions: a proposed framework for design and reporting. Trials 2013; 14: 15.

82.

Donner

Klar

Issues in the meta-analysis of cluster randomized trials. Stat Med 2002; 21: 2971–2980.

83.

Zou

Donner

Extension of the modified Poisson regression model to prospective studies with correlated binary data. Stat Methods Med Res 2013; 22(6): 661–670.

84.

Krull

MacKinnon

DP.

Multilevel mediation modeling in group-based intervention studies. Eval Rev 1999; 23(4): 418–444.

85.

Hedeker

Gibbons

RD.

A random-effects ordinal regression model for multilevel analysis. Biometrics 1994; 50(4): 933–944.

86.

Murray

Varnell

Blitstein

JL.

Design and analysis of group-randomized trials: a review of recent methodological developments. Am J Public Health 2004; 94(3): 423–432.

87.

Eccles

Grimshaw

Campbell

et al. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Health Care 2003; 12(1): 47–52.

88.

Murray

Hannan

PJ.

Planning for the appropriate analysis in school-based drug-use prevention studies. J Consult Clin Psychol 1990; 58(4): 458–468.

89.

Adams

Gulliford

Ukoumunne

et al. Patterns of intra-cluster correlation from primary care research to inform study design and analysis. J Clin Epidemiol 2004; 57(8): 785–794.

90.

Eldridge

Ashby

Kerry

Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. Int J Epidemiol 2006; 35(5): 1292–1300.

91.

Hedeker

Gibbons

RD.

MIXOR: a computer program for mixed-effects ordinal regression analysis. Comput Methods Programs Biomed 1996; 49(2): 157–176.

92.

Feldman

HA.

Families of lines: random effects in linear regression analysis. J Appl Physiol 1988; 64(4): 1721–1732.

93.

Handley

Lyles

McCulloch

et al. Selecting and improving quasi-experimental designs in effectiveness and implementation research. Annu Rev Public Health 2018; 39: 5–25.

94.

Emsley

Dunn

White

IR.

Mediation and moderation of treatment effects in randomised controlled trials of complex interventions. Stat Methods Med Res 2010; 19(3): 237–270.

95.

Rutterford

Copas

Eldridge

Methods for sample size determination in cluster randomized trials. Int J Epidemiol 2015; 44(3): 1051–1067.

96.

Austin

PC.

A tutorial on multilevel survival analysis: methods, models and applications. Int Stat Rev 2017; 85(2): 185–203.

97.

Hedges

Hedberg

EC.

Intraclass correlation values for planning group-randomized trials in education. Educ Eval Policy an 2007; 29: 60–87.

98.

Raudenbush

SW.

Statistical analysis and optimal design for cluster randomized trials. Psychol Method 1997; 2: 173–185.

99.

Cornfield

Randomization by group: a formal analysis. Am J Epidemiol 1978; 108(2): 100–102.

100.

Hedges

LV.

Effect sizes in cluster-randomized designs. J Educ Behav Stat 2007; 32: 341–370.

101.

Boutron

Altman

Moher

et al. CONSORT statement for randomized trials of nonpharmacologic treatments: a 2017 update and a CONSORT extension for nonpharmacologic trial abstracts. Ann Intern Med 2017; 167: 40–47.

102.

Crits-Christoph

Mintz

Implications of therapist effects for the design and analysis of comparative studies of psychotherapies. J Consult Clin Psychol 1991; 59(1): 20–26.

103.

Whiting-O'Keefe

Henke

Simborg

DW.

Choosing the correct unit of analysis in medical care experiments. Med Care 1984; 22(12): 1101–1114.

104.

Nye

Konstantopoulos

Hedges

How large are teacher effects?

Educ Eval Policy an 2004; 26: 237–257.

105.

Hemming

Taljaard

McKenzie

et al. Reporting of stepped wedge cluster randomised trials: extension of the CONSORT 2010 statement with explanation and elaboration. BMJ 2018; 363: k1614.

106.

Woertman

de Hoop

Moerbeek

et al. Stepped wedge designs could reduce the required sample size in cluster randomized trials. J Clin Epidemiol 2013; 66(7): 752–758.

107.

Hemming

Lilford

Girling

AJ.

Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Stat Med 2015; 34: 181–196.

108.

Hemming

Taljaard

Sample size calculations for stepped wedge and cluster randomised trials: a unified approach. J Clin Epidemiol 2016; 69: 137–146.

109.

Hooper

Teerenstra

de Hoop

et al. Sample size calculation for stepped wedge and other longitudinal cluster randomised trials. Stat Med 2016; 35: 4718–4728.

110.

Baio

Copas

Ambler

et al. Sample size calculation for a stepped wedge trial. Trials 2015; 16: 354.

111.

Hemming

Taljaard

Forbes

Analysis of cluster randomised stepped wedge trials with repeated cross-sectional samples. Trials 2017; 18: 101.

112.

Girling

Hemming

Statistical efficiency and optimal design for stepped cluster studies under linear mixed effects models. Stat Med 2016; 35: 2149–2166.

113.

Eldridge

Kerry

A practical guide to cluster randomized trials in health research. London: Arnold, 2012.

114.

The Gambia Hepatitis Study Group. The Gambia hepatitis intervention study: the Gambia Hepatitis Study Group. Cancer Res 1987; 47: 5782–5787.

115.

Hemming

Girling

The efficiency of stepped wedge vs. cluster randomized trials: stepped wedge studies do not always require a smaller sample size. J Clin Epidemiol 2013; 66(12): 1427–1428.

116.

de Hoop

Woertman

Teerenstra

. The stepped wedge cluster randomized trial always requires fewer clusters but not always fewer measurements, that is, participants than a parallel cluster randomized trial in a cross-sectional design: in reply. J Clin Epidemiol 2013; 66: 1428.