Practical considerations in utilizing cluster randomized controlled trials conducted in biopharmaceutical industry

Abstract

Cluster randomized controlled trials (cluster RCTs), also known as parallel-arm group-randomized trials, are trials in which the randomized units are groups of participants, as opposed to individual participants. These trials have largely been implemented to address broad public health issues, but with the growing interest in use of real-world data in the regulatory setting, this design may be increasingly considered for industry trials. The key difference between cluster RCTs and traditional RCTs is the intraclass correlation coefficient (ICC) that needs to be considered in cluster RCTs. In this article, we discuss some key practical considerations that are related to ICC in the design, conduct, analysis, and report stages of a cluster RCT. These key considerations related to ICC can lead to improvement in how we translate research findings from cluster RCTs into practices in the biopharmaceutical industry.

Keywords

Cluster randomized controlled trials design group-randomized trials intraclass correlation coefficients biopharmaceutical industry

Introduction

Medical researchers in biopharmaceutical industry are utilizing pragmatic clinical trials to investigate the effectiveness of healthcare innovations (drugs, devices, treatment policies, or other interventions) and deliver those that are shown to be safe, beneficial, and cost-effective.¹ Cluster randomized controlled trials (cluster RCTs), as a type of pragmatic trial designs, have been commonly utilized interventions that operate at a group level.

A more general name for this type of study design is group-randomized trials,² which includes cross-sectional, cohort, parallel-arm, crossover, and stepped-wedge group-randomized trials. Following two review papers of recent methodological developments in group-randomized trials focused on design and analysis,^3,4 respectively, we review practical considerations in utilizing parallel-arm cluster RCTs.

Cluster RCTs have at least two advantages.⁵ First, a cluster RCT may be preferred when the target population of the investigative intervention is a collective of patients. For example, a cluster RCT is preferred to evaluate whether a new treatment pathway, a new guideline recommendation, or other practice-wide, hospital-wide, or system-wide change is affecting patient outcomes. Second, cluster randomization is often advocated to minimize treatment ‘contamination’ between intervention and control participants.⁶ For example, in the Goals for Eating and Moving (GEM) study of weight loss, people in the control group might learn about the investigative diet and exercise intervention and adopt it themselves.

With these advantages comes challenges. There are challenges associated with improper reporting⁷ and with obtaining informed consent from groups and individual participants.^4,8 And there are challenges associated with the intraclass correlation coefficient (ICC), which expresses similarity among participants within clusters in cluster RCTs. Such similarity among participants within pre-existing clusters leads to that the clustered data contain less information (smaller ‘effective’ sample) than the independent data of the same sample size, resulting in lower power to detect true differences between study arms.⁹ Since data within a cluster of, say, $m$ participants are expected to be correlated, the ‘effective’ sample size for that cluster is somewhere between 1 (when ICC = 1) and $m$ (when ICC = 0). In this article, we discuss some key practical considerations related to ICC in the design, conduct, analysis, and report stages of a cluster RCTs, hoping that these considerations could lead to improvement in how we translate research findings into practice.

The novelty of this review article is twofold. First, in the literature, the reviews of cluster RCT are usually focused on design and analysis.^3,4 We extend the review to include two other important stages (that is, study conduct and report), making the review broader. Second, the existing reviews of cluster RCT are usually comprehensive. We focus on the most important feature of cluster RCTs (that is, the ICC related considerations), making the review deeper.

We participated in designing three cluster RCTs.^10–12 For example, in the Goals for Eating and Moving (GEM) study, they randomized 19 patient-centred medical home teams from New York City healthcare systems to either the GEM intervention arm or the Enhanced Usual Care control arm.¹⁰ Each patient-centred medical home team consisted of two providers and on average each provider recruited 12 patients. The primary outcome variable was the amount weight loss at 12-month follow-up.

We also reviewed several proposals of using cluster RCTs in the pharmaceutical setting. For example, the REACH HCV study is an international, cluster-randomized non-clinical trial with two arms (https://clinicaltrials.gov/ct2/show/NCT03935906). The unit of randomization is the community pharmacy, so all participants in clusters are allocated to one of two pathways for hepatitis C virus testing and treatment. All eligible hepatitis C virus-infected participants will receive treatment with 100 mg glecaprevir/40 mg pibrentasvir (Maviret; an AbbVie pan-genotypic direct acting antiviral drug) for between 8 and 16 weeks, depending on blood test results.

Key considerations

Design

Compared with traditional RCTs, cluster RCTs may be more complex in several design aspects, such as population identification, sampling units, treatment or intervention definition, outcome ascertainment in real-world setting, along with covariates that are specific to cluster or individual subject level. In this article, we focus on the ICC-related aspects in the design, as there is a rich literature delineating other design aspects.^3,13

When we design a cluster RCT, we should first examine the number of clusters and the cluster sizes that are available for the study under certain constraints such as costs, time, and enrolment rate. Let $n$ be the number of clusters that we plan to obtain and $m_{i}$ be their cluster sizes, $i = 1, \dots, n$ .

The primary implication of using a cluster RCT is that patients within clusters are often more likely to respond in a similar manner, leading to a loss of statistical power in comparison with a simple RCT. For example, if the clusters units are families, members from the same family tend to share similar lifestyle, and if the clusters units are providers, patients from the same provider tend to share similar demographic characteristics. To understand the ICC, we decompose the outcome variable into three components, the grand tendency, cluster specific component, and individual specific component. Without loss of generality, consider continuous outcome with the following decomposition¹⁴

Y_{i j} = μ + α_{i} + ϵ_{i j}, i = 1, \dots, n; j = 1, \dots, m_{i}

where $α_{i}$ is cluster-specific component with variance $σ_{a}^{2}$ and $ϵ_{i j}$ is individual-specific component with variance $σ_{e}^{2}$ . Thus, we can define the ICC as

ρ = \frac{σ_{a}^{2}}{(σ_{a}^{2} + σ_{e}^{2})}

The presence of ICC $ρ$ leads to variance inflation,¹⁵ which further leads to power loss. The variance inflation factor for the setting of equal cluster size $m_{i} \equiv m$ is $I F = 1 + (m - 1) ρ$ . For other outcome types, refer to Table 1 in the aforementioned review paper focused on design,³ while for the existing approaches for estimating ICC, refer to Table 1 in the pair review paper focused on analysis.⁴

In the study design stage, when we perform sample size calculation for a cluster RCT, we should take the ICC into account.¹⁶ To do this, we can first calculate the required sample size, say $N^{*}$ , for a simple RCT to detect the expected effect size with the desired power without worrying about the ICC. Then we use the variance inflation factor to inflate the sample size as $N = I F \times N^{*}$ for the cluster RCT that we are designing.¹⁶ Thus, $n = N / m$ is the required number of clusters to be obtained. For a cluster RCT with fixed number of clusters, the optimal cluster size under cost constraint is $m_{opt} = \sqrt{(1 - ρ) c / (ρ s)}$ where $c$ is cost per cluster and $s$ is cost per participant,¹⁷ and the largest gains in power will be obtained when the cluster size increases from 1 to about $m = 1 / ρ$ while any further increase will tend to have only a modest effect on power.¹⁸

Here are four practical considerations. First, if we are unable to find any literature or pilot study to have an estimate of the ICC, we can start the calculation using $ρ = 0.05$ , and then conduct some sensitivity analysis for a range of values of $ρ$ . Note that 0.05 is suggested here, based on a review paper of total 18 ICC estimates which varied in magnitude from 0 to 0.06.¹⁸ Prior information on the ICC can also be incorporated via the Bayesian approach.¹⁸ Second, when we obtain an estimate of ICC from a previous study with a relatively small number of clusters, we need to assess its value, because given the large standard error associated with the estimate of the ICC, it would be dangerous to overestimate the stability of an estimate obtained from a cluster RCT involving less than about 40 clusters.¹⁸ Third, in practice, very often cluster sizes are variable. This will further inflate the required sample size. For this case, we should revise the variance inflation factor as $I F = 1 + [\bar{m} (1 + c v^{2}) - 1] ρ$ , where $\bar{m}$ is the average cluster size and $cv$ is the coefficient of variation of cluster size.¹⁶ Fourth, in practice, there are hierarchical levels of clustering. For example, in a cluster RCT, randomization is over the site level, while within sites, providers are clusters. For this case, let $m_{1}$ be the average number of providers within each site (with ICC $ρ_{1}$ ) and $m_{2}$ be the average number of patients within each provider (with ICC $ρ_{2})$ . For this case, we should revise the variance inflation factor as $I F = [1 + (m_{1} - 1) ρ_{1}] \times [1 + (m_{2} - 1) ρ_{2}]$ .

Conduct

In conducting a simple RCT, very often patients are enrolled sequentially. Therefore, block randomization is usually applied to ensure that the comparison arms are having balanced number of patients at any given time.¹⁹

On the other hand, in conducting a cluster RCT, very often all the clusters are enrolled at the same time and randomized into comparison arms at the beginning of the study, and then individual patients are enrolled sequentially within clusters. Therefore, block randomization is not needed, while balance checking at the cluster level right after randomization is very important. For this reason, in practice, we may consider rerandomization to improve balance in a cluster RCT before the physical experiment takes place.²⁰

We discuss more about the advantages of rerandomization in the conduct of cluster RCT. Randomization at the cluster level plays an important role in the conduct of cluster RCT. However, if in a particular experiment, randomization creates two arms that are notably unbalanced on some important cluster-level covariates (such as cluster size – big or small, place of cluster – rural or urban), should we proceed with the experiment? In practice, it would be better to consider rerandomizing and conducting the experiment on balanced arms. Checking covariates and rerandomizing when needed for balance has been advocated repeatedly for traditional RCTs already. For example, Rubin suggests that if ‘important imbalances exist, rerandomize, and continue to do so until satisfied’.²¹ Similarly, for cluster RCTs, we also want to advocate that we should check cluster-level covariates after randomization and consider rerandomization if notable imbalance is detected, and we should focus on those pre-specified cluster-level covariates that are believed to be associated with the outcome variable. Standardized mean difference has been suggested for checking imbalance of individual baseline characteristics with 0.1 as threshold;²² standardized mean difference can also be used for checking imbalance of cluster-level covariates with a larger threshold, say 0.2.²³ In practice, we may consider stratified randomization to further ensure that the clusters are balanced for some key cluster-level features such as cluster size.

We present two additional practical considerations. First, in some practical cases, the number of clusters is small, and therefore simple randomization cannot be relied upon to balance key cluster-level features across the comparison arms and rerandomization can be very time-consuming. For these cases, we can consider constrained randomization, an allocation technique proposed for ensuring balance.²⁴ Second, in conducting a cluster RCT, we want to make sure we balance both key cluster-level variables and key individual-level variables, making it possible that we are able to perform both the cluster-level inferences and individual-level inferences in the analysis stage.

Analysis

Usually the primary analysis using cluster RCTs is effectiveness comparisons across randomization arms. Such analysis can be divided into cluster level analysis and individual level analysis. Traditionally, analysis has been focused at the cluster level; however, recent advancement in statistics has led to the development of techniques that can incorporate the individual level data.²⁵

The traditional approach to the analysis of cluster RCTs has been to calculate a summary measure for each cluster, such as cluster mean or proportion. Because each cluster then provides only one data point, the cluster level data are independent, allowing standard methods to be used without worrying about the ICC. However, in cases where missing data are an important problem and we need to deal with missing data, even for the cluster level analysis, we still need to worry about the ICC issue. The simplest way to deal with missing is that we exclude missing data and calculate the summary measure for each cluster using the observed data. An explicit assumption for doing this is missing complete at random.²⁶ A better way to deal with missing data is using the multiple imputation approach or mixed-effect models,²⁷ under the missing at random assumption.²⁶ Furthermore, we should also conduct sensitivity analysis for missing not at random.²⁶ There is systematic review on statistical analysis and handling of missing in cluster RCTs.²⁸

Recent developments in the statistical field now allow all the individual-level data to be utilized, while account for the ICC, thus increasing the validity of the analysis. A recent review showed that only 51.2% of the articles they reviewed reported exclusively appropriate methods for analysis.⁷ Broadly speaking, there are three categories of approaches to dealing with clustering in regression models:²⁹ (1) Introduce random effects to account for clustering; (2) Introduce fixed effects to account for clustering; (3) Ignore clustering, but be a ‘clever ostrich’. The second category of approaches is to use a fixed effect for each cluster, so they are useful when the number of clusters is small. The third category of approaches is to ignore clustering in the data (i.e. bury head in the sand), proceed with analysis as though all observations are independent, and then use the generalized estimating equation method to ensure valid inferences based on sandwich-type standard errors.³⁰ Next, we describe in more detail the first category of approaches.

Mixed-effects models (with both random effects and fixed effects) are the most popular approaches for analysing data from cluster RCTs.³¹ The two-level linear mixed-effects model for $Y_{i j}$ is given by

Y_{i j} = X_{i j} β + Z_{i j} b_{i} + e_{i j}

where $i = 1, \dots, n$ index clusters and $j = 1, \dots, m_{i}$ index individuals within clusters. Here $X_{i j}$ is a vector of covariates associated with the $j^{th}$ individual from the $i^{th}$ cluster, corresponding with fixed effect $β$ , and $Z_{i j}$ is a design vector for the random effects $b_{i}$ formed from a subset of the appropriate components of $X_{i j}$ . Assume that random residual components $e_{i j}$ are independent with mean zero and variance $V a r (e_{i j}) = σ^{2}$ , random effects $b_{i}$ are independent at the cluster level with mean zero and variance $V a r (b_{i}) = G$ , and these two are independent with $C o v (e_{i j}, b_{i}) = 0$ . For illustration, consider the following simple mixed-effects model

Y_{i j} = β_{0} + β_{1} X_{i j 1} + \dots + β_{p} X_{ijp} + b_{i} + e_{i j}

where the regression coefficient $β$ are the fixed effects that describe the effects of covariates on the outcome variable, and $C o v (Y_{i j}, Y_{i j'}) = V a r (b_{i}) / [V a r (b_{i}) + V a r (e_{i j})]$ measures the correlation for a pair of individuals from a cluster, which is the ICC. Such mixed-effects models can be easily implemented using SAS software (PROC MIXED) and R software (package ‘nlme’).³² Another advantage of using mixed-effects models that they can deal with missing data automatically under the missing at random assumption. Furthermore, we should conduct sensitivity analysis for missing not at random.³³

Report

When we report the results from the analysis a cluster RCT, usually we can follow a three-step process, descriptive analysis, analysis from a simple test, and analysis from a regression model.

When we report the results from descriptive analysis, we should distinguish between cluster-level descriptive analysis and individual-level descriptive analysis. For example, since the randomization is at the cluster level, the primary descriptive analysis is balance evaluation across comparison arms at the cluster level. Moreover, balance evaluation across comparison arms at the individual level is also useful, because usually the primary objective is to evaluate the intervention effectiveness at the individual level.

When we report the results from a simple test, we may examine and emphasize the impact the variance inflation factor. For example, we want to evaluate the intervention effectiveness using t test to compare two arms. We may first calculate the t-test statistic ignoring the ICC (say, $t_{0})$ , and then divide the calculated t-test statistic by the variance inflation factor (that is, $t = t_{0} / I F)$ . This practice helps the readers understand the impact of the ICC, which would further help them when they design their own cluster RCTs.

When we report the results from a regression model, we should distinguish between cluster-level results and individual-level results. Usually, the results related to fixed effects are at individual level and the results related to random effects are at cluster level.³⁴ Moreover, reporting a model-based estimate of the ICC, along with some confidence interval, is extremely important. Remember that at the design stage when we want to calculate the required sample size, we need to have a good estimate of the ICC $ρ$ . Usually, the clinicians and statisticians who are planning their own cluster RCTs would like to be able to find some estimate of the ICC based on literature review or pilot study. Therefore, if we can report an estimate of the ICC when we complete our cluster RCT, it is extremely helpful for them to conduct sample size calculation in the planning of their own cluster RCTs in the future.

In addition, since there are a variety of methods for estimating ICC, including analysis of variance, generalized estimating equations, and mixed-effect models,⁴ we should also report the method behind the ICC estimate and report both the point estimate and its uncertainty measure (say, 95% confidence interval estimate).

Discussion

Cluster RCTs are a popular design we can use when we want to conduct pragmatic clinical trials to generate real-world evidence and the investigative intervention is operating at a group level. The ICC is a key feature of a cluster RCT. In this article, we present some key practical considerations related to the ICC. Justification of the ICC will be particularly important for industry sponsors in discussions of proposed designs with regulatory agencies. We discuss these considerations in all the four stages, design, conduct, analysis, and report. Our aim here is not to provide a comprehensive review of the design and analysis features of cluster RCTs, which has been provided in the literature.^3,4 Instead, our aim is to provide a set of key practical considerations to understand the essential differences between cluster RCT and traditional RCTs, which are caused by the existence of ICC, from the planning stage to the reporting stage. We hope that this brief review, paired with the existing comprehensive review of the developments in cluster RCTs^3,4 will lead to continued improvements in the design, conduct, analysis, and report of cluster RCTs.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This manuscript was supported by AbbVie. AbbVie participated in the review and approval of the content. Yixin Fang and Weili He are employees of AbbVie Inc. and may own AbbVie stock.

ORCID iD

Yixin Fang

References

Ford

Norrie

. Pragmatic trials. N Engl J Med 2016; 375(5): 454–463.

Murray

Varnell

Blitstein

. Design and analysis of group-randomized trials: a review of recent methodological developments. Am J Public Health 2004; 94(3): 423–432.

Turner

Gallis

, et al. Review of recent methodological developments in group-randomized trials: part 1 – design. Am J Public Health 2017; 107(6): 907–915.

Turner

Prague

Gallis

, et al. Review of recent methodological developments in group-randomized trials: part 2 – analysis. Am J Public Health 2017; 107(7): 1078–1086.

Cook

Delong

Murray

, et al. Statistical lessons learned for designing cluster randomized pragmatic clinical trials from the NIH Health Care Systems Collaboratory Biostatistics and Design Core. Clin Trials 2016; 13(5): 504–512.

Torgerson

. Contamination in trials: is cluster randomisation the answer? BMJ 2001; 322(7282): 355–357.

Murray

Pals

George

, et al. Design and analysis of group-randomized trials in cancer: a review of current practices. Prev Med 2018; 111: 241–247.

Sim

Dawson

. Informed consent and cluster-randomized trials. Am J Public Health 2012; 102(3): 480–485.

Killip

Mahfoud

Pearce

. What is an intracluster correlation coefficient? Crucial concepts for primary care researchers. Ann Fam Med 2004; 2(3): 204–208.

10.

Wittleder

Ajenikoko

Bouwman

, et al. Protocol for a cluster-randomized controlled trial of a technology-assisted health coaching intervention for weight management in primary care: the GEM (goals for eating and moving) study. Contemp Clin Trials 2019; 83: 37–45.

11.

Islam

Gepts

Lief

, et al. Protocol for the CHORD project (community health outreach to reduce diabetes): a cluster-randomized community health worker trial to prevent diabetes. BMC Public Health 2018; 18(1): 521.

12.

Wittleder

Smith

Wang

, et al. Peer-assisted lifestyle (PAL) intervention: a protocol of a cluster-randomised controlled trial of a health-coaching intervention delivered by veteran peers to improve obesity treatment in primary care. BMJ Open 2021; 11(2): e043013.

13.

Hayes

Moulton

. Cluster randomised trials. Boca Raton, FL: Chapman and Hall/CRC Press, 2009.

14.

Laird

Ware

. Random-effects models for longitudinal data. Biometrics 1982; 38(4): 963–974.

15.

Ukoumunne

Gulliford

Chinn

. A note on the use of the variance inflation factor for determining sample size in cluster randomized trials. J R Stat Soc Ser D Stat 2002; 51(4): 479–484.

16.

Hemming

Girling

Sitch

, et al. Sample size calculations for cluster randomised controlled trials with a fixed number of clusters. BMC Med Res Methodol 2011; 11: 102.

17.

van Breukelen

Candel

. Calculating sample sizes for cluster randomized trials: we can keep it simple and efficient! J Clin Epidemiol 2012; 65(11): 1212–1218.

18.

Donner

Klar

. Pitfalls of and controversies in cluster randomization trials. Am J Public Health 2004; 94(3): 416–422.

19.

Turner

Thompson

Spiegelhalter

. Prior distributions for the intracluster correlation coefficient, based on multiple previous estimates, and their application in cluster randomized trials. Clin Trials 2005; 2(2): 108–118.

20.

Efird

. Blocked randomization with randomly selected block sizes. Int J Environ Res Public Health 2011; 8(1): 15–20.

21.

Morgan

Rubin

. Rerandomization to improve covariate balance in experiments. Ann Statist 2012; 40(2): 1263–1282.

22.

Rubin

. For objective causal inference, design trumps analysis. Ann Appl Stat 2008; 2(3): 808–840.

23.

Austin

. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res 2011; 46(3): 399–424.

24.

Stuart

. Matching methods for causal inference: a review and a look forward. Stat Sci 2010; 25(1): 1–21.

25.

Lokhnygina

Murray

, et al. An evaluation of constrained randomization for the design and analysis of group-randomized trials. Stat Med 2016; 35(10): 1565–1579.

26.

Campbell

Mollison

Steen

, et al. Analysis of cluster randomized trials in primary care: a practical approach. Fam Pract 2000; 17(2): 192–196.

27.

Little

Rubin

. Statistical analysis with missing data, vol. 793. New York: John Wiley & Sons, 2019.

28.

Rubin

. Multiple imputation after 18+ years. J Am Stat Assoc 1996; 91(434): 473–489.

29.

Fiero

Huang

Oren

, et al. Statistical analysis and handling of missing data in cluster randomized trials: a systematic review. Trials 2016; 17: 72.

30.

Fitzmaurice

Laird

Ware

. Applied longitudinal analysis, vol. 998. New York: John Wiley & Sons, 2012.

31.

Fitzmaurice

. Overview of methods for analyzing cluster-correlated data. Boston, MA: Harvard School of Public Health, 2005.

32.

Twisk

JWR

. Applied mixed model analysis: a practical guide. Cambridge: Cambridge University Press, 2019, p. 249.

33.

West

Welch

Galecki

. Linear mixed models: a practical guide using statistical software. 2nd ed. Boca Raton, FL: CRC Press, 2014, p. 442.

34.

Fiero

Hsu

C-H

Bell

. A pattern-mixture model approach for handling missing continuous outcome data in longitudinal cluster randomized trials. Stat Med 2017; 36(26): 4094–4105.