Random effect misspecification in stepped wedge designs

Abstract

Stepped wedge cluster randomized trials are often analysed using linear mixed effects models that may include random effects for cluster, time and/or treatment. We investigate the impact of misspecification of the random effects structure of the model. Specifically, we considered two cases of misspecification of the random effects in a cross-sectional stepped wedge cluster randomized trials model – fit a linear mixed effects model with random time effects but the true model includes random treatment effects (case 1) or fit a linear mixed effects model with random treatment effect but the true model includes random time effects (case 2) – and derived the variance of the estimated treatment effect under misspecification. We defined two measures of the effect of misspecification: validity and efficiency. Validity is the ratio of the model-based variance of the treatment effect from the mis-specified model divided by the true variance of the treatment effect from the mis-specified model (based on a sandwich estimate of the variance). Efficiency is the ratio of the model-based variance of the treatment effect from the correctly specified model divided by the true variance of the treatment effect from the mis-specified model. We found that validity is less than 1.0 (anti-conservative) in almost all situations investigated with the exception of case 1 with two sequences, when validity could be greater than 1.0. Efficiency is less than 1 in all cases and depends on the intracluster correlation coefficient, the relative magnitude of the variance of the misclassified variance component, and the number of sequences. In general, there is no universal recommendation as to the most robust approach except for the case of a classic stepped wedge cluster randomized trial with only 2 sequences, where fitting a random time model is less likely to lead to anti-conservative inference compared with fitting a random intervention model.

Keywords

Stepped wedge model misspecification random effects variance estimation model selection

Introduction

Cluster randomized stepped wedge designs implement an intervention across participating clusters in phases (Figure 1). Following a period of baseline data collection, clusters are randomly assigned to ‘sequences’ (with 1 or more clusters to a sequence) and the intervention is implemented at a different time in each sequence (sequence 1 clusters implement the intervention at time 1, sequence 2 clusters at time 2, etc.). Typically, once the intervention is implemented, it remains ‘on’. The wedge-shaped appearance of the staggered intervention start times across the clusters gives the design its name.^1,2

A consequence of this design is that the intervention effect in a stepped wedge trial (SWT) is partly confounded with time – early on in the trial, most clusters remain in the standard-of-care condition, and later on, most clusters are in the intervention condition. Thus, any analysis of the intervention effect must adjust for time in some fashion. In addition, the clustered nature of the design (including multiple observations over time within a given cluster and, possibly, repeated observations on individuals) requires careful consideration of the correlation structure of the observations. Often, adjustments for time and correlation are modelled using a linear mixed effects modelling framework.^3–6 In this presentation, we investigate the impact of misspecification of the random effects structure of the model.

Mixed effects models for the analysis of SWTs

We consider cross-sectional SWT designs with M unique treatment sequences, each observed over J time points. We assume that each sequence contains N clusters and each cluster contains K (different) individuals at each point in time. We assume that each individual is observed only once.

Let $Y_{ijk}$ denote the response recorded for individual $k$ from cluster $i$ at time $j$ , where $k = 1, \dots, K$ , $j = 1, \dots, J$ , and $i = 1, \dots, M \times N$ . Also, let $X_{ij}$ have a value of 1 if cluster $i$ is assigned to treatment at time $j$ , and 0 otherwise. Define the vector of outcomes from cluster $i$ as $Y_{i} = (Y_{i 11}, Y_{i 12}, \dots, Y_{iJK})^{T}$ . Then, for a normal outcome and identity link, the linear mixed effects model can be written as

Y_{i} = X_{i} Φ + Z_{i} a_{i} + ϵ_{i}

(1)

where $X_{i}$ is the design matrix for the fixed effects (here, we include treatment and time modelled as a linear trend; the latter choice is made for convenience to keep the number of fixed effect parameters constant across different SWT designs and has minimal impact on the generalizability of our findings), $Φ$ is the vector of fixed effect coefficients, $Z_{i}$ is the design matrix for the random effects, $a_{i}$ is the vector of random effects, and $ϵ_{i}$ is the residual variance. We assume the vector $ϵ_{i}$ has a $M V N (0, σ^{2} I_{JK})$ distribution, where $I_{JK}$ represents an identity matrix of dimension JK. Throughout, we assume that $ϵ_{i}$ and $a_{i}$ are independent. We also assume that $a_{i} ~ M V N (0, G)$ . $Z_{i}$ , $a_{i}$ and $G$ all depend on which random effects are included in the model.

In addition to a random cluster effect, denoted by $u_{i}$ , models for the analysis of SWT often include random effects for time and/or treatment to account for random variations in the time trend or treatment effect, respectively, between clusters.^7,8 In the random time effect model, $a_{i} = (u_{i}, w_{i 1}, \dots, w_{iJ})^{T}$ , where $w_{ij}$ is the random time effect (here, time is treated as categorical for maximum flexibility). The covariance matrix of the random effects is $G = diag (τ^{2}, γ^{2}, \dots, γ^{2})$ , a diagonal matrix with dimension J + 1. The within-period intracluster correlation (wpICC) is frequently used to summarize the dependence between observations within the same cluster at the same time.⁷ In this model, the wpICC is $(τ^{2} + γ^{2}) / (σ^{2} + τ^{2} + γ^{2})$ , which represents a ratio of the between-cluster and total variances. In the random treatment effect model, $a_{i} = (u_{i}, v_{i})^{T}$ , where $v_{i}$ is the random treatment effect. The covariance matrix of the random effects is $G = diag (τ^{2}, η^{2})$ . Sometimes random intercept and treatment effects are allowed to be correlated,⁵ which involves an additional parameter. In a model with a random treatment effect, wpICC is not well defined because clusters have different correlations while on control and treatment. For convenience, we will define the average cluster correlation (ACC) as $(τ^{2} + (1 / JM) (\sum_{m = 1}^{M} T_{m}) η^{2}) / (σ^{2} + τ^{2} + (1 / JM) (\sum_{m = 1}^{M} T_{m}) η^{2})$ where $T_{m}$ is the number of time periods assigned to intervention for sequence m, which represents a ratio of the average of the between-cluster and total variances when $X_{ij} = 0$ and $X_{ij} = 1$ , weighted by the proportion of cluster-time spent on treatment. For a classic design where there are an equal number of cluster-periods on and off treatment, the expression simplifies to ACC = $(τ^{2} + η^{2} / 2) / (σ^{2} + τ^{2} + η^{2} / 2)$ . We will use ACC to refer collectively to ACC in the random treatment effect model and wpICC in the random time effect model. Note that in a random time effect model, since correlation is constant over all clusters and time points, calculating an ACC in an analogous way produces the wpICC.

Model misspecification in SWT

We considered two cases of misspecification of the random effects in an SWT model. In the first case, the researcher fits the random time effect model, but the true model is the random treatment effect model (case 1). In the second case, the researcher chooses the random treatment effect model, but the true model is the random time effect model (case 2).

In general, the goal of an SWT is to estimate the treatment effect. Specifically, one would like an (asymptotically, at least) unbiased estimate of the treatment effect and a correct estimate of the sampling variance of the estimated treatment effect. In addition, one would prefer an estimate of the treatment effect with the smallest possible variance to increase precision and power. When the model is correctly specified, the maximum likelihood estimator satisfies these criteria. However, under a mis-specified model the maximum likelihood estimator of the parameters converges to the value $α^{*}$ that satisfies

\underset{n \to \infty}{l i m} E_{α_{t}} [\sum_{i = 1}^{N M} \frac{\partial}{\partial α} l o g p_{α} (Y_{i} | X_{i}) |_{α^{*}}] = \vec{0}

(2)

where $α_{t}$ represents the true values of the (correctly specified) parameters and $p_{α} (Y_{i} | X_{i})$ is the marginal (mis-specified) likelihood for cluster i.⁹ While the fixed effects remain unbiased,⁶ equation (2) yields multiple roots for the estimated random effects under misspecification. We found solutions for the roots of this system of equations, which then allowed us to assess the (asymptotic) variance of the treatment effect estimate under the different misspecification scenarios.

To evaluate the variance of the treatment effect estimate under misspecification according to the two criteria noted above (correct variance and minimum variance), we defined two measures of the effect of misspecification: validity and efficiency. Validity is defined as the ratio of the model-based variance of the treatment effect from the mis-specified model divided by the true variance of the treatment effect from the mis-specified model (i.e. the variance of the treatment effect estimate over repeated sampling when the data are generated from the true model and the treatment effect is estimated using the mis-specified model). The latter is based on a sandwich estimate of the variance, which provides a consistent estimate of the true variance of the estimated treatment effect under misspecification of the variance components.¹⁰ That is

V a l i d i t y = \frac{{(\sum_{i = 1}^{M N} X_{i}^{T} \sum_{m} {(α *)}_{i}^{- 1} X)}^{- 1}}{A^{- 1} B A^{- 1}}

where $Σ_{m} (α)_{i} = Cov (Y_{i})$ based on the mis-specified model and variance components α, $A = \sum_{i = 1}^{MN} X_{i}^{T} Σ_{m} (α^{*})_{i}^{- 1} X_{i}$ and $B = \sum_{i = 1}^{MN} X_{i}^{T} Σ_{m} (α^{*})^{- 1} Σ_{t} (α_{t})_{i} Σ_{m} (α^{*})^{- 1} X_{i}$ .

Drawing on the concept of asymptotic relative efficiency, efficiency is defined as the ratio of the model-based variance of the treatment effect from the correctly specified model divided by the true variance of the treatment effect from the mis-specified model. That is

E f f i c i e n c y = \frac{{(\sum_{i = 1}^{MN} X_{i}^{T} \sum_{t} {(α_{t})}_{i}^{- 1} X_{i})}^{- 1}}{A^{- 1} B A^{- 1}}

where $α_{t}$ is the true value of variance components and $Σ_{t} (α)_{i} = Cov (Y_{i})$ based on the true model and variance components α. Values of validity may be greater (mis-specified model yields conservative inference) or less (anti-conservative) than 1.0; efficiency is always less than 1.0.

We summarize our findings in Table 1. Further details are available in Voldal et al.¹¹ In case 1 (true model is random intervention but random time is fit), validity can be greater than (2 sequences) or less than (>2 sequences) 1.0 depending on the number of sequences. Validity deviates further from 1.0 as the ACC increases, as K increases and usually as the proportion of the cluster variance attributable to the mis-specified variance component increases. In case 2 (true model is random time but random intervention is fit), validity is less than 1.0 in all cases examined. Again, validity deviates further from 1.0 as the ACC increases, as K increases and as the proportion of the variance attributable to the mis-specified variance component increases. Thus, for classic SWT designs (except for the special case of 2 sequences), there is no modelling approach that will guarantee nominal or conservative inference in the face of random effect misspecification. In addition, some of these trends are sensitive to details of the SWT design, so do not hold universally.

Table 1.

Summary of findings for the classic stepped wedge design.

	Case 1: True model is random intervention; random time is fit	Case 2: True model is random time; random intervention is fit
Validity	•1 for 2 sequences; <1 for more than 2 sequences •Further from 1 as K increases •Further from 1 as ACC increases •Further from 1 as the percent of total variance attributable to the mis-specified variance component increases (except for the 2 sequence case where the relationship depends on the ACC)	•<1 for any number of sequences •Further from 1 as K increases •Further from 1 as ACC increases •Further from 1 as the percent of total variance attributable to the mis-specified variance component increases
Efficiency	•Declines as ACC increases •Declines as the percent of total variance attributable to the mis-specified variance component increases •When the percent of total variance attributable to the mis-specified variance component is large, efficiency declines with fewer sequences; when the percent of total variance attributable to the mis-specified variance component is small, efficiency declines with more sequences	•Declines as ACC increases •Declines as the percent of total variance attributable to the mis-specified variance component increases •Number of sequences has smaller effect on efficiency

ACC: average cluster correlation.

As noted above, efficiency is always less than 1.0 (less efficient) under model misspecification. Generally, efficiency decreases (resulting in lower power/precision) with increasing ACC and as the proportion of the cluster variance attributable to the mis-specified variance component increases. In case 1, designs with more sequences generally (but not always) have less efficiency loss than designs with fewer sequences. In case 2, number of sequences has less impact on efficiency.

Conclusion

The analytic framework developed here allows one to quickly calculate the validity and efficiency of treatment effect estimates under different scenarios of misspecification without simulations. This can help a researcher understand the risks of choosing one model over another, particularly when convergence issues prevent fitting a model that includes all possible random effects.

In general, there is no universal recommendation as to the most robust approach, although one model may be more robust than the other for particular SWT designs (e.g. in a classic 2-sequence design, fitting a random time model is less likely to lead to anti-conservative inference compared with fitting a random intervention model; note that this may not be true for non-classic designs, however).¹¹ Our results also suggest that using robust variances for inference is recommended when possible.

Footnotes

Acknowledgements

This is a summary of a presentation made at the University of Pennsylvania’s 13th Conference on Statistical Issues in Clinical Trials – Cluster Randomized Clinical Trials: Opportunities and Challenges. The material is authored by the current authors and accepted in Statistics in Medicine.¹¹

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Research reported in this paper was supported in part by the National Institutes of Health under award number AI29168. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

ORCID iD

Emily C Voldal

References

Barker

McElduff

D’Este

, et al. Stepped wedge cluster randomised trials: a review of the statistical methodology used and available. BMC Med Res Methodol 2016; 16: 69.

Hughes

Granston

Heagerty

. Current issues in the design and analysis of stepped wedge trials. Contemp Clin Trials 2015; 45(PtA): 55–60.

Hussey

Hughes

. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007; 28(2): 182–191.

Thompson

Fielding

Davey

, et al. Bias and inference from misspecified mixed-effect models in stepped wedge trial analysis. Stat Med 2017; 36(23): 3670–3682.

Hemming

Taljaard

Forbes

. Modeling clustering and treatment effect heterogeneity in parallel and stepped-wedge cluster randomized trials. Stat Med 2018; 37(6): 883–898.

Verbeke

Molenberghs

. Linear mixed models for longitudinal data. New York: Springer, 2000.

Hemming

Taljaard

Forbes

. Analysis of cluster randomised stepped wedge trials with repeated cross-sectional samples. Trials 2017; 18: 101.

Korevaar

Kasza

Taljaard

, et al. Intra-cluster correlations from the Clustered Outcome Dataset (CLOUD) bank to inform the design of longitudinal cluster trials. Clin Trials 2021; 18: 529–540.

Heagerty

Kurland

. Misspecified maximum likelihood estimates and generalised linear models. Biometrika 2001; 88(4): 973–985.

10.

White

. Maximum likelihood estimation of misspecified models. Econometrica 1982; 50: 1–25.

11.

Voldal

Xia

Kenny

, et al. Model misspecification in stepped wedge trials: random effects for time or treatment. Stat Med. Epub ahead of print 8 February 2022. DOI: 10.1002/sim.9326.