Optimal two-stage sampling for mean estimation in multilevel populations when cluster size is informative

1 Introduction

For the purpose of estimating the mean or prevalence of an outcome variable (e.g. alcohol consumption or smoking) in a hierarchical population (e.g. students within schools, patients within general practices), or of comparing subpopulations with respect to such a mean or prevalence, it is often convenient, for economic or logistic reasons, to sample in two stages: first, clusters (e.g. schools, general practices) are sampled and then individuals (e.g. students, patients) are drawn from the sampled clusters.^1–3 Examples of these multi-stage sampling designs include school-based surveys for monitoring substance use among adolescents,^4–6 and national surveys for estimating the average length of stay for discharges from hospitals, ⁷ or nursing homes. ⁸ The topic of this paper is the efficient design of two-stage sampling (TSS) schemes for estimating the mean of a quantitative outcome variable in a two-level population.

In practice, clusters usually vary in size (e.g. small versus large schools) and then, to estimate the population mean, a sample can be drawn with at least three alternative TSS schemes: sampling clusters with probability proportional to cluster size, and then sampling the same number of individuals from each selected cluster (TSS1); sampling clusters with equal probability, and then sampling the same percentage of individuals from each sampled cluster (TSS2); sampling clusters with equal probability, and then sampling the same number of individuals per cluster (TSS3). These three TSS schemes will be considered in this paper and compared with Simple Random Sampling (SRS) of individuals.

Additionally to cluster size variation, further complications arise with informative cluster sizes, that is, when cluster size is related to the outcome of interest. ⁹ , ¹⁰ For instance, cluster size is informative when the amount of alcohol consumed by an adolescent is related to the number of students enrolled in the school, as small schools might provide a more supportive environment,^11–13 or when the number of patients registered to a general practice affects its efficacy in preventing expensive hospitalisations, ¹⁴ thus impacting on public expenditure on health per patient. Informative cluster sizes not only can have direct policy implications, such as introducing a limit to school or general practice size, they also have consequences for statistical data analysis and sample size planning. In informative cluster size literature (see the review by Seaman et al., ⁹ and references therein), the main focus has been on how to handle informative cluster size when the target of inference is the association between the outcome variable and some covariates (e.g. a risk factor). For instance, Seaman et al. ⁹ have discussed several methods to make cluster-specific inferences with Generalized Linear Mixed Models and population-average inferences with Generalized Estimating Equations when cluster size is informative. Innocenti et al., ¹⁵ instead, have investigated a different topic: the implications of informative cluster size for unbiased and efficient estimation of a population mean in surveys conducted with the three aforementioned TSS schemes. The present paper is also about mean estimation for these three TSS schemes when cluster size is informative, but focuses instead on sample size planning, and the consequences of informative cluster size for the required sample sizes and budget.

Innocenti et al.’s results ¹⁵ are the starting point of this paper and therefore summarized here. First, there are two definitions of overall mean in a two-level population, namely the average of all individual outcomes and the average of all cluster-specific means. These two definitions coincide only if cluster sizes are either equal or non-informative. Second, when cluster size is informative, estimation of the mean of all individual outcomes (i.e. the definition used in this paper) is unbiased under TSS1 with the unweighted average of cluster means, and asymptotically unbiased under TSS2 and TSS3 with the average of cluster means weighted by cluster size. In contrast, when cluster size is non-informative, the unweighted average of cluster means is unbiased for all sampling schemes, but optimally efficient for TSS1 and TSS3 only. Third, under the constraint of a fixed total sample size, SRS is more efficient than any TSS scheme, TSS3 is the least efficient TSS scheme, and TSS1 is the most efficient for many cluster size distributions. Indeed, when cluster size is informative, the relative efficiency of these sampling schemes depends on some features of the cluster size distribution in the population, such as the coefficient of variation, the skewness, and the kurtosis. However, when cluster size is non-informative, TSS1 and TSS3 are equally efficient and outperform TSS2. Fourth, the two inferential paradigms in survey sampling, namely the model-based ³ and the design-based approach, ¹ , ² give similar results in terms of unbiased and efficient estimation of the average of all individual outcomes with the three aforementioned TSS schemes, at least if the model assumptions are met. Furthermore, sample size planning and sampling schemes comparisons, which are the topics of this paper, are much more feasible with the assumption of a model for the outcome variable of interest. ¹⁵ For these two reasons, the model-based approach is adopted here.

This work extends the results of Innocenti et al. ¹⁵ in the following ways. First, for each of the three aforementioned TSS schemes, the optimal design is derived. Here, the optimal design is defined as that design (i.e. number of clusters and number of individuals per cluster) that minimizes the sampling variance of the population mean estimator subject to a cost constraint. Second, the three optimal TSS schemes are compared with SRS and with each other under the constraint of a fixed budget. Third, to take care of uncertainty with respect to model parameters and distributional features of cluster size, as a practical alternative, maximin designs are derived. Fourth, sample size calculations for making comparisons between populations are derived and illustrated.

This paper is structured as follows. In section 2, the assumptions of this paper are presented, as well as the sampling schemes and the corresponding mean estimators. Furthermore, the findings of a simulation study to assess the accuracy of some results in Innocenti et al. ¹⁵ that are relevant to the present paper are summarized. In section 3, the optimal design for each TSS scheme is derived, and these optimal TSS designs are compared with each other and with SRS for a fixed budget. Furthermore, the consequences of ignoring informative cluster size at the design phase of a study are investigated. Section 4 deals with the maximin approach, that is, a strategy to solve the dependency of the optimal design on unknown nuisance parameters. Section 5 provides a procedure for computing sample sizes for surveys aimed to make cross-population comparisons, and the procedure is illustrated in planning a survey for comparing the average alcohol consumption among adolescents in France and Italy. Section 6 offers some final remarks. The mathematical derivations of the results, the description of the simulation study discussed in section 2, and additional figures and tables can be found in the Supplementary Material 1 (S.M.1). The Supplementary Material 2 (S.M.2) provides the R ¹⁶ code of the simulation study and other R codes to apply some of the mathematical results of this paper.

2 Assumptions, sampling schemes and mean estimators

The results of Innocenti et al. ¹⁵ and this paper are based on the following assumptions (the notation used in the main text is summarized in the Appendix).

Assumption 1: The population is composed of K clusters and each cluster j contains N j individuals, that is, in the population clusters vary in size ( N j ). The population size is N pop = ∑ j = 1 K N j .

Assumption 2: Sampling is either SRS of individuals in one stage, or else TSS. In TSS, we first sample k clusters, and then sample n or n j individuals per selected cluster j . In case of TSS, the population is very large relative to the sample size at each design level (i.e. k K → 0 and n ¯ θ N → 0 , where n ¯ = ∑ j = 1 k n j k is the average sample size per sampled cluster, and θ N = N pop K is the population mean of cluster size). In case of SRS, N pop is very large relative to m , the number of individuals sampled (i.e. m N pop → 0 ).

Assumption 3: The outcome variable Y i j is quantitative (e.g. alcohol consumption) and measured at the individual level. Further, Y i j shows variation at the cluster level as well as at the individual level. Therefore, sampling error occurs at each design level. This is taken into account by assuming the following two-level random intercept model for the outcome of the i -th individual from the j -th cluster ³ , ¹⁷

Assumption 4: Cluster effect u j is linearly related to cluster size N j , that is, u j = α 0 + α 1 N j + ν j = α 1 N j − θ N + ν j , where α 0 = − α 1 θ N for model identifiability, ν j ∼ N 0 , σ ν 2 , and ν j is the component of cluster effect u j that does not depend on cluster size (i.e. ν j ⊥ N j ). Thus, the conditional distribution of u j given N j is u j | N j ∼ N α 1 N j − θ N , σ ν 2 .

Innocenti et al. ¹⁵ show that β 0 in model (1) is the average of all cluster-specific means in the population, and differs from the average of all individual outcomes in the population μ , unless cluster size is non-informative ( α 1 = 0 ) or constant across clusters, as can be seen from the following expression

With the aim of estimating μ , the three aforementioned TSS schemes are studied in this paper. For each of these TSS schemes and SRS, Table 1 summarizes the sampling procedure (i.e. sample size and inclusion probability per design stage) and the required knowledge before sampling. Furthermore, Table 1 shows the population mean estimator μ ^ and the sampling variance V μ ^ for each sampling scheme. Denote by ρ u N = E u j − E u j N j − E N j σ u σ N = E u j N j − θ N σ u σ N the correlation between u j and N j , where E u j = 0 and V u j = σ u 2 = σ ν 2 + α 1 2 σ N 2 , and by ψ = ρ u N 2 1 − ρ u N 2 the degree of informativeness of cluster size. From Table 1, note that n ¯ = ∑ j = 1 k n j k = n for TSS1 and TSS3, while n ¯ = ∑ j = 1 k n j k = p ∑ j = 1 k N j k = p N ¯ for TSS2, where N ¯ is the average population size of the k sampled clusters (not to be confused with n ¯ , that is, the average sample size of the sampled clusters). Furthermore, for TSS2 n = E n ¯ = p E N ¯ = p θ N . The sampling variances in Table 1 are functions of the total unexplained outcome variance σ y 2 = σ ν 2 + σ ε 2 , the intraclass correlation coefficient ρ = σ ν 2 σ y 2 ∈ 0 , 1 , the sample sizes ( k , n ), the parameter ψ , and some features of the cluster size distribution in the population: the coefficient of variation τ N , the skewness ζ N , and (for TSS2 and TSS3 only) the kurtosis η N . When cluster size is non-informative ( ψ = 0 ), V μ ^ depends only on σ y 2 , ρ , k , n , and (for TSS2 and TSS3 only) τ N . The estimators μ ^ associated with SRS and TSS1 are unbiased, and their sampling variances V μ ^ are exact expressions. ¹⁵

OPEN IN VIEWER

Table 1.

Sampling schemes, required prior knowledge, population mean estimators, and sampling variances.

TSS1	Stage 1	k clusters with probability π j ≈ k N j ∑ j = 1 K N j
	Stage 2	n individuals per sampled cluster with probability π i | j = n N j
	Required prior knowledge	List of all K clusters in the population and their sizes N j . List of all individuals within the k sampled clusters.
	μ ^	∑ j = 1 k y ¯ j k
	V μ ^	σ y 2 n k 1 + ρ n − 1 + n ψ τ N ζ N − τ N + 1
TSS2	Stage 1	k clusters with probability π j = k K
	Stage 2	n j = p N j individuals per sampled cluster with probability π i | j = n j N j = p
	Required prior knowledge	List of all K clusters in the population. List of all individuals within the k sampled clusters.
	μ ^	∑ j = 1 k n j y ¯ j ∑ j = 1 k n j = ∑ j = 1 k p N j y ¯ j ∑ j = 1 k p N j = ∑ j = 1 k N j y ¯ j ∑ j = 1 k N j
	V μ ^	σ y 2 n k 1 + ρ τ N 2 + 1 τ N 2 k + 1 n − 1 + n ψ k − 1 k 2 τ N 2 η N − k − 3 k − 1 + τ N τ N − 2 ζ N + 2 k − 1 k τ N ζ N − τ N + 1
TSS3	Stage 1	k clusters with probability π j = k K
	Stage 2	n individuals per sampled cluster with probability π i | j = n N j
	Required prior knowledge	List of all K clusters in the population. List of all individuals within the k sampled clusters.
	μ ^	∑ j = 1 k N j y ¯ j ∑ j = 1 k N j
	V μ ^	σ y 2 n k τ N 2 + 1 τ N 2 k + 1 + ρ τ N 2 + 1 τ N 2 k + 1 n − 1 + n ψ k − 1 k 2 τ N 2 η N − k − 3 k − 1 + τ N τ N − 2 ζ N + 2 k − 1 k τ N ζ N − τ N + 1
SRS	Stage 1	m individuals with probability π i = m N pop
	Stage 2
	Required prior knowledge	List of all N pop individuals in the population.
	μ ^	∑ i = 1 m y i m
	V μ ^	σ y 2 m 1 + ρ ψ τ N ζ N − τ N + 1

Note: For TSS1, π j ≈ k N j ∑ j = 1 K N j follows from π j = 1 − 1 − N j ∑ j = 1 K N j k if N j ∑ j = 1 K N j → 0 ∀ j = 1 , … , K . For TSS2, n = E n ¯ = p θ N , where n ¯ = ∑ j = 1 k n j k = p ∑ j = 1 k N j k = p N ¯ , and E N ¯ = θ N .

The estimators associated with TSS2 and TSS3 are only asymptotically unbiased, and the corresponding sampling variances are based on first-order Taylor series approximations. ¹⁵ The accuracy of these approximations was evaluated through a simulation study discussed in supplementary material S.M.1 (section 1), but the main findings are summarized here. Sampling k = 20 clusters guarantees nearly unbiased estimates of μ under TSS2 and TSS3 independently of the cluster size distribution, and fair accuracy (i.e. bias ≤ 5 % ) of the variances in Table 1 (TSS2 and TSS3 row) when | ρ u N | ≤ 0.75 , ρ ≤ 0.3 , and ζ N and η N are relatively close (say, ± 1.5 ) to those of the Normal distribution (i.e. ζ N = 0 and η N = 3 ). However, for cluster size distributions with extreme skewness and kurtosis (e.g. ζ N ≥ 2 and η N ≥ 9 ) at least k = 100 clusters must be sampled to achieve a reasonable accuracy (i.e. bias ≤ 6%) of the sampling variances in Table 1, for ρ u N ≤ 0.5 and ρ ≤ 0.3 . Furthermore, the simulations showed that the two lower-bounds for k (i.e. 20 , and 100 ) guarantee the corresponding accuracy level across different values for n (at least for 2 ≤ n ≤ 100 ). To contextualize these two lower-bounds for k , in a school-based survey for studying substance use among adolescents in 21 European countries, Shackleton et al. ¹⁸ have reported that, across countries, k ∈ 36 , 531 (Median  =   123 ) and n ¯ ∈ 5. 92 , 119. 62 (Median  =   20.74 ).

3 Optimal design and relative efficiencies for a given budget

3.1 Optimal design

For any sampling scheme, the precision of the estimator μ ^ , and thus also the width of a confidence interval for μ and the statistical power for testing a hypothesis on μ , depends on the number of clusters and on the sample size per cluster (Table 1). This raises the question of the best combination of sample sizes at each design stage (i.e. sampling many clusters versus sampling many individuals per cluster). Define the optimal design as that design (i.e. number of clusters and number of individuals per cluster), which minimizes V μ ^ subject to a cost constraint, given that time and budget are limited in practice. For TSS, the cost constraint is assumed to be C = k c 2 + c 1 n , where C is the budget for sampling and measuring (excluding costs for constructing the sampling frame and other costs not related to sample size). From now on C is called the research budget. Furthermore, c 2 is the average cost for sampling a cluster, c 1 is the average cost for sampling an individual from a sampled cluster, and c 2 + c 1 n is the cost per cluster including the costs for sampling n individuals from that cluster (recall that for TSS2 n = p θ N ). For SRS, the cost constraint is C = c 0 + c srs m , where m is the number of individuals to sample, c srs is the average cost for sampling an individual directly from the population, and c 0 represents the extra-cost due to constructing the sampling frame for a SRS compared with the sampling frame for a TSS.

For each TSS scheme, the optimal design (i.e. the optimal sample sizes k * and n * ) for estimating μ and the optimal variance V μ ^ * (i.e. V μ ^ under the optimal design) are given in Table 2 (for proofs, see section 2.2 of S.M.1). For TSS2, one can obtain the optimal proportion of individuals to sample per cluster p * from the optimal n * , by dividing n * as given in Table 2 (TSS2 row) by θ N . The optimal TSS2 and TSS3 designs depend on two approximations of V μ ^ : the first-order Taylor approximation mentioned in section 2 and evaluated in S.M.1 (section 1), which underlies the equations in Table 1, and an approximation based on large k (i.e. k such that τ N 2 k ≈ 0 , k − 1 k ≈ 1 , and k − 3 k − 1 ≈ 1 ) to simplify the expressions in Table 1. These two approximations give the following equations (for details, see section 2.1 of S.M.1)

V TSS 2 μ ^ ≈ σ y 2 n k 1 + ρ n τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 − 1

(3)

and

V TSS 3 μ ^ ≈ σ y 2 n k τ N 2 + 1 + ρ τ N 2 + 1 n − 1 + n ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1

(4)

where for TSS2, n = p θ N . Recall from section 2 that, for TSS2 and TSS3, k must be large anyway, because the estimators μ ^ TSS 2 and μ ^ TSS 3 given in Table 1 are only asymptotically unbiased. As a special case, ψ = 0 gives the optimal design and optimal variance for non-informative cluster size (for which case β 0 = μ ), which under TSS1 coincide with the equations available for cluster randomized trials (for instance, see Moerbeek et al. ¹⁹ ). There is no such equivalence under TSS2 due to sample size variation between clusters, and under TSS3 due to weighting cluster means by cluster size if informative cluster size is assumed in the design phase. Indeed, under non-informative cluster size, no weighting is needed under TSS3, ¹⁵ and then the optimal design equations for TSS1 apply to TSS3 as well.

OPEN IN VIEWER

Table 2.

Optimal design and optimal variance V μ ^ * for each sampling scheme.

SRS	V μ ^ *	c srs σ y 2 1 + ρ ψ τ N ζ N − τ N + 1 C − c 0
TSS1	Optimal design	n * = c r 1 − ρ ρ 1 1 + ψ τ N ζ N − τ N + 1 , k * = C c 1 c r + n *
	V μ ^ *	c 1 σ y 2 c r ρ 1 + ψ τ N ζ N − τ N + 1 + 1 − ρ 2 C
TSS2	Optimal design	n * = c r 1 − ρ ρ 1 τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 , k * = C c 1 c r + n *
	V μ ^ *	c 1 σ y 2 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ 2 C
TSS3	Optimal design	n * = c r 1 − ρ ρ τ N 2 + 1 τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 , k * = C c 1 c r + n *
	V μ ^ *	c 1 σ y 2 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ τ N 2 + 1 2 C

Note: Derivations are given in section 2.2 in supplementary material S.M.1. Note that c r = c 2 c 1 > 1 , ζ N ≥ τ N − 1 τ N implies that τ N ζ N − τ N + 1 ≥ 0 that, in turn, entails that 1 + ψ τ N ζ N − τ N + 1 > 0 since ψ ≥ 0 . Note that τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 ≥ 0 for any distribution (for proof, see section 2.1, S.M.1). Recall that for TSS2 n * = p * θ N .

Note from Table 2 that the optimal number of clusters k * and the optimal number of individuals per cluster n * are inversely related, and that n * is an increasing function of the cluster-to-individual cost ratio c r = c 2 c 1 > 1 and a decreasing function of ρ and ψ . These relations between the optimal design and c r , ρ , and ψ hold, under TSS1, for ζ N > τ N − 1 τ N , and always under TSS2 and TSS3 (for proof, see section 2.1 of S.M.1). The condition ζ N > τ N − 1 τ N is met by all the distributions in Tables S.2 and S.7 (S.M.1). Hence, this condition is assumed to be satisfied when considering results for TSS1 in the sequel.

3.2 Effect of cluster size informativeness on the optimal design and study budget needed

The optimal number of individuals per cluster n * for TSS1 and TSS3 is plotted in Figure 1, for two real-life cluster size distributions: the general practice list size distribution in England, and the public high school size distribution in Italy (both distributions are shown in Figure S.1, S.M.1). The behaviour of n * for other cluster size distributions is shown in Figures S.2 and S.4 (S.M.1) for TSS1 and TSS3, respectively, and in Figure S.3 (S.M.1) for TSS2. In most scenarios in Figure 1 and Figures S.2–S.4 (S.M.1), the difference between n * for ψ = 0.35 (i.e. ρ u N = ± 0.51 ) and n * for ψ = 0 (i.e. ρ u N = 0 ) is small, which means that the ratio of V μ ^ under the design assuming ψ = 0.35 to V μ ^ under the design assuming ψ = 0 , when the true ψ = 0.35 , is close to 1. So, the optimal designs in Table 2 are quite robust against misspecification of ψ , in the sense of being efficient relative to the optimal design for the true ψ and given a fixed research budget C . However, ignoring informativeness can lead to serious underestimation of the sampling variance of the mean estimator, and thereby also of the budget needed, as will be seen below. Further, the optimal design depends not only on ψ , but also on ρ and the cluster size distribution ( τ N , ζ N , η N ). That dependence will be addressed in section 4.

OPEN IN VIEWER

Figure 1.

Optimal number of individuals per cluster n* under TSS1 (left column) and TSS3 (right column), as a function of ρ, for different values of c_r and ψ (curves), and different cluster size distributions (rows). The cluster size distributions are shown in Figure S.1 (S.M.1). Note that ψ = 0.35 corresponds to ρ_uN=±0.51.

An example will now show that (a) given a study budget, the optimal design is robust against misspecification of cluster size informativeness, but (b) the budget needed is very sensitive to misspecification. Suppose we plan a survey to estimate μ in the population of all patients of all general practices in England. The parameters of the general practice patient list size distribution are τ N = 0.633 , ζ N = 2.12 , and η N = 14.549 (Table S.2, S.M.1). Furthermore, suppose that ρ = 0.05 , c r = 10 , and C / c 1 = 1000 . The optimal TSS1 samples n * = 10.74 individuals and k * = 48.22 clusters assuming ψ = 1 / 3 , and n * = 13.78 and k * = 42.04 assuming ψ = 0 (see Table 2, TSS1 row). If the true ψ = 1 / 3 , V μ ^ = σ y 2 × 0.00354 for the design correctly assuming ψ = 1 / 3 , and V μ ^ = σ y 2 × 0.00360 for the design incorrectly assuming ψ = 0 (see variance equation in Table 1, TSS1 row), giving a variance ratio 0.00354 / 0.00360 = 0.983 . Additional results for TSS1, TSS2, and TSS3 are given in Table S.8 (S.M.1), which shows that even in some more extreme cases (e.g. ψ = 1 , i.e. ρ u N = ± 0.707 ) the variance ratio still exceeds 0.8. The example given here and those in Table S.8 (S.M.1) show that the optimal designs in Table 2 are quite robust against misspecification of ψ , in the sense of being efficient relative to the optimal design for the true ψ and given a fixed research budget C .

However, ignoring informativeness can lead to serious underestimation of the budget needed. Suppose one wants to test the null hypothesis H 0 that μ = μ 0 against the alternative hypothesis H 1 that μ ≠ μ 0 . The budget that guarantees the desired power level 1 − γ for the chosen type I error rate α , is then obtained by equating V μ ^ * in Table 2 with μ − μ 0 z 1 − γ + z 1 − α 2 2 , where z q is the q th percentile of the standard normal distribution. This gives C = g ρ , ψ z 1 − γ + z 1 − α 2 2 d 0 2 , where g ρ , ψ is the numerator of V μ ^ * in Table 2 excluding σ y 2 , and d 0 = μ − μ 0 σ y is the standardized difference between true mean and mean according to H 0 . Since g ρ , ψ is an increasing function of c 1 , c 2 , and ψ , the required budget C for the desired power level also increases with c 1 , c 2 , and ψ . Likewise, C increases with ρ , at least up to ρ = 0.5 (for proofs, see section 2.2 in S.M.1). The required budget C to detect a standardized difference of medium size ( d 0 = 0.5 ), with 90 % power and two-tailed α = 0.05 , is plotted in Figure 2 for TSS1 and TSS3, as function of ψ , for the general practice list size distribution in England and the public high school size distribution in Italy, and assuming c 1 = 10 . As can be seen in Figure 2, the research budget C is not robust against misspecification of ψ . For example, the required budget C for the optimal TSS1, assuming the English general practice list size distribution, c r = 30 , c 1 = 10 , and ρ = 0.10 (Figure 2, left column, first row), is underestimated by 29% if one incorrectly assumes ψ = 0 when the true ψ = 0.35 . The required budget C is also shown, for other cluster size distributions, in Figures S.5 and S.7 (S.M.1) for TSS1 and TSS3, respectively, and in Figure S.6 (S.M.1) for TSS2. These figures show that C increases with ρ , c 2 , and ψ , and that the impact of the cluster size distribution on C becomes more relevant as ψ increases. Hence, ignoring informative cluster size at the design phase of the survey can lead to underestimating the required budget for the chosen effect size and desired power level. Finally, for the desired power level, the required budget is smallest with the optimal TSS1, and largest with the optimal TSS3.

OPEN IN VIEWER

Figure 2.

Budget C needed for the optimal design to detect a standardized difference between hypothesized and true population mean of medium size (d₀ = 0.5), with 90% power using a two-tailed test with α = 0.05, as a function of ψ, for different values of ρ and c_r (curves) with c₁ = 10, different sampling schemes (columns), and different cluster size distributions (rows). The cluster size distributions are shown in Figure S.1 (S.M.1). Note that ψ∈[0,1.3] corresponds to ρ_uN∈[−0.75,+0.75].

3.3 Relative efficiencies for a given budget

We now compare the efficiency of the optimal designs in Table 2 with each other and with SRS, under the constraint of a fixed research budget. The relative efficiency ( R E ) of the optimal designs for two sampling schemes is defined as the ratio of their optimal variances V μ ^ * in Table 2, more specifically, RE D 1 v s D 2 = V D 2 μ ^ * V D 1 μ ^ * . These R E s are shown in Table 3 (for proofs, see section 2.3, S.M.1), which also gives the sufficient (but not necessary) conditions under which each R E is smaller than one.

OPEN IN VIEWER

Table 3.

Relative efficiencies of TSS schemes versus SRS and each other for a given budget.

D 1 v s D 2	R E D 1 v s D 2 = V D 2 μ ^ * V D 1 μ ^ *	Sufficient (but not necessary) conditions such that R E ≤ 1
TSS 1 v s SRS	1 + ρ ψ τ N ζ N − τ N + 1 c r ρ 1 + ψ τ N ζ N − τ N + 1 + 1 − ρ 2 × c srs c 1 × C C − c 0	ζ N ≥ τ N − 1 τ N − 1 τ N ψ and c srs c 1 = C C − c 0 = 1
TSS 2 v s SRS	1 + ρ ψ τ N ζ N − τ N + 1 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ 2 × c srs c 1 × C C − c 0	ζ N ≤ τ N − 1 τ N or ζ N ≥ τ N + 1 τ N c r − 1 τ N or N j ∼ N θ N , σ N 2 , and c srs c 1 = C C − c 0 = 1
TSS 3 v s SRS	1 + ρ ψ τ N ζ N − τ N + 1 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ τ N 2 + 1 2 × c srs c 1 × C C − c 0	ζ N ≤ τ N − 1 τ N or ζ N ≥ τ N + 1 τ N c r − 1 τ N or N j ∼ N θ N , σ N 2 , and c srs c 1 = C C − c 0 = 1
TSS 2 v s TSS 1	c r ρ 1 + ψ τ N ζ N − τ N + 1 + 1 − ρ 2 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ 2	τ N − 1 τ N − 1 τ N ψ ≤ ζ N ≤ τ N − 1 τ N or ζ N ≥ τ N or N j ∼ N θ N , σ N 2
TSS 3 v s TSS 1	c r ρ 1 + ψ τ N ζ N − τ N + 1 + 1 − ρ 2 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ τ N 2 + 1 2	τ N − 1 τ N − 1 τ N ψ ≤ ζ N ≤ τ N − 1 τ N or ζ N ≥ τ N or N j ∼ N θ N , σ N 2
TSS 3 v s TSS 2	c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ 2 c r ρ τ N 2 + 1 + ψ τ N 4 + τ N 2 η N − 3 + 2 ζ N τ N 1 − τ N 2 + 1 + 1 − ρ τ N 2 + 1 2

Note: Derivations are given in section 2.3 in supplementary material S.M.1. Recall that V μ ^ * is the optimal variance in Table 2, c r = c 2 c 1 > 1 , ψ = ρ u N 2 1 − ρ u N 2 , c srs c 1 ≥ 1 and C C − c 0 ≥ 1 . The conditions for ζ N in the rightmost column are valid for ψ ≠ 0 and are satisfied by all distributions in Table S.7 (S.M.1).

The R E of a TSS scheme compared with SRS (Table 3, first three rows) is composed of three ratios. The first ratio is a function of ρ , ψ , τ N , ζ N , η N , and c r , and is always smaller than one for ψ = 0 , and also for ψ ≠ 0 at least under the conditions for ζ N given in the rightmost column of Table 3 (for proofs, see section 2.3, S.M.1). The other two components of R E TSS v s SRS are the ratio c srs c 1 , for the costs per individual in SRS relative to TSS, and the budget ratio C C − c 0 . Since sampling an individual directly from the population will be more expensive than sampling an individual after having sampled the cluster to which he/she belongs (i.e. c srs > c 1 ), and constructing the sampling frame for a SRS has extra-costs compared with constructing the sampling frame for a TSS (i.e. c 0 > 0 ), the ratios c srs c 1 and C C − c 0 will always be at least one and often larger than one. As a result, the RE can become larger than one, implying that SRS can be less efficient than TSS under the constraint of a fixed budget.

The R E s of the optimal TSS1 and TSS3 versus SRS are shown in Figure 3, for the general practice list size distribution in England and the public high school size distribution in Italy, and assuming c srs c 1 = C C − c 0 = 1 (note that values greater than 1 give a higher R E of TSS versus SRS). Further, Figures S.8–S.10 (S.M.1) show the R E s of the optimal TSS1, TSS2, and TSS3 versus SRS for other cluster size distributions. For ψ = 0 , the R E of any optimal TSS versus SRS is a decreasing function of (i) c r (Table 3), (ii) ρ (at least for ρ ≤ 0.5 , see Figure 3, and Figures S.8–S.10 in S.M.1), and (iii), only for TSS2 and TSS3, τ N (Table 3). For ψ ≠ 0 , the patterns remain almost the same as before and the R E s also do not seem to vary much across cluster size distributions (Figure 3, and Figures S.8–S.10 in S.M.1).

OPEN IN VIEWER

Figure 3.

Relative efficiency of the optimal TSS1 versus SRS (left column), and of the optimal TSS3 versus SRS (right column), for a given research budget C and assuming (c_srs/c₁) = (C/(C-c₀)) = 1 (values greater than 1 give a higher RE of TSS versus SRS), as a function of ρ, for different values of c_r and ψ (curves), and different cluster size distributions (rows). The cluster size distributions are shown in Figure S.1 (S.M.1). Note that ψ  =  0.35 corresponds to ρ_uN = ±0.51.

The R E s of the three TSS schemes compared with each other (Table 3, last three rows) are functions of ρ , c r , ψ , τ N , ζ N , and η N . The optimal TSS2 is more efficient than the optimal TSS3 since R E TSS 3 v s TSS 2 < 1 (unless τ N = 0 , Table 3, or ρ u N ≈ ± 1 , ¹⁵ since in both cases R E TSS 3 v s TSS 2 = 1 ). The R E s of TSS2 and TSS3 versus TSS1 are smaller than one, and so the optimal TSS1 is the most efficient TSS scheme, at least for cluster size distributions satisfying the conditions in Table 3 (rightmost column), such as all distributions in Table S.7 (S.M.1). For other cluster size distributions, one must compute the R E for that particular distribution to see whether R E < 1 . However, for ψ = 0 , the R E s in the last three rows of Table 3 are all smaller than one for any cluster size distribution, making TSS1 the most efficient TSS scheme, followed by TSS2. Note that this only holds if informative cluster size ( ψ ≠ 0 ) is assumed at the design stage, such that in TSS3 cluster means are weighted by cluster size to estimate μ (Table 1). If non-informative cluster size ( ψ = 0 ) is assumed already in the design stage, then no weighting is needed for TSS3, ¹⁵ and TSS3 then is as efficient as TSS1.

The R E of the optimal TSS2 and TSS3 versus the optimal TSS1 are shown in Figure 4, for the general practice list size distribution in England and the public high school size distribution in Italy, and in Figures S.11–S.12 (S.M.1) for other four cluster size distributions. For ψ = 0 , these reduce to R E TSS 2 v s TSS 1 = c r ρ + 1 − ρ 2 c r ρ τ N 2 + 1 + 1 − ρ 2 and R E TSS 3 v s TSS 1 = 1 τ N 2 + 1 , which are both decreasing functions of τ N , but R E TSS 2 v s TSS 1 also decreases as ρ and/or c r increases. For ψ ≠ 0 , the patterns are the same as before with two major differences. First, both R E s decrease as η N increases (Table 3). Second, for ψ = 0.35 , both R E s differ at most 6 % from their values at ψ = 0 (Figure 4, and Figures S.11–S.12 in S.M.1), except for the English general practice (GP) list size distribution that, having an extreme kurtosis (i.e. η N = 14.55 ), shows a drop in R E (compared with the case ψ = 0 ) larger than 20 % . Note that TSS1 is the most efficient design in Figure 4 and Figures S.11–S.12 (S.M.1).

OPEN IN VIEWER

Figure 4.

Relative efficiency of the optimal TSS2 versus the optimal TSS1 (left column), and of the optimal TSS3 versus the optimal TSS1 (right column), for a given research budget, as a function of ρ, for different values of c_r and ψ (curves), and different cluster size distributions (rows). The cluster size distributions are shown in Figure S.1 (S.M.1). Note that ψ = 0.35 corresponds to ρ_uN = ±0.51.

4 Maximin design

In section 3.2 it has been noticed that the optimal designs in Table 2 require a priori knowledge of some nuisance parameters (i.e. ρ , τ N , ζ N , η N , and ψ ). This is known as the local optimality problem in optimal design literature. ²⁰ , ²¹ Basically, this means that the optimal design is optimal only for certain values of these nuisance parameters. In this paper, the local optimality problem is solved taking a maximin approach.^20–22 This approach has been applied in several contexts, such as longitudinal studies,^23–25 fMRI experiments, ²⁶ cluster randomized and multicentre trials,^27–29 cost-effectiveness studies, ³⁰ , ³¹ life-event studies, ³² test construction, ³³ and biological and pharmacological studies.^34–37 The maximin approach is composed of the following steps:

Define the parameter space, that is, for each unknown parameter (i.e. ρ , τ N , ζ N , η N , and ψ ) determine the range of plausible values (e.g. ρ ∈ 0 , 0. 30 ).

The resulting design is called the maximin design, which is the optimal design for the worst-case scenario, as defined by that set of parameter values chosen in step 3. The advantage of the maximin design is that it not only maximizes the efficiency and the power in the worst-case scenario, but it also guarantees at least that same efficiency and power level for all the other parameter values within the parameter space. Indeed, V μ ^ is smaller and the power for hypothesis testing on μ is larger, for all other parameter values than for the worst-case values chosen in step 3, given any fixed sample size (i.e. k and n ).

Following the four steps above, we now explain how to find the maximin design for each sampling scheme. The optimal design for TSS1 depends on ρ , τ N , ζ N , and ψ . However, to draw a TSS1 sample we need to know the cluster size distribution in the population anyway, which means that τ N and ζ N are also known before sampling. Thus, for TSS1, only ρ and ψ are unknown. The maximin design for TSS1 is obtained by plugging into the optimal sample sizes equations (Table 2, TSS1 row) the largest realistic values of ρ and ψ (for proofs, see section 3.1 in S.M.1). Unlike for TSS1, when sampling with TSS2 or TSS3 the researcher needs no prior knowledge of the whole cluster size distribution. Indeed, if such information is available, sampling with TSS1 is a better choice (Table 3). The maximin design for TSS2 and TSS3 is obtained by plugging into the optimal design equations (Table 2) the upper-bounds of the ranges for ρ , ζ N , η N , and ψ , and the worst-case value of τ N (for proofs, see section 3.1 in S.M.1). The latter value can be obtained with an R function given in S.M.2 (section 2), which searches numerically for the value of τ N that maximizes V μ ^ (i.e. equations (3) and (4)) within its range of plausible values, given the worst-case values for ρ , ζ N , η N , and ψ . For several upper-bounds for ρ , ζ N , η N , and ψ , a numerical evaluation was performed and this always gave τ N = 1 as worst-case value of τ N within the range 0 , 1 (for details, see section 3.2 in S.M.1).

To be on the safe side in sample size planning, one can assume for ρ the parameter range 0 , 0.10 in health and medical research, ³⁸ , ³⁹ and 0 , 0.25 in educational research. ¹⁸ , ⁴⁰ Lacking empirical evidence for ψ or ρ u N , we propose ψ ∈ 0 , 0.35 , which corresponds to ρ u N ∈ − 0.51 , 0.51 . The range τ N ∈ 0 , 1 can be justified by considering Table S.7 (S.M.1), and the extreme cases of an exponential cluster size distribution, for which τ N = 1 , and of a binary distribution with half of all clusters having size 2 and the other half having size 2 θ N − 2 , for which τ N ≈ 1 . Finally, for ζ N and η N , the ranges ζ N ∈ 0.5 , 2 and η N ∈ 3 , 15 can be chosen based on Table S.7 (S.M.1). Since V μ ^ under TSS2 and TSS3 is an increasing function of ζ N and η N (at least if τ N ≤ 1 , which will usually hold), assuming positive skewness and positive excess kurtosis (i.e. η N − 3 > 0 ) is a safe choice.

As mentioned in section 3.1, the optimal design for TSS2 and TSS3 depends on two approximations: the first-order Taylor series approximation used to derive V μ ^ for TSS2 and TSS3 in Table 1, and the large k approximation to simplify the equations in Table 1 into equations (3) and (4). Since the maximin design is the optimal design for the worst-case scenario, the same approximations also underlie the maximin design. Based on the simulation study and the numerical evaluation discussed in S.M.1 (sections 1 and 3.3), it turned out that each approximation induces a bias of at most 5 % in the V ( μ ^ ) used to derive the optimal/maximin design if the optimal/maximin k M D ≥ 20 , or, for ζ N ≥ 2 and η N ≥ 9 , k M D ≥ 100 . Since V ( μ ^ ) ∝ 1 k , a simple solution is to increase the maximin k M D with 10 % to ensure sufficient power at the expense of a 10 % higher budget C . However, if the maximin k M D < 20 or (for ζ N ≥ 2 and η N ≥ 9 ) k M D < 100 both approximations are biased by more than 5 % . A solution is to first increase C such that maximin k M D ≥ 20 or (for ζ N ≥ 2 and η N ≥ 9 ) k M D ≥ 100 , and then further increase C by 10 % .

5 Sample size calculation for cross-population comparisons

The results of the previous sections allow to efficiently plan a survey not only for estimating a mean, but also for comparing different populations, if the samples are independent. An example of such a study is the ESPAD study, ⁶ which compares substance use among 15–16-year-old students across 35 European countries. For a fixed separate budget per population, the optimal design per population is given in Table 2 and the maximin design in section 4. However, the design can be further optimized by constraining the total budget (i.e. the sum of the separate budgets) instead of each separate budget and finding the optimal (or maximin) budget split between populations (for details, see section 4 of S.M.1). For the case of comparing two populations, this optimization was formalized into a procedure to compute maximin sample sizes per population and the maximin budget split between populations, obtained by extending Van Breukelen and Candel ²⁸ to TSS1 with informative cluster sizes and different cluster size distributions per population. This procedure for comparing two populations is implemented in an R code given in section 4 in S.M.2. To use this program, the researcher needs to specify c 1 and c 2 per population, τ N and ζ N of the cluster size distribution of each population, the largest plausible values for ρ and ψ , a range for the ratio of the outcome standard deviations ( σ y ) between the two populations, the smallest difference μ F − μ I that is worthwhile being detected, the maximum sum of outcome variances in both populations V max , the power level 1 − γ , and the type I error rate α . The R code (S.M.2, section 4) returns the maximin sample sizes per population and the maximin budget split. The steps of this procedure are given in S.M.1 (section 4). This procedure is presented only for TSS1, because it is the most efficient sampling scheme for many cluster size distributions.

Let us demonstrate the procedure with the following example. Suppose that we want to plan a survey to estimate and compare the average alcohol consumption among high school students between France and Italy. Similar to the ESPAD study, ⁶ alcohol consumption Y i j is measured as the average volume of ethanol (in centilitres) consumed on the last drinking day. Based on adolescent health literature, at the design stage, school size (i.e. total number of students) can be assumed to be informative, that is, related to alcohol consumption. Indeed, it has been found that school size and school connectedness, broadly defined as the degree of belonging at school, are inversely related, ¹² , ¹³ as well as school connectedness and alcohol use. ¹¹ TSS1 is the most efficient two-stage sampling scheme for both high school size distributions (this can be verified by checking the conditions in the rightmost column of Table 3, with the numbers given in the second and third row of Table S.7 of S.M.1), and so it is chosen for both populations. Suppose that we want to test the null hypothesis H 0 that μ F = μ I against the alternative hypothesis H 1 that μ F ≠ μ I , where μ F and μ I are the population means of alcohol consumption in France and Italy, respectively. Since the French and the Italian samples are independent, we can apply the procedure above to determine how many schools and how many students per school one has to sample per country, and how to split the total budget between countries.