Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Abstract

Traditional methods based on the assumption that the treatment distribution is a pure shift of the control distribution may not always hold. The possibility that an individual from the treatment group may not respond to the treatment motivates the use of a mixture distribution for the treatment group. This paper considers two test procedures based on the Wilcoxon rank-sum statistic for a group sequential design to detect the one-sided mixture alternative. Error spending functions are used for the allocation of error rates at each stage. The two tests are evaluated individually in determination of critical values and arm sizes and asymptotic multivariate normality is shown to hold for both. Upon comparison, the tests are presented to be asymptotically equivalent. Both test statistics maintain the Type I error rate even if $F$ is misspecified in the design alternative. A more general definition of the treatment effect is used with the mixture distribution. Method of moments estimators and constrained $k$ -means estimators for the treatment effect are evaluated.

Keywords

Group sequential design mixture model error spending power

1. Introduction

The setting in this article is that of testing the efficacy of a treatment versus a control. Clinical trials are a popular and effective design used to perform these tests while ensuring safety of the participants and general population. Group sequential clinical trials take these ideals a step further by incorporating the possibility of terminating the trial early by rejecting the null hypothesis due to strong evidence of efficacy or accepting the null hypothesis for futility. Instead of a one-time test to evaluate a treatment, group sequential methods conduct hypothesis tests at a predetermined number of stages. That is, a stage represents each time analysis is performed. The trial may terminate at an earlier stage than the final one if there is sufficient evidence to make a decision in the hypothesis test. One way of achieving the boundaries for such tests is through error spending methods. Lan and Demets¹ developed rejection boundaries based on an error spending function that controls the rate at which the Type I error is spent. Similar methods are developed by Chang et al.² to include an error spending function for the Type II error rate.

Traditional methods of evaluating a treatment versus control assume that the treatment effect should be represented by an additive shift of the control distribution. In this pure shift situation, all treated individuals have responses that come from the shifted distribution. However, this may not always be the case. Good³ and Boos and Brownie⁴ both show instances of “nonresponders” to a given treatment. For additional evidence of this phenomenon, see Jeske and Yao⁵ and Peng et al.⁶ We define nonresponders as individuals that have been given the treatment and were unaffected by it. Therefore, this treatment group will have responses that come from the same distribution as the control group responses. The proportion of nonresponders is considered unknown, and with their existence, the treatment distribution will be represented by a mixture of the control distribution and a location shift applied to the control distribution.

For the formulation of hypothesis tests, we define the distribution function of the control group as $F$ and the distribution function of the treatment group as $G$ . The null hypothesis is $H_{0} : F = G$ . In the traditional pure shift setting, the alternative hypothesis is $H_{A} : G (u) = F (u - δ)$ for a specified $δ$ . However, the pure shift is inappropriate with the presence of nonresponders. A new alternative hypothesis that we may refer to as a “mixture alternative” is defined as $H_{A} : G (u) = (1 - θ) F (u) + θ F (u - δ)$ , where $θ \in (0, 1)$ is the proportion of responders and $δ > 0$ is the specified shift size.

Peng et al.⁶ work in this context and develop a group sequential test based on the difference of means. They work under the assumption that the control and treatment group observations follow a normal distribution and a mixture of normal distributions, respectively, but their work can be used for other distribution functions if the central limit theorem is invoked. R code is available from their paper to set up the design for user-inputted choices of various parameters. While working in a fixed sample setting, Jeske and Yao⁵ remove the normal theory assumption in favor of a less restrictive assumption of the control distribution being some continuous cumulative distribution function, $F$ . Their work utilizes the Wilcoxon rank-sum test to determine sample sizes needed to detect a design alternative including the mixture distribution.

In the group sequential setting, there is no literature aimed at detecting the mixture alternative with a general $F$ . Spurrier and Hewett⁷ use the Wilcoxon rank-sum test in two-stage group sequential clinical trials to detect the general alternative $F < G$ , where $G$ is the continuous cumulative distribution function of the treatment group. Shuster et al.⁸ extend the test statistic from Spurrier and Hewett⁷ from two stages to an arbitrary number of stages in tests based on the ordinal categorical data. Shuster et al.⁸ extend the test statistic from Spurrier and Hewett⁷ from two stages to an arbitrary number of stages in tests but for ordinal categorical data. Madsen and Hewett⁹ develop a test based on ranks to detect the pure shift alternative for some continuous $F$ and an arbitrary number of stages. In this article, we look to develop an $S$ -stage group sequential clinical trial test to detect the mixture alternative with some continuous $F$ .

The rest of the article is organized as follows. In Section 2, we introduce our proposed test procedure including discussion of the test statistic, its distribution, the normal approximation of its distribution, and computation of arm size. Section 3 examines the power and robustness of the test procedure. Section 4 investigates a comparison of the test statistic used by Shuster et al.⁸ and the test statistic, we introduce in Section 2. In Section 5, we explore estimation of the treatment effect via extension of the estimator proposed by Jeske and Yao.⁵ We conclude with a brief summary in Section 6.

2. Group sequential average rank procedure

As we look to extend fixed sample methods into the group sequential space, an issue to be determined is how to combine information across the stages in the statistic. In this section, we develop a group sequential average rank (SAR) procedure.

2.1. Test statistic and critical values

The SAR test statistic is calculated by finding the Wilcoxon rank-sum statistic for each stage using only the observations within that stage and averaging these statistics across stages. We will focus on the scenario where the number of observations in the treatment group at each stage, $n$ , are proportional to the number of observations in the control group at each stage, $m$ , so that $m = λ n$ for $λ > 0$ . These values will remain the same for each stage. We subsequently refer to $m$ and $n$ as the “arm size” of the control and treatment group, respectively. Let $S$ be the maximum number of stages in a study, then let $X_{11}, \dots, X_{1 m}, \dots, X_{S 1}, \dots, X_{S m} \overset{i i d}{\sim} F$ represent potential observations from the control group and $Y_{11}, \dots, Y_{1 n}, \dots, Y_{S 1}, \dots, Y_{S n} \overset{i i d}{\sim} G$ represent potential observations from the treatment group. For trials that terminate early, we would not see all $(m + n) S$ observations. We are interested in testing the null hypothesis $H_{0} : F = G$ against the one-sided alternative, $H_{A} : G$ is the mixture distribution. We take $G (u) = (1 - θ) F (u) + θ F (u - δ)$ , where $θ \in (0, 1]$ is the proportion of responders and $δ = K σ_{F}$ is the shift size for $K > 0$ and $σ_{F} > 0$ . With $σ_{F}$ defined as the standard deviation of $F$ , $K$ represents the size of the shift in the terms of number of standard deviations. For identifiability, define the null case as the point $(θ, δ) = (0, 0)$ . For $R_{s 1}, \dots, R_{s n}$ , the standard ranks of the treatment observations amongst the ordered combined control and treatment group observations at stage $s$ , we get the Wilcoxon rank-sum statistic at stage $s$ , $W_{s} = \sum_{j = 1}^{n} R_{s j}$ .¹⁰

In order to determine the mean and variance of the SAR statistic, we must first obtain the mean and variance of $W_{s}$ . We begin with the definitions

\begin{aligned} \begin{aligned} p & = P (X < Y) = \int_{- \infty}^{\infty} F (u) d G (u) \\ p_{1} & = P (X_{1} < Y_{1}, X_{1} < Y_{2}) = \int_{- \infty}^{\infty} [1 - G (u)]^{2} d F (u) \\ p_{2} & = P (X_{1} < Y_{1}, X_{2} < Y_{1}) = \int_{- \infty}^{\infty} [F (u)]^{2} d G (u) \end{aligned} \end{aligned}

(1)

where

X

and

Y

denote random variables representing the control and treatment group observations, respectively. The subscripts indicate whether the probabilities are calculated for the same or different observations. Under the null hypothesis, these probabilities are

p = 1 / 2

and

p_{1} = p_{2} = 1 / 3

.¹¹ Under the mixture alternative

G (u) = (1 - θ) F (u) + θ F (u - δ)

, they can be written as one-dimensional integrals:

\begin{aligned} p & = \int_{- \infty}^{\infty} F (u) [(1 - θ) f (u) + θ f (u - δ)] d u \\ p_{1} & = \int_{- \infty}^{\infty} [1 - (1 - θ) F (u) - θ F (u - δ)]^{2} f (u) d u \\ p_{2} & = \int_{- \infty}^{\infty} [F (u)]^{2} [(1 - θ) f (u) + θ f (u - δ)] d u \end{aligned}

(2)

where

f (u)

is the probability density function of

F

The general forms of the mean and variance, respectively, of $W_{s}$ ¹⁰ are

\begin{aligned} μ_{W} & = n (m p + \frac{(n + 1)}{2}) \end{aligned}

(3)

\begin{aligned} σ_{W}^{2} & = m n (p - (m + n - 1) p^{2} + (n - 1) p_{1} + (m - 1) p_{2}) \end{aligned}

(4)

Let

μ

and

σ^{2}

be the mean and variance, respectively, of

W_{s}

under the null hypothesis. The SAR test statistic at stage

s = 1, \dots, S

Z_{s} = \frac{\frac{1}{s} \sum_{i = 1}^{s} W_{i} - μ}{σ / \sqrt{s}}

The marginal mean and variance of each

Z_{s}

are zero and one, respectively, under the null hypothesis. The general form of the mean and variance of

Z_{s}

are

E [Z_{s}] = \frac{μ_{W} - μ}{σ / \sqrt{s}} Var (Z_{s}) = \frac{σ_{W}^{2}}{σ^{2}}

(5)

Finally, the covariance of the SAR statistic between stages

s

and

s^{'}

s \leq s^{'}

, can be shown to be

Cov (Z_{s}, Z_{s^{'}}) = \frac{σ_{W}^{2}}{σ^{2}} \sqrt{\frac{s}{s^{'}}}

(6)

Note that under the null hypothesis, the covariance only depends on the stage numbers of interest.

When implementing the SAR procedure, we wish to set up a group sequential design to detect a treatment effect. This may be done in a variety of ways including pre-fixed boundaries or error spending boundaries. We will focus on the latter in this article. The design should be able to reject the null hypothesis for efficacy and accept the null hypothesis for futility. This setup will involve both upper and lower critical values unless the preset maximum number of stages is reached, where a single critical value will be used to reach a final accept or reject decision. Figure 1 displays a visual of this process.

The critical values $r_{1}, \dots, r_{S - 1}, a_{1}, \dots, a_{S - 1}$ , and $u$ need to be determined. Here, the $r_{i}$ ’s, $i = 1, \dots, S - 1$ , represent the upper critical values at which we would reject the null hypothesis, the $a_{i}$ ’s, $i = 1, \dots, S - 1$ , represent the lower critical values at which we would accept the null hypothesis, and $u$ is the critical value for the final stage. Thus, after stage $s = 1, \dots, S - 1$ , we make the following comparisons and corresponding decisions

\begin{aligned} if Z_{s} \geq r_{s} reject H_{0}, end experiment \\ if Z_{s} \leq a_{s} accept H_{0}, end experiment \\ otherwise continue to next stage \end{aligned}

After stage S

\begin{aligned} if Z_{S} \geq u reject H_{0}, end experiment \\ if Z_{S} < u accept H_{0}, end experiment \end{aligned}

The critical values are found using error spending functions set by the experimenter at the beginning of the study and the joint distribution of

(Z_{1}, \dots, Z_{S})

. Let functions

f (t)

and

g (t)

be used to set the Type I and Type II error rates at each stage, respectively. These may be any nondecreasing functions that are equal to zero for

t = 0

and that

f (t) = α

and

g (t) = β

for

t \geq 1

. We use

f (t) = α t^{ρ} g (t) = β t^{ρ}

for

t \in (0, 1)

and

ρ > 0

. With equal increases in arm size,

t

is defined at stage

s

t_{s} = s / S

. The experimenter may choose

ρ = 1

for equal spending at each stage and larger

ρ

to have smaller Type I and Type II error rates for early stages. Values such as

ρ = 2

ρ = 3

, may be a desired strategy in order to make it more likely to avoid an early false positive or false negative.

The resulting Type I and Type II error rates for each stage are calculated as follows

\begin{aligned} α_{1} & = f (t_{1}) & β_{1} & = g (t_{1}) \\ α_{s} & = f (t_{s}) - f (t_{s - 1}) & β_{s} & = g (t_{s}) - g (t_{s - 1}) \end{aligned}

where

s = 2, \dots, S

. Then the overall Type I and Type II error rates for the entire study are

α = \sum_{s = 1}^{S} α_{s}

and

β = \sum_{s = 1}^{S} β_{s}

Once the desired error spending functions are set, the critical values are found using the joint distribution of $(Z_{1}, \dots, Z_{S})$ under both the null and alternative hypotheses. The joint distribution and, therefore, the critical values depend on $m$ and $λ$ . The critical values that correspond to the results of the error spending functions can only be achieved upon reaching a necessary arm size. They are chosen to satisfy the following set of equations

\begin{aligned} P_{H_{0}} (Z_{1} \geq r_{1}) & = α_{1} \\ P_{H_{A}} (Z_{1} \leq a_{1}) & = β_{1} \end{aligned}

(7)

For

s = 2, \dots, S - 1

\begin{aligned} P_{H_{0}} (a_{1} < Z_{1} < r_{1}, \dots, a_{s - 1} < Z_{s - 1} < r_{s - 1}, Z_{s} \geq r_{s}) & = α_{s} \\ P_{H_{A}} (a_{1} < Z_{1} < r_{1}, \dots, a_{s - 1} < Z_{s - 1} < r_{s - 1}, Z_{s} \leq a_{s}) & = β_{s} \end{aligned}

(8)

Finally, at stage

S

, we must also have

\begin{aligned} P_{H_{0}} (a_{1} < Z_{1} < r_{1}, \dots, a_{S - 1} < Z_{S - 1} < r_{S - 1}, Z_{S} \geq u) & = α_{S} \end{aligned}

(9)

\begin{aligned} P_{H_{A}} (a_{1} < Z_{1} < r_{1}, \dots, a_{S - 1} < Z_{S - 1} < r_{S - 1}, Z_{S} < u) & = β_{S} \end{aligned}

(10)

The critical values are best found in pairs beginning with

r_{1}

and

a_{1}

then moving through subsequent stages until the final stage is reached. Equations (9) and (10) are solvable only when

m

and

n

are large enough. Therefore, we determine them by iterating until the necessary values are reached.

Under the alternative hypothesis, the joint distribution of the SAR test statistic depends on the choice of $F$ . This is due to the fact that the integrals in equation (2) vary with $F$ . Thus, an $F$ must be chosen when designing the test in order to determine the critical values and arm sizes. However, under the null hypothesis, the joint distribution of the SAR test statistic inherits the distribution-free property of the Wilcoxon rank-sum statistic. This is rooted in the probabilities from equation (1) having the values 1/2, 1/3, and 1/3, respectively, under the null hypothesis for any $F$ . Consequently, the SAR procedure guarantees a test with size $α$ , even if the hypothesized $F$ for the alternative is misspecified.

2.2. Critical values via simulation

We suggest using the simple algorithm visualized in the flowchart in Figure 2 to find the critical values. The algorithm starts at an arm size $m = m_{0}$ which can easily be chosen to be $m = 0$ , and increases $m$ by one for each iteration. Then $n$ is calculated as the smallest whole number larger than $m / λ$ . It will eventually reach the necessary arm sizes to detect the given alternative. However, the arm sizes for small shifts can be large, resulting in many iterations of the algorithm when increasing $m$ by one. Through empirical investigations, the authors have noticed that the arm sizes for two stages are found to be slightly greater than half of the fixed sample arm size, the arm sizes for three stages are slightly greater than one-third of the fixed sample arm size, for four stages are slightly greater than one-fourth, etc. Using this observation together with the fact that one can quickly find the arm sizes for the fixed sample case with the formula from Jeske and Yao,⁵ an improved starting value can easily be found. We implement this approach in the algorithm by starting with $m_{0}$ equal to this approximate value rather than zero to reduce the number of iterations needed to achieve the final result.

The exact joint and marginal distributions of the SAR are discrete with the number of possible values increasing as $m$ and $n$ increase. The discrete nature of the distribution makes achieving the exact Type I and Type II error rates impossible in most situations. Because of this, the procedure will give critical values that correspond to Type I and Type II error rates that are less than or equal to the design alternative values.

2.3. Multivariate normal approximation

The Wilcoxon rank-sum statistic at stage $s$ , $W_{s}$ , used for the SAR is calculated using only the observations at stage $s$ . Therefore, the Wilcoxon rank-sum statistics calculated at each stage are independent. Furthermore, since the arm sizes are the same at stage $s$ and $s^{'}$ , $W_{s}$ and $W_{s^{'}}$ are identically distributed. The distribution of $W_{s}$ is approximately normal for large $m$ and $n$ under both the null and general alternative hypotheses.¹¹

Let $W = (W_{1}, \dots, W_{S})$ , be a vector of the independent Wilcoxon rank-sum statistics calculated at each stage. Under the large sample assumption, any linear combination of $W$ has a univariate normal distribution. Then, $W$ has a limiting $S$ -variate normal distribution with mean vector $μ_{W} 1_{S}$ , where $1_{S}$ is an $S \times 1$ vector of ones, and variance-covariance matrix $σ_{W}^{2} I_{S}$ , where $I_{S}$ is an $S \times S$ identity matrix.¹²

The existence of the multivariate normal approximation for $W$ leads to the same type of result for the SAR statistic. Let

\begin{aligned} A_{S \times S} & = [\begin{matrix} 1 & 0 & 0 & \dots & 0 \\ 1 / \sqrt{2} & 1 / \sqrt{2} & 0 & \dots & 0 \\ ⋮ \\ 1 / \sqrt{S} & 1 / \sqrt{S} & 1 / \sqrt{S} & \dots & 1 / \sqrt{S} \end{matrix}] & b_{S \times 1} & = [\begin{matrix} 1 \\ \sqrt{2} \\ ⋮ \\ \sqrt{S} \end{matrix}] \end{aligned}

Then the SAR statistic can be written in vector form as

Z = \frac{1}{σ} A W - \frac{μ}{σ} b

Since

W

follows a limiting

S

-variate normal distribution,

Z

has a limiting

S

-variate normal distribution with mean vector

(μ_{W} / σ) A 1_{S} - (μ / σ) b

and variance–covariance matrix

(σ_{W}^{2} / σ^{2}) {AI}_{S} A^{⊤}

.¹²

Using the normal approximation to obtain the critical values and arm sizes for a given alternative will follow a similar algorithm as simulating the exact joint distribution, as we need to reach the $m$ and $n$ necessary to detect the alternative. Instead of approximating equations (7) to (10) empirically using simulation, the multivariate normal approximation could alternatively be used.

2.4. Comparison of simulated distribution and normal approximation

Due to the large number of simulations needed for a good simulation approximation of (7) to (10), the multivariate normal approximation offers a great increase in computation speed. Figure 3 highlights this computation time difference. Both approaches recommend the same arm sizes in each example and the critical values become virtually indistinguishable for large $m$ and $n$ . Along with the gain in computation speed, the multivariate normal approximation can be done without the task of determining the number of simulations that are necessary to represent the distribution. The multivariate normal approximation may be reasonable for $m$ and $n$ as small as 10.

3. Power and robustness of SAR procedure

3.1. Determining arm size

The focus will be on the standardized location-scale family of distributions defined by Jeske and Yao.⁵ For the normal distribution, $Ψ$ is the CDF of the standard normal distribution. The logistic distribution will have CDF $Ψ (u) = (1 + \exp (- c u))^{- 1}$ with $c = π / 3$ . The Laplace distribution will use $Ψ (u) = (e^{c u} / 2) I (u < 0) + (1 - e^{- c u} / 2) I (u \geq 0)$ with $c = \sqrt{2}$ . Finally, the $t$ distribution with three degrees of freedom has $ψ (u) = 2 / π (1 + u^{2})^{2}$ . With these location-scale families, each distribution has zero mean and unit variance. Then $F (u) = Ψ ((u - μ_{F}) / σ_{F})$ with mean $μ_{F}$ and standard deviation $σ_{F}$ . While we will illustrate the procedure outlined in Section 2 with distributions in the location-scale family, other choices of continuous $F$ may also be used. As demonstrated by Friel¹³ with a selection of scale family distributions, adjustment of the alternative hypothesis and equation (2) may be desired when using other $F$ .

We begin with a demonstration of the arm sizes needed to detect certain alternatives. Various possibilities of the mixture alternative are shown in Table 1 along with the required arm size to detect each alternative. Note the decrease in arm size if the proportion of responders increases. The cases with $θ = 1$ highlight the fact that the experimenter would not have the necessary arm size if they erroneously worked under the pure shift assumption.

Table 1.
Arm sizes per group per stage necessary to detect the mixture alternative with sequential average rank (SAR) using equal arm sizes $(λ = 1)$ in the control and treatment groups for $α = 0.05$ and $80 %$ power with $ρ = 2$ determined by simulation of the joint distribution. Arm sizes determined by the normal approximation are in parentheses. The values for the proportion of responders, $θ$ , and size of the shift, $K$ , are displayed. $S = 1$ shows the results for a fixed sample design.

$K = 0.5$ $K = 1$

$θ$ Stages ( $S$ ) Normal Logistic Laplace $t_{3}$ Normal Logistic Laplace $t_{3}$

0.4 1 335 (337) 298 (297) 231 (232) 181 (181) 95 (94) 86 (86) 75 (75) 61 (60)

2 175 (176) 155 (155) 121 (122) 95 (95) 50 (50) 45 (45) 40 (40) 32 (32)

3 121 (121) 106 (107) 83 (84) 65 (65) 35 (34) 31 (31) 27 (27) 22 (22)

4 93 (92) 82 (81) 64 (64) 50 (50) 27 (26) 24 (24) 21 (21) 17 (17)

0.6 1 149 (149) 131 (132) 103 (103) 80 (80) 41 (42) 38 (38) 33 (33) 27 (26)

2 79 (78) 69 (69) 54 (54) 43 (42) 22 (22) 20 (20) 18 (18) 15 (14)

3 54 (54) 47 (48) 37 (37) 29 (29) 16 (15) 14 (14) 13 (12) 10 (10)

4 41 (41) 37 (36) 29 (29) 23 (23) 12 (12) 11 (11) 10 (10) 8 (8)

0.8 1 84 (84) 73 (74) 58 (58) 45 (45) 24 (23) 21 (21) 18 (18) 15 (15)

2 44 (44) 39 (39) 31 (31) 24 (24) 13 (12) 12 (11) 10 (10) 9 (8)

3 30 (30) 27 (27) 22 (21) 17 (17) 9 (9) 8 (8) 7 (7) 6 (6)

4 24 (23) 21 (21) 17 (16) 13 (13) 7 (7) 7 (6) 6 (6) 5 (5)

1.0 1 53 (53) 47 (47) 37 (37) 29 (29) 15 (14) 14 (13) 12 (11) 9 (9)

2 29 (28) 25 (25) 20 (20) 16 (15) 8 (8) 8 (7) 7 (6) 6 (5)

3 20 (20) 18 (17) 14 (14) 11 (11) 6 (6) 6 (5) 5 (5) 4 (4)

4 16 (15) 14 (13) 11 (11) 9 (8) 5 (5) 5 (4) 4 (4) 4 (3)

		$K = 0.5$	$K = 1$
0.4	1	335 (337)	298 (297)	231 (232)	181 (181)	95 (94)	86 (86)	75 (75)	61 (60)
	2	175 (176)	155 (155)	121 (122)	95 (95)	50 (50)	45 (45)	40 (40)	32 (32)
	3	121 (121)	106 (107)	83 (84)	65 (65)	35 (34)	31 (31)	27 (27)	22 (22)
	4	93 (92)	82 (81)	64 (64)	50 (50)	27 (26)	24 (24)	21 (21)	17 (17)
0.6	1	149 (149)	131 (132)	103 (103)	80 (80)	41 (42)	38 (38)	33 (33)	27 (26)
	2	79 (78)	69 (69)	54 (54)	43 (42)	22 (22)	20 (20)	18 (18)	15 (14)
	3	54 (54)	47 (48)	37 (37)	29 (29)	16 (15)	14 (14)	13 (12)	10 (10)
	4	41 (41)	37 (36)	29 (29)	23 (23)	12 (12)	11 (11)	10 (10)	8 (8)
0.8	1	84 (84)	73 (74)	58 (58)	45 (45)	24 (23)	21 (21)	18 (18)	15 (15)
	2	44 (44)	39 (39)	31 (31)	24 (24)	13 (12)	12 (11)	10 (10)	9 (8)
	3	30 (30)	27 (27)	22 (21)	17 (17)	9 (9)	8 (8)	7 (7)	6 (6)
	4	24 (23)	21 (21)	17 (16)	13 (13)	7 (7)	7 (6)	6 (6)	5 (5)
1.0	1	53 (53)	47 (47)	37 (37)	29 (29)	15 (14)	14 (13)	12 (11)	9 (9)
	2	29 (28)	25 (25)	20 (20)	16 (15)	8 (8)	8 (7)	7 (6)	6 (5)
	3	20 (20)	18 (17)	14 (14)	11 (11)	6 (6)	6 (5)	5 (5)	4 (4)
	4	16 (15)	14 (13)	11 (11)	9 (8)	5 (5)	5 (4)	4 (4)	4 (3)

It can be seen in the arm size tables that for a given alternative, there is an ordering of the distributions in terms of arm size needed to detect the same treatment effect. This ordering of the distributions is explained by Jeske and Yao⁵ with the ordering of the Kullback–Leibler distance between $H_{0}$ and $H_{A}$ . The distributions ordered by these distances from smallest to largest are normal, logistic, Laplace, and $t_{3}$ .

The arm size results determined via simulation may differ slightly from other runs due to the number of simulations. This variation may also be an explanation for the differences in results between simulation and the normal approximation, especially in large arm size situations. Otherwise, the arm sizes determined by both algorithms are nearly the same, illustrating the accuracy of the multivariate normal approximation.

The average sample number shows what the total sample size will be on average for a given design. It accounts for the random number of stages associated with the trial. Average sample numbers for a variety of designs are displayed in Table 2. The values may be calculated by simulating each setting until termination, finding the average stage at which it stopped, then multiplying the average by the sum of the number of observations per stage of the two groups, $m + n$ . Since the average sample numbers for group sequential designs $(S = 2, 3, 4)$ are less than the average sample numbers of their fixed sample $(S = 1)$ counterparts, it follows that group sequential methods can offer smaller average sample sizes over fixed sample methods.

Table 2.

Average sample number to detect the mixture alternative with sequential average rank (SAR) using equal arm sizes in the control and treatment groups for $α = 0.05$ , $80 %$ power, $ρ = 2$ , and equal arm sizes between groups $(λ = 1)$ determined with the normal approximation to the exact joint distribution. Critical values and arm sizes were determined by normal approximation. The values for the proportion of responders, $θ$ , and size of the shift, $K$ , are displayed. $S = 1$ shows results for a fixed sample design.

		$K = 0.5$				$K = 1$
$θ$	Stages ( $S$ )	Normal	Logistic	Laplace	$t_{3}$	Normal	Logistic	Laplace	$t_{3}$
0.4	1	674.0	594.0	464.0	362.0	188.0	172.0	150.0	120.0
	2	571.7	503.7	396.1	308.9	162.7	146.8	130.0	104.3
	3	537.7	475.3	373.0	289.6	152.0	138.6	120.7	98.3
	4	517.9	456.3	359.8	281.4	147.2	135.4	118.4	96.0
0.6	1	298.0	264.0	206.0	160.0	84.0	76.0	66.0	52.0
	2	253.9	224.6	175.9	137.1	72.2	65.7	58.8	46.2
	3	240.3	213.3	165.2	129.6	67.8	62.9	54.2	45.0
	4	231.4	203.6	162.8	128.8	67.9	62.2	55.9	45.0
0.8	1	168.0	148.0	116.0	90.0	46.0	42.0	36.0	30.0
	2	143.5	127.1	100.9	78.4	40.0	36.7	33.1	26.7
	3	134.4	120.4	94.0	75.6	40.4	36.3	31.8	27.0
	4	130.5	118.0	90.9	73.3	39.7	34.9	33.3	27.7
1.0	1	106.0	94.0	74.0	58.0	28.0	26.0	22.0	18.0
	2	91.7	81.8	65.2	49.5	26.8	23.7	20.5	17.2
	3	89.0	76.5	62.3	49.1	27.0	23.2	22.3	18.2
	4	85.1	74.3	61.5	46.1	27.8	23.4	22.4	17.5

Equal arm sizes in the control and treatment groups may not always be feasible due to practical limitations such as finding sufficient participants for the treatment group. In these cases, the experimenter may get the arm sizes necessary to detect a given design alternative for a specified ratio between the groups. Table 3 displays the treatment arm sizes and average sample numbers for various design alternatives where the control group arm size is two times the treatment group arm size.

Table 3.

Treatment arm size with average sample number (in parentheses) to detect the mixture alternative with sequential average rank (SAR) for $α = 0.05$ , $80 %$ power, $ρ = 2$ , and the size of the control group is two times the size of the treatment group $(λ = 2)$ determined with the normal approximation to the exact joint distribution. Critical values and arm sizes were determined by normal approximation. The values for the proportion of responders, $θ$ , and size of the shift, $K$ , are displayed. $S = 1$ shows results for a fixed sample design.

		$K = 0.5$				$K = 1$
$θ$	Stages ( $S$ )	Normal	Logistic	Laplace	$t_{3}$	Normal	Logistic	Laplace	$t_{3}$
0.4	1	253 (759.0)	224 (672.0)	175 (525.0)	137 (411.0)	72 (216.0)	66 (198.0)	57 (171.0)	46 (138.0)
	2	133 (645.2)	117 (567.7)	92 (445.7)	72 (348.5)	38 (183.1)	35 (168.2)	30 (144.4)	25 (119.3)
	3	91 (603.8)	80 (531.0)	63 (417.2)	49 (326.5)	26 (171.0)	24 (157.2)	21 (137.1)	17 (110.6)
	4	69 (582.2)	61 (514.4)	48 (401.8)	38 (316.6)	20 (165.7)	18 (151.8)	16 (131.7)	13 (106.5)
0.6	1	112 (336.0)	99 (297.0)	78 (234.0)	61 (183.0)	32 (96.0)	29 (87.0)	25 (75.0)	20 (60.0)
	2	59 (286.0)	52 (252.0)	41 (198.1)	32 (154.5)	17 (81.5)	15 (73.7)	14 (66.4)	11 (52.2)
	3	41 (270.6)	36 (237.7)	28 (184.9)	22 (144.9)	12 (77.7)	11 (71.0)	10 (63.7)	8 (50.8)
	4	31 (259.1)	28 (232.2)	22 (181.9)	17 (140.8)	9 (73.8)	8 (68.2)	7 (59.7)	6 (48.1)
0.8	1	63 (189.0)	56 (168.0)	44 (132.0)	34 (102.0)	18 (54.0)	16 (48.0)	14 (42.0)	11 (33.0)
	2	33 (159.9)	29 (142.0)	23 (110.9)	18 (86.7)	10 (47.3)	9 (42.6)	8 (37.6)	6 (28.4)
	3	23 (151.6)	20 (134.1)	16 (104.9)	13 (84.2)	7 (44.7)	6 (38.8)	5 (34.4)	5 (30.6)
	4	18 (148.6)	16 (131.7)	12 (102.1)	10 (81.3)	5 (43.1)	5 (39.6)	4 (34.6)	4 (30.1)
1.0	1	40 (120.0)	35 (105.0)	28 (84.0)	21 (63.0)	11 (33.0)	10 (30.0)	9 (27.0)	7 (21.0)
	2	21 (101.7)	19 (91.5)	15 (71.9)	12 (57.1)	6 (28.6)	6 (28.1)	5 (23.5)	4 (18.8)
	3	15 (98.2)	13 (85.3)	10 (67.6)	8 (52.1)	4 (28.0)	4 (25.4)	4 (24.4)	3 (18.5)
	4	11 (94.2)	10 (82.6)	8 (65.4)	6 (51.8)	4 (30.2)	3 (23.9)	3 (22.9)	2 (18.2)

3.2. Power simulation

Next, we wish to investigate the power of the SAR. Here we set the design alternative with a particular $S$ , $θ$ , $K$ , $α$ , $β$ , $ρ$ , and $λ$ . While the SAR is distribution-free in the sense that its distribution is the same under the null hypothesis for any choice of $F$ , an $F$ must be chosen when calculating the power, arm sizes, and critical values. For each distribution, we get the arm sizes and critical values via simulation for the chosen design alternative. Then we use the same distribution to simulate scenarios where $K$ is different than the set design alternative. That is, we are calculating the power for a range of $K$ . The design alternative for the simulations uses $S = 2$ , $θ = 0.8$ , $K = 0.5$ , $α = 0.05$ , $β = 0.2$ , $ρ = 2$ , and $λ = 1$ for each of four choices of $F$ .

The alternative hypothesis used in this article is a specific case of testing if $G$ is stochastically larger than $F$ . Therefore, we may compare to other tests that are also based around this more general alternative hypothesis. The test statistics we will consider are the log-rank,¹⁴ MaxCombo,¹⁵ and the statistic from Peng et al.⁶ The Wilcoxon rank-sum test statistic is a special case of the weighted log-rank test statistic.¹⁶

The power curves for each test statistic and the four different design scenarios are shown in Figure 4. Clearly, each test statistic provide essentially the same power curves. Since the procedure from Peng et al.⁶ does not use $F$ when determining the critical values and arm size, they are the same in all four panels. The large arm size used for Peng et al.⁶ allows the Central Limit Theorem to work effectively and in doing so, the test can handle data that is not normal.

The SAR, log-rank, and MaxCombo are able to take a hypothesized distribution of the data into account, thus adapting its design to the data. This results in different critical values (not shown) as well as different arm sizes needed to detect the alternative. The noticeable difference between the methods is the arm size. The method from Peng et al.⁶ will recommend the same arm size for different $F$ , all else equal. The log-rank and MaxCombo tests consistently need arm sizes larger than the SAR test in order to achieve the same power. The only $F$ for which the SAR does not have the smallest arm size than the competition is the normal distribution. If $F$ is normal, then the test statistic developed by Peng et al.⁶ will be more efficient. Therefore, it offers smaller arm sizes in this particular case. Thus, the SAR is less efficient than this competitor when $F$ is normal.

It can be seen that the order of distributions from greatest to least arm size is normal, logistic, Laplace, $t_{3}$ . Therefore, if we believe the distribution of the data to be Logistic, Laplace, or $t_{3}$ then using the SAR test statistic allows us to design the test appropriately and use a reduced arm size.

3.3. Robustness

In this section, we discuss the robustness of the SAR procedure. We address the concern of the procedure’s performance if a hypothesized value of the design alternative, namely $θ$ or $F$ , differs from the true value.

First, consider $θ$ . If $θ$ is less than the hypothesized value used in the design, there will be fewer responders to the treatment. This will make it more difficult to detect the shifted component of the treatment distribution. Therefore, the actual power will be lower than the design’s power. If $θ$ is greater than the hypothesized value used in the design alternative, then we can expect the shift to be easier to detect. The true power will be greater than the design’s power. Evidence for both of these scenarios is provided by the arm sizes in Table 1 and average sample numbers in Table 2. The arm sizes are decreasing for increasing $θ$ . Therefore, the test will not have the necessary arm size to detect the shift if $θ$ is less than the design value, resulting in lower power. For $θ$ greater than the design alternative value, the test will have more than enough observations to detect the shift which will give higher power.

The same reasoning can be applied to the situation where $F$ is different than the hypothesized value. Since the arm sizes needed to detect a given mixture alternative when $F$ is normal are always greater than the arm sizes for the same alternative when $F$ is logistic, we know that we will have lower power if we used a design alternative with $F =$ logistic but the data came from the normal distribution. Similarly, we will have higher power if we used a design alternative with $F =$ logistic but Laplace was the true $F$ .

3.4. Average treatment effect

Instead of thinking of the treatment effect as the pair $(θ, δ)$ , it may be of interest to define a higher level view of the treatment effect with $Δ = θ δ$ . This can be thought of as the average treatment effect on the population. With $δ = K σ$ , define $K^{'} = K θ$ . Then $K$ is no longer independently selected from $θ$ and is found $K = K^{'} / θ$ . Then we can find arm sizes as in the previous sections of this article. The experimenter may begin with $K^{'}$ , then get the appropriate $θ$ and $K$ from a sensitivity study.

Table 4 presents the arm sizes necessary to detect several different design alternatives based on $K^{'}$ . The larger arm sizes for smaller $θ$ with all else equal suggest that designing a test around the incorrect assumption of a pure shift, $θ = 1$ , will result in an underpowered test. Since some information is lost by using a single value when discussing the treatment effect, we return in the rest of the article to defining the treatment effect as the pair $(θ, δ)$ .

Table 4.
Arm sizes per group per stage necessary to detect the mixture alternative with sequential average rank (SAR) using equal arm sizes $(λ = 1)$ in the control and treatment groups for $α = 0.05$ and $80 %$ power with $ρ = 2$ determined using the normal approximation. The values for the proportion of responders, $θ$ , and $K^{'} = K θ$ , where $K$ is the size of the shift, are displayed. $S = 1$ shows results for a fixed sample design.

$K^{'} = 0.5$ $K^{'} = 1$

$θ$ Stages ( $S$ ) Normal Logistic Laplace $t_{3}$ Normal Logistic Laplace $t_{3}$

0.4 1 66 61 56 46 30 30 30 29

2 35 32 29 24 16 16 16 15

3 24 22 20 17 11 11 11 11

4 19 17 16 13 9 9 9 8

0.6 1 57 52 43 34 19 18 18 15

2 30 27 23 18 10 10 10 8

3 21 19 16 13 7 7 7 6

4 16 15 12 10 6 6 5 5

0.8 1 54 48 39 30 16 15 13 11

2 29 26 21 16 9 8 7 6

3 20 18 15 11 6 6 5 5

4 16 14 11 9 5 5 4 4

1 1 53 47 37 29 14 13 11 9

2 28 25 20 15 8 7 6 5

3 20 17 14 11 6 5 5 4

4 15 13 11 8 5 4 4 3

		$K^{'} = 0.5$	$K^{'} = 1$
0.4	1	66	61	56	46	30	30	30	29
	2	35	32	29	24	16	16	16	15
	3	24	22	20	17	11	11	11	11
	4	19	17	16	13	9	9	9	8
0.6	1	57	52	43	34	19	18	18	15
	2	30	27	23	18	10	10	10	8
	3	21	19	16	13	7	7	7	6
	4	16	15	12	10	6	6	5	5
0.8	1	54	48	39	30	16	15	13	11
	2	29	26	21	16	9	8	7	6
	3	20	18	15	11	6	6	5	5
	4	16	14	11	9	5	5	4	4
1	1	53	47	37	29	14	13	11	9
	2	28	25	20	15	8	7	6	5
	3	20	17	14	11	6	5	5	4
	4	15	13	11	8	5	4	4	3

4. Group sequential rerank procedure

The second method we wish to consider to combine the observations at each stage we will deem the group sequential rerank (SR) procedure. This method is explored by Spurrier and Hewett⁷ for two stages and general alternative. It is used by Shuster et al.⁸ for ordinal categorical data with an arbitrary number of stages and general alternative. We investigate the test statistic in an arbitrary number of stages with continuous data under a mixture alternative.

4.1. Test statistic and critical values

For this method, we will again be getting the Wilcoxon rank-sum statistic at each stage. However, the SR statistic at stage $s$ will use the Wilcoxon rank-sum statistic calculated from all observations up to and including stage $s$ . The SR statistic will take information from each stage into account by directly using both within-stage and between-stage ranks.

Let $R_{i 1}, \dots, R_{i n}$ be the ranks of $Y_{i 1}, \dots, Y_{i n}$ among all $X$ and $Y$ observations up to and including stage $i$ , $i = 1, \dots, s$ . At each stage, we calculate the Wilcoxon rank-sum statistic, $W_{s} = \sum_{i = 1}^{s} \sum_{j = 1}^{n} R_{i j}$ , using all observations from previous stages as well as the current stage. At stage $s$ , the mean and variance of $W_{s}$ are found by extending the mean and variance of the fixed sample Wilcoxon rank-sum statistic with $m$ and $n$ observations in each group to using $s m$ and $s n$ observations in each group. We simply replace $m$ with $s m$ and $n$ with $s n$ in equations (3) and (4). Define $μ_{s}$ and $σ_{s}^{2}$ as the mean and variance of $W_{s}$ under the null hypothesis. Then the SR statistic at stage $s$ is calculated by

{\tilde{Z}}_{s} = \frac{W_{s} - μ_{s}}{σ_{s}}

Using the probabilities defined in equation (1), the mean and variance of the SR statistic at stage

s

are

E [{\tilde{Z}}_{s}] = \frac{s n (s m p + \frac{s n + 1}{2}) - μ_{s}}{σ_{s}}

(11)

Var ({\tilde{Z}}_{s}) = \frac{s^{2} m n}{σ_{s}^{2}} (p - (s (m + n) - 1) p^{2} + (s n - 1) p_{1} + (s m - 1) p_{2})

Under the null hypothesis,

p = 1 / 2

and

p_{1} = p_{2} = 1 / 3

which result in zero mean and unit variance for the SR statistic.

The covariance of the SR statistic between stages $s$ and $s^{'}$ , $s \leq s^{'}$ , can be found using the Mann-Whitney formulation of the Wilcoxon rank-sum statistic. The result is the following¹⁷

Cov ({\tilde{Z}}_{s}, {\tilde{Z}}_{s^{'}}) = \frac{s^{2} m n}{σ_{s} σ_{s^{'}}} (p - (s^{'} (m + n) - 1) p^{2} + (s^{'} n - 1) p_{1} + (s^{'} m - 1) p_{2})

(12)

The same arm size algorithm from Figure 2 can be used to determine the arm sizes and critical values where we replace the SAR statistic with the SR statistic

4.2. Discussion of the joint distribution

Like the SAR, we consider a multivariate normal approximation to circumvent the need for running the large number of simulations that would be necessary to enumerate the exact distribution. It is clear that the marginal distribution of the SR statistic at each stage is asymptotically normal both under the null and alternative hypothesis, given that each is a standardized Wilcoxon rank-sum statistic. The limiting multivariate normal distribution of the SR statistic has been investigated and shown to hold by several authors.^18,8,7

Figure 5 shows a comparison of simulation and normal approximation critical values, arm sizes, and computation time. In the various scenarios, both algorithms give the exact same arm size. There are some small differences in the critical values in the smaller arm size situations. The critical values become indistinguishable between the two algorithms as the arm size increases. As with the SAR, we see a great improvement in computation time by using the normal approximation. The multivariate normal approximation may be reasonable for $m$ and $n$ as small as 10.

4.3. Comparison of SAR and SR

With both the SAR statistic and SR statistic based on the Wilcoxon rank-sum statistic, it is of interest to see how they compare and whether one may be preferred over the other. We begin by looking at the limits of the mean, variance and covariance of each statistic. At each stage, $s$ and $p$ are fixed, since $s$ is the stage number and $p = P (X < Y)$ is determined by $θ$ , $δ$ , and $F$ . First, we compare the means by looking at the limit of their ratio for increasing $m$ . Using the means from equations (5) and (11), it can be shown that for each fixed $λ$ we have

lim_{m \to \infty} \frac{E [Z_{s}]}{E [{\tilde{Z}}_{s}]} = 1

Therefore, the means of the two test statistics are asymptotically equivalent.

Next, we examine the covariance of the test statistics for stages $s$ and $s^{'}$ , $s \leq s^{'}$ . Setting $s = s^{'}$ gives the variance. Like $p$ above, $p_{1}$ and $p_{2}$ are fixed. It can be shown with equations (6) and (12) that the limit of the ratio of the covariances of the SAR and SR as $m \to \infty$ for each fixed $λ$ is

lim_{m \to \infty} \frac{Cov (Z_{s}, Z_{s^{'}})}{Cov ({\tilde{Z}}_{s}, {\tilde{Z}}_{s^{'}})} = 1

Thus the test statistics have the same asymptotic mean and variance–covariance matrix and both have a limiting multivariate normal distribution. Empirical evidence suggests that there is almost no difference in the means, variances, and covariances with

m

as small as 10 with

λ = 1

. As a consequence we can expect the arm size and critical values needed to detect a given design alternative to the same. The asymptotic equality of the SAR and SR suggests there is little gain from the between-stage ranks, likely due to the fact that the between-stage ranks and within-stage ranks are not independent.

For a further comparison of the test statistics, Table 5 shows the arm sizes needed to detect various alternatives for the SR statistic. The arm sizes in parentheses are those needed by the SAR statistic. Almost all comparisons are equal. The SR statistic requires one fewer observation per group in only a handful of scenarios.

Table 5.
Arm sizes necessary per group per stage to detect the mixture alternative using the SR statistic with equal arm sizes $(λ = 1)$ for $α = 0.05$ and $80 %$ power with $ρ = 2$ determined by the normal approximation. The arm sizes for the SAR statistic determined by the normal approximation are in parentheses. The values for the proportion of responders, $θ$ , and size of the shift, $K$ , are displayed. $S = 1$ shows results for a fixed sample design.

$K = 0.5$ $K = 1$

$θ$ Stages ( $S$ ) Normal Logistic Laplace $t_{3}$ Normal Logistic Laplace $t_{3}$

0.4 1 337 (337) 297 (297) 232 (232) 181 (181) 94 (94) 86 (86) 75 (75) 60 (60)

2 176 (176) 155 (155) 122 (122) 95 (95) 49 (50) 45 (45) 39 (40) 32 (32)

3 120 (121) 106 (107) 83 (84) 65 (65) 34 (34) 31 (31) 27 (27) 22 (22)

4 92 (92) 81 (81) 64 (64) 50 (50) 26 (26) 24 (24) 21 (21) 17 (17)

0.6 1 149 (149) 132 (132) 103 (103) 80 (80) 42 (42) 38 (38) 33 (33) 26 (26)

2 78 (78) 69 (69) 54 (54) 42 (42) 22 (22) 20 (20) 17 (18) 14 (14)

3 54 (54) 47 (48) 37 (37) 29 (29) 15 (15) 14 (14) 12 (12) 10 (10)

4 41 (41) 36 (36) 28 (29) 22 (23) 12 (12) 11 (11) 9 (10) 8 (8)

0.8 1 84 (84) 74 (74) 58 (58) 45 (45) 23 (23) 21 (21) 18 (18) 15 (15)

2 44 (44) 39 (39) 30 (31) 24 (24) 12 (12) 11 (11) 10 (10) 8 (8)

3 30 (30) 27 (27) 21 (21) 16 (17) 9 (9) 8 (8) 7 (7) 6 (6)

4 23 (23) 20 (21) 16 (16) 13 (13) 7 (7) 6 (6) 5 (6) 4 (5)

1 1 53 (53) 47 (47) 37 (37) 29 (29) 14 (14) 13 (13) 11 (11) 9 (9)

2 28 (28) 25 (25) 19 (20) 15 (15) 8 (8) 7 (7) 6 (6) 5 (5)

3 19 (20) 17 (17) 13 (14) 11 (11) 5 (6) 5 (5) 4 (5) 4 (4)

4 15 (15) 13 (13) 10 (11) 8 (8) 4 (5) 4 (4) 3 (4) 3 (3)

		$K = 0.5$	$K = 1$
0.4	1	337 (337)	297 (297)	232 (232)	181 (181)	94 (94)	86 (86)	75 (75)	60 (60)
	2	176 (176)	155 (155)	122 (122)	95 (95)	49 (50)	45 (45)	39 (40)	32 (32)
	3	120 (121)	106 (107)	83 (84)	65 (65)	34 (34)	31 (31)	27 (27)	22 (22)
	4	92 (92)	81 (81)	64 (64)	50 (50)	26 (26)	24 (24)	21 (21)	17 (17)
0.6	1	149 (149)	132 (132)	103 (103)	80 (80)	42 (42)	38 (38)	33 (33)	26 (26)
	2	78 (78)	69 (69)	54 (54)	42 (42)	22 (22)	20 (20)	17 (18)	14 (14)
	3	54 (54)	47 (48)	37 (37)	29 (29)	15 (15)	14 (14)	12 (12)	10 (10)
	4	41 (41)	36 (36)	28 (29)	22 (23)	12 (12)	11 (11)	9 (10)	8 (8)
0.8	1	84 (84)	74 (74)	58 (58)	45 (45)	23 (23)	21 (21)	18 (18)	15 (15)
	2	44 (44)	39 (39)	30 (31)	24 (24)	12 (12)	11 (11)	10 (10)	8 (8)
	3	30 (30)	27 (27)	21 (21)	16 (17)	9 (9)	8 (8)	7 (7)	6 (6)
	4	23 (23)	20 (21)	16 (16)	13 (13)	7 (7)	6 (6)	5 (6)	4 (5)
1	1	53 (53)	47 (47)	37 (37)	29 (29)	14 (14)	13 (13)	11 (11)	9 (9)
	2	28 (28)	25 (25)	19 (20)	15 (15)	8 (8)	7 (7)	6 (6)	5 (5)
	3	19 (20)	17 (17)	13 (14)	11 (11)	5 (6)	5 (5)	4 (5)	4 (4)
	4	15 (15)	13 (13)	10 (11)	8 (8)	4 (5)	4 (4)	3 (4)	3 (3)

SR: sequential rerank; SAR: sequential average rank.

With the evidence of asymptotic equivalence and equality of arm sizes in Table 5, the SAR statistic is the recommended test statistic over the SR statistic. An added benefit of the SAR statistic is the computational convenience of calculating the Wilcoxon rank-sum statistic with only the observations at each stage, whereas all observations need to be used throughout the entirety of the experiment with the SR statistic.

4.4. R code

An R function provided in the Supplemental Material with user documentation can be used to calculate the arm size and critical values for various design alternatives using the four choices of $F$ discussed in this article. The user can input the number of stages, Type I error rate, Type II error rate, $ρ$ , $θ$ , $K$ , $σ_{F}$ , $F$ , $λ$ , number of simulations to use to estimate the joint distribution, choice of test statistic, and whether or not they wish to use the normal approximation of the joint distribution to determine the arm size and critical values. While the procedure outline in Section 2 is adaptable to other choices of error spending functions and design boundaries, the function provided uses the error spending functions defined previously. The function outputs the arm sizes required to detect the inputted design alternative, the critical values for each stage, and the associated error rates.

5. Estimation

The treatment effect in our setting is $(θ, δ)$ . In this section, we consider a modified method of moments estimator based on the one proposed by Jeske and Yao⁵ and an estimator based on constrained $k$ -means. Each estimator will be calculated based on observations up to and including stage $s^{'}$ , the stage at which the trial ends.

5.1. Method of moments

First, we consider a method of moments (MoM) estimator that builds upon the estimator proposed by Jeske and Yao.⁵ These estimates use the sample mean and sample variance of the control and treatment groups, $(\bar{X}, S_{X}^{2})$ and $(\bar{Y}, S_{Y}^{2})$ , respectively, from all observations up to and including the termination stage $s^{'}$ .

Due to the randomness of the sample statistics, it is possible for the estimators to be outside the parameter space. In order for the estimators to appropriately represent the null scenario, if either estimator is equal to zero then the estimate of the other will also be set to zero. For the MoM estimators, this situation would occur only if $\bar{Y} - \bar{X} \leq 0$ which would lead to the estimate of $δ$ to be zero. Therefore, we modify the estimators from Jeske and Yao⁵ such that they are calculated if $\bar{Y} - \bar{X} > 0$ and are set to zero otherwise.

Our MoM estimators for $(θ, δ)$ can be calculated as

\begin{aligned} {\hat{θ}}_{M o M} & = {\begin{cases} {[1 + \frac{(S_{Y}^{2} - S_{X}^{2})_{+}}{(\bar{Y} - \bar{X})^{2} + ε_{s^{'}}}]}^{- 1} & if \bar{Y} - \bar{X} > 0 \\ 0 & otherwise \end{cases} \\ {\hat{δ}}_{M o M} & = {\begin{cases} (\bar{Y} - \bar{X}) [1 + \frac{(S_{Y}^{2} - S_{X}^{2})_{+}}{(\bar{Y} - \bar{X})^{2} + ε_{s^{'}}}] & if \bar{Y} - \bar{X} > 0 \\ 0 & otherwise \end{cases} \end{aligned}

where

t_{+} = t

t > 0

and 0, otherwise is used to restrict the estimates to the parameter space. For

ε_{s^{'}}

, we will use an adaptive approach suggested by Lubich et al.¹⁹ It is calculated by

ε_{s^{'}} = S_{X}^{2} \frac{\log ((s^{'} (m + n))^{2})}{s^{'} (m + n)}

Use of

ε_{s^{'}}

ensures consistency of the estimators and makes the estimators invariant to scale transformations.

5.2. Constrained $k$ -means

Next, we explore estimating the treatment effect using the constrained $k$ -means algorithm, CKM, proposed by Wagstaff et al.²⁰ The algorithm takes a set of “must-link” and “cannot-link” constraints to be incorporated into the traditional $k$ -means algorithm.²¹ The must-link constraints are the observations the user identifies as needing to be in the same cluster, while the cannot-link constraints are the observations the user identifies as needing to be in different clusters. In the CKM algorithm, points are assigned to clusters in a way that does not violate the constraints.

With this constrained algorithm, we can utilize both the control and treatment observations together in the $k$ -means clustering approach. We assign the control observations to the must-link constraints, since we know they are all from $F$ . CKM will always keep these observations in the same cluster. We assign the largest observation from the treatment group and the smallest observation from the combined sample to the cannot-link constraint. The largest observation from the treatment group is likely a responder to the treatment that comes from the shifted version of $F$ . The smallest observation in the entire data set is likely to come from $F$ . Therefore these two observations should not be grouped into the same cluster. In order to avoid a local minimum solution, we run the algorithm 50 times with random starting values and choose the clustering that has the smallest total within cluster sum of squares.

Occasionally for two clusters, the cluster that includes the control group observations will also include the largest observations of the treatment group. This contradicts the assumption that the largest observations in the treatment group would come from a shifted version of $F$ . It may suggest that there is no shift and give evidence of the null. If CKM clusters the observations in this way, we set the estimates for $(θ, δ)$ to $(0, 0)$ .

Under the mixture alternative, there should be two clusters. Let the cluster with the control group observations be Cluster 1 with mean ${\bar{C}}_{1}$ and the cluster without the control group observations be Cluster 2 with mean ${\bar{C}}_{2}$ . Then we calculate the estimates for $(θ, δ)$ as

\begin{aligned} {\hat{θ}}_{C K M} & = {\begin{cases} \frac{\# of observations in cluster 2}{s^{'} m} & if {\bar{C}}_{2} - {\bar{C}}_{1} > 0 \\ 0 & otherwise \end{cases} \\ {\hat{δ}}_{C K M} & = {\begin{cases} {\bar{C}}_{2} - {\bar{C}}_{1} & if {\bar{C}}_{2} - {\bar{C}}_{1} > 0 \\ 0 & otherwise \end{cases} \end{aligned}

5.3. Results

This section will discuss the results of simulating estimates of $(θ, δ)$ using the estimators from the previous sections. They will be evaluated with the square root of their mean squared error (RMSE). The process will involve simulating a group sequential trial until termination, then calculating the estimates using the observations up to and including the termination stage. We present the results such that a specific group sequential design alternative is set and the true parameter values may be different than the values chosen for the design alternative.

RMSE values for estimating $θ$ can be seen in Table 6. When $θ = 0.5$ , CKM is the preferred estimator. However, as $θ$ moves away from 0.5, MoM becomes the better choice. Based on empirical results that are not shown, the MoM is similarly better as $θ$ decreases from 0.5. As $θ$ increases with all else equal, the RMSE of the MoM decreases while the RMSE of the CKM estimator increases. When estimating $θ$ , both estimators see their RMSE decrease as $K$ increases with all else equal. A final consideration is the effect of $F$ . For both estimators a general pattern of slightly increasing RMSE is present when moving from the normal distribution to the $t_{3}$ distribution. This can be attributed to the increasingly heavy tails of the distributions. When estimating $θ$ , the MoM is the overall better choice. However, if the experimenter had prior knowledge that $θ$ may be around 0.5, they may wish to use CKM.

Table 6.
RMSE values for estimating $θ$ based on 1000 simulations where the data are generated from the $F$ , proportion of responders $θ$ , and shift size $K$ values indicated in the table with $σ_{F} = 1$ . The design alternative uses $S =$ 2, $θ =$ 0.7, $K =$ 1.5, $F =$ Normal, $α =$ 0.01, $β =$ 0.1, $ρ = 2$ , and $λ = 1$ resulting in an arm size of 17 for both the control and treatment groups.

$θ$ $K$ Estimator Normal Logistic Laplace $t_{3}$

0.5 1 MoM 0.341 0.366 0.364 0.364

CKM 0.236 0.266 0.292 0.296

1.5 MoM 0.276 0.293 0.303 0.315

CKM 0.187 0.196 0.221 0.239

2 MoM 0.223 0.238 0.256 0.246

CKM 0.146 0.149 0.162 0.196

0.7 1 MoM 0.272 0.277 0.293 0.296

CKM 0.324 0.354 0.395 0.417

1.5 MoM 0.217 0.221 0.226 0.232

CKM 0.239 0.260 0.282 0.310

2 MoM 0.171 0.182 0.185 0.206

CKM 0.173 0.195 0.188 0.251

0.9 1 MoM 0.217 0.216 0.227 0.230

CKM 0.447 0.470 0.511 0.526

1.5 MoM 0.133 0.153 0.168 0.195

CKM 0.340 0.371 0.397 0.390

2 MoM 0.115 0.123 0.137 0.146

CKM 0.267 0.268 0.273 0.297

$θ$	$K$	Estimator	Normal	Logistic	Laplace	$t_{3}$
0.5	1	MoM	0.341	0.366	0.364	0.364
		CKM	0.236	0.266	0.292	0.296
	1.5	MoM	0.276	0.293	0.303	0.315
		CKM	0.187	0.196	0.221	0.239
	2	MoM	0.223	0.238	0.256	0.246
		CKM	0.146	0.149	0.162	0.196
0.7	1	MoM	0.272	0.277	0.293	0.296
		CKM	0.324	0.354	0.395	0.417
	1.5	MoM	0.217	0.221	0.226	0.232
		CKM	0.239	0.260	0.282	0.310
	2	MoM	0.171	0.182	0.185	0.206
		CKM	0.173	0.195	0.188	0.251
0.9	1	MoM	0.217	0.216	0.227	0.230
		CKM	0.447	0.470	0.511	0.526
	1.5	MoM	0.133	0.153	0.168	0.195
		CKM	0.340	0.371	0.397	0.390
	2	MoM	0.115	0.123	0.137	0.146
		CKM	0.267	0.268	0.273	0.297

RMSE: root mean squared error; MoM: method of moments; CKM: constrained $k$ -means.

Represent the lower RMSE between the MoM and CKM estimators for each combination of $θ$ and $K$ .

We also consider the RMSE when estimating $δ$ , as presented in Table 7. It can quickly be seen that the MoM has the smaller RMSE in all but one scenario, which is close to being a tie. Patterns similar to those seen in the RMSE when estimating $θ$ also arise in the RMSE when estimating $δ$ . For both estimators, we generally see the RMSE decreasing as $θ$ or $δ$ increases with all else equal. The effects of an $F$ with larger tails are much more apparent in the RMSE of both estimators when estimating $δ$ than when estimating $θ$ . Overall, the MoM is the preferred choice when estimating $δ$ .

Table 7.

RMSE values for estimating $δ$ based on 1000 simulations where the data are generated from the $F$ , proportion of responders $θ$ , and shift size $K$ values indicated in the table with $σ_{F} = 1$ . The design alternative uses $S =$ 2, $θ =$ 0.7, $K =$ 1.5, $F =$ Normal, $α =$ 0.01, $β =$ 0.1, $ρ = 2$ , and $λ = 1$ resulting in an arm size of 17 for both the control and treatment groups.

$θ$	$K$	Estimator	Normal	Logistic	Laplace	$t_{3}$
0.5	1	MoM	0.591	0.627	0.716	1.375
		CKM	0.893	0.996	1.172	2.312
	1.5	MoM	0.606	0.677	0.745	1.457
		CKM	0.707	0.811	1.002	2.276
	2	MoM	0.602	0.656	0.698	1.068
		CKM	0.579	0.656	0.832	1.298
0.7	1	MoM	0.481	0.523	0.621	1.203
		CKM	0.835	0.984	1.175	1.978
	1.5	MoM	0.442	0.500	0.554	0.810
		CKM	0.670	0.802	0.940	1.599
	2	MoM	0.426	0.477	0.478	1.235
		CKM	0.450	0.647	0.708	1.777
0.9	1	MoM	0.426	0.451	0.541	1.475
		CKM	0.865	0.950	1.264	1.594
	1.5	MoM	0.378	0.420	0.494	0.718
		CKM	0.637	0.839	1.025	1.244
	2	MoM	0.394	0.395	0.455	0.523
		CKM	0.497	0.575	0.791	1.194

RMSE: root mean squared error; MoM: method of moments; CKM: constrained $k$ -means.

Represent the lower RMSE between the MoM and CKM estimators for each combination of $θ$ and $K$ .

6. Summary

In this article, we argue that the traditional assumption of the pure shift model to represent the treatment distribution may not always be correct. A mixture model may be a more appropriate representation of the treatment group to account for the possibility that there are individuals who are unaffected by the treatment. This phenomenon was introduced by Good³ and Boos and Brownie.⁴ The Food and Drug Administration²² and Lubich et al.¹⁹ provide further evidence and examples of such nonresponders. If nonresponders exist but the experimenter erroneously works under the pure shift assumption, a design will be underpowered.

We present a novel method of using the Wilcoxon rank-sum statistic in group sequential clinical trials with the SAR test procedure. It’s usefulness over existing methods if the experimenter has some idea about $F$ is shown in the reduction of the arm size needed to detect a given alternative. A similar conclusion is reached in the comparison with fixed sample methods by the average sample numbers. The SAR is compared to the SR, an alternative application of the Wilcoxon rank-sum statistic in the group sequential setting, and their asymptotic equivalence is shown. Both test statistics maintain the Type I error rate irrespective of $F$ . The SAR is presented as the preferred method due to its slightly simpler computation.

We have shown how error spending methods may be used to set up a group sequential design with the SAR test procedure. However, the SAR test may be used with other group sequential boundaries since the distribution of the test statistic is unchanged by the choice of boundary.

Two estimators are evaluated for the treatment effect $(θ, δ)$ . Based on their RMSE, the method of moments estimators for $(θ, δ)$ are preferred over the CKM estimators. Prior knowledge that roughly half of the individuals may respond to the treatment could warrant the use of CKM to estimate $θ$ .

Supplemental Material

sj-pdf-1-smm-10.1177_09622802231184634 - Supplemental material for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Supplemental material, sj-pdf-1-smm-10.1177_09622802231184634 for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials by Dylan C Friel and Daniel R Jeske in Statistical Methods in Medical Research

Supplemental Material

sj-r-2-smm-10.1177_09622802231184634 - Supplemental material for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Supplemental material, sj-r-2-smm-10.1177_09622802231184634 for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials by Dylan C Friel and Daniel R Jeske in Statistical Methods in Medical Research

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

ORCID iD

Daniel R Jeske

Supplemental material

Supplemental material for this article is available online.

References

Lan

KKG

DeMets

. Discrete sequential boundaries for clinical trials. Biometrika 1983; 70: 659–663.

Chang

Hwang

Shin

. Group sequential designs using both type i and type ii error probability spending functions. Commun Stat - Theory Method 1998; 27: 1323–1339.

Good

. Detection of a treatment effect when not all experimental subjects will respond to treatment. Biometrics 1979; 35: 483–489.

Boos

Brownie

. Testing for a treatment effect in the presence of nonresponders. Biometrics 1986; 42: 191–197.

Jeske

Yao

. Sample size calculations for mixture alternatives in a control group vs. treatment group design. Statistics 2020; 54: 1–17.

Peng

Jeske

Sengupta

et al. Designing one-sided group sequential clinical trials to detect a mixture alternative. Seq Anal 2018; 37: 268–291.

Spurrier

Hewett

. Two-stage Wilcoxon tests of hypotheses. J Am Stat Assoc 1976; 71: 982–987.

Shuster

Chang

Tian

. Design of group sequential clinical trials with ordinal categorical data based on the Mann–Whitney–Wilcoxon test. Seq Anal 2004; 23: 413–426.

Madsen

Hewett

. Multi-stage tests based on sequential ranks. J Stat Comput Simul 1978; 7: 93–105.

10.

Randles

Wolfe

. Introduction to the Theory of Nonparametric Statistics. New York: John Wiley & Sons, 1979.

11.

Gibbons

Chakraborti

. Nonparametric Statistical Inference. 4 ed. New York, NY: Marcel Dekker, Inc., 2003.

12.

Mardia

Kent

Bibby

. Multivariate analysis. London: Academic Press, 1979.

13.

Friel

. Wilcoxon Rank Sum Tests to Detect One-Sided Mixture Alternatives in Group Sequential Clinical Trials. PhD Thesis, UC Riverside, 2022.

14.

Mantel

. Evaluation of survival data and two new rank order statistics arising in its consideration. Cancer Chemother Rep 1966; 50: 163–170.

15.

Prior

. Group sequential monitoring based on the maximum of weighted log-rank statistics with the Fleming–Harrington class of weights in oncology clinical trials. Stat Methods Med Res 2020; 29: 3525–3532.

16.

Klein

Moeschberger

. Survival Analysis Techniques for Censored and Truncated Data. 2nd ed. New York: Springer, 2003.

17.

Hewett

Spurrier

. A survey of two stage tests of hypotheses: theory and application. Commun Stat - Theory Method 1983; 12: 2307–2425.

18.

Sen

. Weak convergence of generalized

u

-statistics. Ann Proba 1974; 2: 90–102.

19.

Lubich

Jeske

Yao

. Statistical inference for method of moments estimators of a semi-supervised two-component mixture model. Am Stat 2022; 76: 376–383.

20.

Wagstaff

Cardie

Rogers

et al. Constrained k-means clustering with background knowledge. In Proceedings of 18th International Conference on Machine Learning. pp. 577–584.

21.

MacQueen

. Some methods for classification and analysis of multivariate observations. In Cam LML and Neyman J (eds.) Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability, volume 1. University of California Press, pp. 281–297.

22.

Food and Administration D. Paving the way for personalized medicine 2013; https://www.fdanews.com/ext/resources/files/10/10-28-13-Personalized-Medicine.pdf.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.07 MB

Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Abstract

Keywords

1. Introduction

2. Group sequential average rank procedure

2.1. Test statistic and critical values

2.4. Comparison of simulated distribution and normal approximation

3.1. Determining arm size

3.4. Average treatment effect

4.1. Test statistic and critical values

5. Estimation

5.1. Method of moments

5.2. Constrained k -means

5.3. Results

Supplemental Material

sj-pdf-1-smm-10.1177_09622802231184634 - Supplemental material for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Supplemental Material

sj-r-2-smm-10.1177_09622802231184634 - Supplemental material for Wilcoxon rank-sum tests to detect one-sided mixture alternatives in group sequential clinical trials

Footnotes

Declaration of conflicting interests

Funding

ORCID iD

Supplemental material

References

Supplementary Material

5.2. Constrained $k$ -means