Re-randomization tests for unplanned changes in clinical trials

Abstract

Despite the best efforts of investigators, problems forcing design changes can occur in clinical trials. Changes are usually relatively minor, but sometimes not. The primary endpoint or analysis may need to be revised, for example. It is common to regard any conclusion from such a tarnished trial as hypothesis-generating rather than definitive. This article reviews a very useful technique, re-randomization tests, for dealing with such anomalies. Re-randomization tests remain valid for testing a strong null hypothesis that treatment has no effect on the data that led to design changes. Another way of expressing this is that the data used to inform a design change must give no information about the treatment labels. This restriction has implications for limiting the amount of information examined by a committee deciding whether to make design alterations. While nothing can eliminate the pall cast by breaches of protocol, re-randomization tests following blinded and limited data examination go a long way toward amelioration.

Keywords

Adaptive methods blinding clinical trials conditioning permutation/re-randomization tests primary analysis/endpoint

Introduction

This article reviews the role of re-randomization tests when unplanned changes are made before breaking the treatment blind in clinical trials. Are conclusions from such a trial only hypothesis-generating, or are they definitive? We will see that the truth lies somewhere between these two extremes.

Randomization, blinding, and pre-specification of outcomes and analysis methods are cornerstones of clinical trials. Randomization prevents biased assignment of patients to treatments, tends to balance groups with respect to important prognostic factors, and can form the basis for testing the null hypothesis of no effect of a treatment. Regarding the last point, a re-randomization test, also called a randomization test or permutation test, can be applied whereby the null distribution is approximated by fixing the data at their observed values, re-randomizing according to the randomization method (e.g. permuted blocks), computing the value of the test statistic under these re-randomized treatment labels, and repeating this process a huge number of times. This idea was first used by Fisher in connection with two experiments, one to confirm or refute a woman’s claim that she could taste whether milk or tea was added first to a cup, and the other involving growth rates of plants.¹ He recognized that with large sample sizes, re-randomization tests give nearly the same answer as their parametric counterparts, a fact that has been corroborated by several authors.^2–4 Permutation tests have been applied in simple settings such as those considered by Fisher, and to more complicated settings involving ordering p-values from multiple comparisons and testing them in a way that protects the family-wise error rate.⁵ Section 1.1.1 of Good⁶ presents an A–Z list of the multitude of applications of these tests in different fields.

Blinding of treatment assignments tends to balance the groups with respect to the power of positive thinking, and prevents patients or doctors in the control arm from supplementing their therapy to make up for the fact that they did not receive the experimental treatment, for example. Blinding may also lessen dropout, as patients in an unblinded trial who know they are receiving placebo may discontinue participation.

Pre-specification of analyses is essential to avoid inflation of the type-1 error rate due to multiplicity. Imagine, for example, switching from a signed rank test to a sign test after observing that all paired differences have positive signs. The actual type-1 error rate is impossible to even quantify because if we had seen data that were favorable for a different test (e.g. the paired t-test), we might have switched to that test. For this reason, data-driven changes in clinical trials are rightly criticized.

Despite the fact that unplanned changes should be avoided whenever possible, they are sometimes unavoidable. It is not unusual to see a primary event rate lower than anticipated. In extreme cases, power may be dismal. This might lead to increasing the sample size, changing recruitment to target a higher risk population, or changing the primary endpoint. The first two options are less extreme and more common than the third, but changing the primary outcome can be appealing in certain situations. For example, suppose that, a priori, there are two nearly equally compelling endpoints A and B, but A is selected as primary. If the overall event rate for A turns out to be much lower than for B, a switch from A to B may be in order. Alternatively, in a trial of lung disease, A and B may be outcomes defined in terms of imaging using a new technique, and blinded evaluation of scans reveals that A is much more difficult than B to measure reliably. Again, a change in primary endpoint may be in order.

This unplanned change setting and the closely related article of Posch and Proschan⁷ is somewhat different from the long literature on pre-planned adaptations based on blinded data. For example, Hogg et al.⁸ examined rank statistics from lumped data from both groups to deduce the heaviness of the tails of the distribution and thereby choose a powerful rank test. Gould⁹ and Gould and Shih¹⁰ showed how to recalculate sample size using blinded data for binary and normal outcomes, respectively. Edwards¹¹ used blinded data to select from a set of pre-specified models, and then applied a re-randomization test. Proschan et al.¹² considered a similar adaptive regression setting. These methods exploit the fact that if the alternative hypothesis is true, we can gain an advantage by tailoring the analysis to what is seen in blinded data, while if the null hypothesis is true, we seem to lose nothing. We will see that this is not quite correct; there can be difficulties in interpretation. In the setting we consider, something untoward has caused us to change the outcome or analysis plan. The same reasoning used for pre-planned analyses can be used to justify re-randomization tests for unplanned changes, and the same limitations apply. The difference is that when we apply these methods out of necessity, the advantages far outweigh the disadvantages.

The outline for the remainder of this article is as follows. We first review re-randomization tests in a static setting with no changes. We show that they control the conditional type-1 error rate, given the observed outcome data $\underline{Y} = \underline{y}$ . We then expand this result to a setting in which we examine, at the end of the trial, a larger set of blinded data $\underline{D}$ to select an outcome variable and test statistic. We show that the re-randomization test remains valid for testing the null hypothesis that the treatment assignment vector is independent of $\underline{D}$ . We then provide applications of the key result and reveal limitations of conclusions that can be drawn. In particular, the data leading to a design change must not give information about the treatment labels, and this is shown to have implications on limiting the amount of data used to inform design changes. Next, we discuss modification of the technique when the design change is made before the end of the trial. We conclude with a discussion, including the difficulties of estimation following unplanned changes.

Re-randomization tests and conditioning

This section summarizes the well-known results that (1) re-randomization tests are conditional tests given a set of data and (2) controlling the conditional type-1 error rate controls the unconditional error rate as well. Lehmann¹³ recognized these facts in connection with a two-sample shift alternative setting with unknown and arbitrary continuous density $f$ . He showed that the permutation test conditions on the set of order statistics, which form a complete, sufficient statistic. He further showed that only a permutation test can control the type-1 error rate regardless of the underlying continuous $f$ .

Throughout this article, we consider the setting of rejecting a null hypothesis of no treatment effect for small values of a test statistic $T (\underline{Y}, \underline{Z})$ , where $\underline{Y}$ is an aggregate outcome vector and $\underline{Z}$ is the randomization vector ( $Z_{i} = 1$ or $0$ if patient $i$ is assigned treatment or control). For instance, $T (\underline{Y}, \underline{Z})$ might be the t-statistic parameterized so that negative values indicate treatment benefit on continuous outcome $Y$ .

To understand why re-randomization tests work, imagine a time reversal in which we observe data $\underline{Y} = \underline{y}$ and then randomize patients using assignment vector $\underline{Z}$ . It is clear that the conditional distribution function $F (t | \underline{y})$ of test statistic $T (\underline{y}, \underline{Z})$ , given $\underline{Y} = \underline{y}$ would be

\begin{array}{l} F (t | \underline{y}) \overset{D e f}{=} P (T \leq t | \underline{Y} = \underline{y}) \\ = \sum_{z} I {T (\underline{y}, \underline{z}) \leq t} P (\underline{Z} = \underline{z}) \end{array}

(1)

where $P (\underline{Z} = \underline{z})$ is dictated by the randomization method used. Equation (1) is known as the re-randomization distribution of the test statistic $T (\underline{Y}, \underline{Z})$ . The number of possible randomization vectors $\underline{z}$ is often prohibitively large to enumerate. If we write equation (1) as $E [I {T (\underline{y}, \underline{Z}) \leq t}] = P {T (\underline{y}, \underline{Z}) \leq t}$ , we see that it can be estimated by re-randomizing a huge number of times and computing the proportion of re-randomized trials such that $T (\underline{y}, \underline{Z})$ is $t$ or less.

In reality, we randomize before collecting outcome data, but if the null hypothesis is true that the outcome vector and randomization vector are independent, it does not matter whether randomization occurs before or after outcome data $\underline{Y} = \underline{y}$ are observed. We summarize as follows

Theorem 1. The re-randomization distribution (1) of a test statistic $T (\underline{Y}, \underline{Z})$ is the conditional distribution function of $T (\underline{Y}, \underline{Z})$ , given that $\underline{Y} = \underline{y}$ , computed under the null hypothesis that $\underline{Y}$ and $\underline{Z}$ are independent. □

The formal short proof is as follows

\begin{array}{l} P {T (\underline{Y}, \underline{Z}) \leq t | \underline{Y} = \underline{y}} = P {T (\underline{y}, \underline{Z}) \leq t | \underline{Y} = \underline{y}} \\ = \sum_{z} I {T (\underline{y}, \underline{z}) \leq t} P (\underline{Z} = \underline{z} | \underline{Y} = \underline{y}) \\ = \sum_{z} I {T (\underline{y}, \underline{z}) \leq t} P (\underline{Z} = \underline{z}) \end{array}

(2)

because $\underline{Y}$ and $\underline{Z}$ are independent under $H_{0}$ . Note that equation (2) is the re-randomization distribution given by equation (1).

The critical value $c$ is the largest realizable value of T such that $F (c | \underline{Y} = \underline{y}) \leq α$ . The re-randomization test then rejects the null hypothesis if and only if $T (\underline{y}, {\underline{z}}_{a ct}) \leq c$ , where ${\underline{z}}_{a ct}$ is the actual randomization vector used in the trial. By construction, this procedure bounds the conditional type-1 error rate given $\underline{Y} = \underline{y}$ at $α$ .

Theorem 2. Whenever type-1 error rate is controlled at level $α$ conditioned on a collection of data, unconditional type-1 error rate is also controlled at level $α$ . □

This follows immediately from the fact that unconditional type-1 error rate is simply the expected value of conditional type-1 error rate. Theorem 2 tells us that the type-1 error rate of a re-randomization test is controlled at level $α$ .

Remark 1. The null hypothesis being tested with a re-randomization test, namely, that $\underline{Y}$ and $\underline{Z}$ are independent, is quite strong. It implies that the distribution of $\underline{Y}$ is identical in the two arms. For instance, a re-randomization test may reject $H_{0}$ even if treatment and control means are equal but variances differ. This is not a false positive, but a correct conclusion, because $\underline{Y}$ and $\underline{Z}$ are not independent under this circumstance.

Remark 2. In some settings, it is advisable to condition not only on $\underline{Y}$ but also on the numbers $n_{T}$ and $n_{C}$ of patients assigned to the two arms when computing the distribution of the test statistic. With standard randomization techniques such as permuted block randomization, $n_{T}$ and $n_{C}$ do not inform us about the treatment effect, so conditioning on them follows the accepted principle that one should condition on ancillary statistics. However, with response-adaptive randomization,¹⁴ one would ordinarily not condition on the sample size to compute the distribution of the test statistic. Sample size gives information about the treatment effect, so conditioning on sample size would inadvertently condition away the treatment effect (see Wei¹⁵ and Begg¹⁶ or section 2 of Proschan and Nason¹⁷ for a famous example). Computing the conditional distribution of a test statistic should not be confused with improving an estimator by using its conditional expectation, given a sufficient statistic. The latter is always advisable, but the former is not. □

Unplanned change setting

The preceding section showed that a re-randomization test applied to a pre-specified outcome variable $Y$ controls the conditional type-1 error rate given $\underline{Y} = \underline{y}$ , and therefore the unconditional type-1 error rate. Now imagine examining, at the end of the trial and before breaking the treatment blind, data from several outcome variables. We can denote these data by a single vector $\underline{D}$ that is a concatenation of data from different outcomes. We allow arbitrary changes in primary outcome or analysis. There are uncountably infinitely many potential options. For example, we might use $\underline{D}$ to determine whether to use trimmed means or not, and what proportion of the tails to trim.

As before, it is helpful when trying to assess validity to picture the time reversal scenario in which the data $\underline{D}$ are observed before randomization occurs. We switch from statistic $T$ on outcome $Y$ to statistic $T^{'}$ on outcome $Y^{'}$ . Then randomization occurs. It is clear that a re-randomization test on $T^{'} ({\underline{Y}}^{'}, \underline{Z})$ would still be valid. In reality, randomization precedes outcome data, but again that should not matter if the treatment assignment vector is independent of $\underline{D}$ .

Before making precise the notion in the preceding paragraph, we need some notation. Denote the new test statistic and outcome variable selected after examining $\underline{D}$ by $T_{\underline{D}}$ . For instance, suppose that data from possible outcome $1$ occupy indices $1, 2, \dots, n$ of $\underline{D}$ ; data from possible outcome 2 occupy indices $n + 1, \dots, 2 n$ of $\underline{D}$ , and so on. Then $T_{\underline{D}}$ identifies, on the basis of $\underline{D}$ , the section of $\underline{D}$ (i.e. the outcome variable) and test statistic to use. The random quantity $T_{\underline{D}} (\underline{D}, \underline{Z})$ is the value of the selected test statistic as a function of $\underline{D}$ and $\underline{Z}$ . Theorem 3 below generalizes a result of Edwards,¹¹ who implicitly assumes a finite or countably infinite number of potential models.

Theorem 3. At the end of the trial, select a test statistic $T_{\underline{D}}$ using blinded data $\underline{D}$ . If $ψ (\underline{D}, \underline{Z}) \overset{Def}{=} T_{\underline{D}} (\underline{D}, \underline{Z})$ is a Borel function of $\underline{D}$ for each fixed value of $\underline{Z}$ , then a re-randomization test applied to $T_{\underline{D}}$ protects the conditional type-1 error rate given $\underline{D}$ and the unconditional type-1 error rate under the strong null hypothesis that $\underline{D}$ and $\underline{Z}$ are independent. □

The proof is as follows. The Borel assumption clearly implies that $ψ (\underline{D}, \underline{Z})$ is a Borel function of $(\underline{D}, \underline{Z})$ . Under the null hypothesis that $\underline{D}$ and $\underline{Z}$ are independent, a conditional distribution function $F (t | \underline{d})$ of $ψ (\underline{D}, \underline{Z})$ , given that $\underline{D} = \underline{d}$ , is given by $E {ψ (\underline{d}, \underline{Z})} = E [I {T_{\underline{d}} (\underline{d}, \underline{Z}) \leq t}]$ (see Corollary 4.38 of Breiman¹⁸). But this is the re-randomization distribution of $T_{\underline{D}} (\underline{D}, \underline{Z})$ . Therefore, the re-randomization test finds a critical value such that the conditional type-1 error rate, given $\underline{D}$ , is no greater than $α$ . By Theorem 2, the unconditional type-1 error rate is protected as well.

Remark 3. In accordance with Remark 2, we sometimes condition on sample sizes $n_{T}$ and $n_{C}$ . When we do so, the term “unconditional type-1 error rate” refers to the error rate averaged over $\underline{D}$ , but still conditioned on $n_{T}$ and $n_{C}$ . □

When we use Theorem 3 in an unplanned change setting, we are implicitly assuming that our actions can be represented by some function $T_{\underline{D}}$ , which seems reasonable. The condition that $T_{\underline{D}} (\underline{D}, \underline{Z})$ is a Borel function of $\underline{D}$ for fixed $\underline{Z}$ is fairly weak. It includes situations in which there are at most countably infinitely many possible adaptations, or when $T_{\underline{D}} (\underline{D}, \underline{Z})$ is continuous in $\underline{D}$ for each fixed value of $\underline{Z}$ . Examples of the latter scenario include a weighted logrank statistic, where the weights are a continuous function of $\underline{D}$ , or a t-test using a Box–Cox transformation $(Y^{p} - 1) / p$ , where the rule selecting $p$ is a continuous function of $\underline{D}$ .

A crucial part of Theorem 3 is that it tests the strong null hypothesis that $\underline{D}$ and $\underline{Z}$ are independent. Implicit in this assumption is that $\underline{D}$ does not give us any information about the treatment labels. This assumption would be violated, for example, if we saw data broken down by arm A versus arm B, without knowing which of A or B was treatment. Other violations involving unwittingly breaking the treatment blind are discussed in the next section.

Examples and limitations

We begin with a fairly innocuous example to illustrate the usefulness of re-randomization tests in an unplanned change setting. We then move to murkier examples and illustrate why there are down sides of the procedure.

Example 1. Consider a trial with a continuous endpoint to be compared between arms using a t-test. Suppose that the blinded data reveal an apparent outlier. A t-test is not robust to outliers because both the mean and variance are profoundly affected. For instance, in a one-sample t-test setting, if one fixes all but one observation and sends the remaining observation to infinity at a fast rate, the t-statistic converges to $1$ , which is not statistically significant at any conventionally used alpha level. Section 5.1 of Staudte and Sheather¹⁹ vividly demonstrates this problem with the dataset Gosset used to illustrate his t-test. They show that the originally highly statistically significant t-statistic for testing $μ = 0$ versus $μ > 0$ becomes nonsignificant when the largest value is quadrupled! A similar problem occurs with the two-sample t-test.

The solution to this problem in the clinical trial is simple: switch to a Wilcoxon rank-sum test and use its re-randomization distribution to test for statistical significance. By Theorem 3, this controls the conditional and unconditional type-1 error rate.□

The following examples illustrate the restriction on conclusions that can be made from re-randomization tests in an unplanned change setting.

Example 2. Posch and Proschan⁷ consider an apparent paradox in which $\underline{D}$ consists of data from three possible outcome variables, two of which are reasonable clinical endpoints and the third is the level of study drug in the blood. Examining this third variable unblinds us, of course. Now that we have the complete set of treatment assignments, we can compute the values of the test statistics for the other two possible outcome variables and pick the one with the smaller p-value. Even if treatment has no effect on either of the legitimate clinical outcomes, the probability of rejecting the null hypothesis exceeds the intended level, $α$ . The problem is even worse with more than two legitimate clinical outcomes and the level of study drug in the blood.□

To understand why Example 2 does not contradict Theorem 3, consider the following remark.

Remark 4. The null hypothesis being tested with a re-randomization test in an adaptive setting is even stronger than that of the static setting of the section on conditioning (see Remark 1). The proper conclusion is not that the null hypothesis associated with the selected test statistic is false, but that the joint distribution of the examined data $\underline{D}$ is not identical in the treatment and control arms.□

We did not make a type-1 error in the apparent paradox of Example 2 because the joint distribution of $\underline{D}$ is not identical in the treatment and control arms: treatment does have an effect on the level of study drug in the blood! We rejected $H_{0}$ and reached a correct but insipid conclusion. This is an appropriate punishment for examining a variable that obviously unblinds us.

The fact that the re-randomization test allows rejection of only the strong null hypothesis is problematic because we are always looking at more data than we think. When we look at any outcome variable, we see not only the blinded data, but also the amount of missing data. Therefore, $\underline{D}$ contains missingness indicators. It is not uncommon to have more missing data in one arm than in the other. Technically, when we reject the null hypothesis, we are allowed to say only that the joint distribution of $\underline{D}$ differed in the two arms, and that difference could conceivably be in the amount of missing data. It is common in re-randomization tests to condition additionally on the sample sizes per arm, as in Remark 2. This mitigates the problem. Whether we condition on sample sizes or not, we have the same problem with missing data as in a fixed design trial, namely, that analyzing only nonmissing data is a nonrandomized comparison.

Example 3. The introduction mentioned a lung disease trial whose primary endpoint, endpoint A, was based on imaging. Blinded scans revealed that endpoint A could not be measured reliably, whereas endpoint B could be. The decision was made to switch to endpoint B. Assume that only these two possible outcomes were considered. With $\underline{D}$ denoting outcome data on endpoints A and B, Theorem 3 implies that a re-randomization test is a valid test of the strong null hypothesis that the joint distribution of $\underline{D}$ differs in the two arms. It does not necessarily imply that the marginal distribution of data from endpoint B is the same in the two arms. Therefore, technically, one cannot conclude from a statistically significant re-randomization test applied to outcome B that treatment was beneficial on outcome B! Any clinician would argue that outcome B was the only one for which a meaningful comparison could be made, so the only sensible conclusion would be that treatment provided benefit on outcome B. One would be hard-pressed to argue against that logic.

Although Example 2 showed that one can technically conclude only that treatment had some impact on $\underline{D}$ rather than the specific outcome selected, it was extreme. The amount of unblinding that occurs in a realistic setting is probably quite small. Therefore, a conclusion that treatment affected the new outcome selected is probably fairly safe when the amount of data examined is small. If we look at many variables, there is likely to be more inadvertent unblinding. In that case, we might feel compelled to report only that $\underline{D}$ and $\underline{Z}$ are not independent. That conclusion becomes weaker with the amount of blinded data examined. This discussion illustrates that some changes are much more consequential than others. A change in test statistic is likely to be viewed more favorably than a change in primary outcome variable, for instance.

Remember that the methods of this article are to be used in emergency settings only. When something untoward occurs, the onus is on the trial investigators to argue, based on the facts of the particular trial, that results were not just a consequence of inadvertent unblinding. For instance, suppose there were very few missing observations, and only two outcomes were examined, either of which could reasonably have been selected a priori as the primary endpoint. If, after a forced change of endpoints based on blinded data, the ultimately selected outcome showed a statistically significant result, one could make a good case that treatment benefitted that endpoint.

Mid-course changes

Up to now, we have considered a simple situation with no interim monitoring, and any change is made at the end of the trial. A slightly different approach is to examine blinded data at a single intermediate time point, make a change in primary outcome or analysis, and then use a re-randomization test on the selected outcome at the end of the trial. The proper analysis at the end of the trial is a stratified re-randomization test. For instance, suppose that we are conditioning on per-arm sample sizes, as in Remark 2. Then we construct the re-randomization distribution by considering only those re-randomizations that match the per-arm sample sizes at both the time of the change and the end of the trial. For example, if the numbers assigned to treatment and control are (206, 197) at the time the change was made and (400, 395) at the end of the trial, then one would consider only those re-randomizations yielding these same per-arm sample sizes at the time of change and end of study.

A much more complicated setting is when there is interim monitoring in addition to an unplanned change. For example, monitoring may have occurred with the original primary endpoint, then a change is made, and monitoring proceeds with the new primary outcome. In most cases, this presents enormous problems with both statistics and interpretation. Monitoring is often based on a Brownian motion approximation to the joint distribution of a standardized statistic over time (Lan and Zucker,²⁰ chapter 2 of Proschan et al.²¹). It would be very odd and difficult to calculate boundaries using one endpoint for some time points and another for other time points. One potential option would be to subtract the amount of alpha used by the time of change. For instance, suppose that the change in endpoint is made after using cumulative alpha $0.015$ . The remaining alpha is $0.05 - 0.015 = 0.035$ . Calculate the information fraction for the new endpoint as the total number of events thus far divided by the number expected at the end of the trial, and use a spending function with overall error rate of $0.035$ . Now use a re-randomization test over time (see section 8.2 of Proschan et al.²¹ or Pawitan and Hallstrom²²). Such a trial would not be pristine, but may be the best alternative in a bad situation.

Our simple setting does not cover response-adaptive randomization, which continually changes treatment assignment probabilities in response to observed outcome data on a short-term binary endpoint. Interestingly, a re-randomization test in that setting is also valid under a very commonly satisfied condition given by Simon and Simon.²³

Discussion

Re-randomization tests are invaluable tools when something goes awry in a clinical trial and forces an unplanned design change before breaking the treatment blind, though the methodology is not a panacea. A major drawback is that, technically, the conclusion one can draw from a statistically significant result is that the joint distribution of all data examined differs by arm. The more data one uses to make a change, the weaker the conclusion must be when rejecting this null hypothesis. Moreover, we are always looking at more than we think, such as the amount of missing data.

Another problem with unplanned changes is that they do not lend themselves to estimation. It is easy to compute the null distribution of the test statistic because the strong null hypothesis implies that $\underline{D}$ and $\underline{Z}$ are independent. $\underline{D}$ and $\underline{Z}$ are not independent under an alternative hypotheses. Of course, they are not independent under an alternative hypothesis even when no design changes are made, but we can figure out the distribution of the test statistic in that setting. For example, with a continuous outcome and paired data, inverting the test statistic leads to the Hodges–Lehmann confidence interval for the median difference. With unpaired binary data, we can invert the test using the noncentral hypergeometric distribution to get a confidence interval for the odds ratio. However, looking at data and making changes that were not pre-planned makes it virtually impossible to compute the distribution of a test statistic under an alternative hypothesis. For instance, suppose we notice an apparent outlier, and then switch from a t-test to a Wilcoxon test. How can we be sure what we would have done had the outlier not been quite as pronounced? We might have remained with the t-test. One possible approach to estimation is to invert the confidence interval for the test statistic ultimately selected. That way, the confidence interval and test will be consistent.

One word of warning is in order. Sometimes data and safety monitoring boards review data with labels “Arm A” and “Arm B,” without knowing which arm corresponds to the experimental treatment. They argue that keeping the arms “blinded” in this way until they are ready to make a definitive conclusions prevents bias. This does not constitute blinding in the sense of this article. The only valid re-randomization distribution if one sees data by arm, but without knowing which arm is the experimental treatment, would assign probability 1/2 to arm A being treatment and 1/2 to control. In other words, the re-randomization distribution takes on only two possible values. The one-tailed re-randomization p-value can be either $1 / 2$ or $1$ . Clearly, adaptations made using this definition of “blinded” are not appropriate.

The methods described in this article are intended for emergency use. Besides the limitations mentioned above, there is a very practical concern that results may not be accepted. Many clinical trialists are steadfast about following the protocol. They believe that whenever there is a major change, such as a new primary endpoint or analysis method, results can only be hypothesis-generating. This skepticism is understandable. Nonetheless, I believe that when the change is made before breaking the treatment blind, a re-randomization test provides evidence that is closer to definitive than to hypothesis-generating.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Fisher

. The design of experiments (Sixth edition in 1951). Edinburgh: Oliver and Boyd, 1935.

Wald

Wolfowitz

. Statistical tests based on permutations of the observations. Ann Math Stat 1944; 15: 358–372.

Noether

. On a theorem by Wald and Wolfowicz. Ann Math Stat 1949; 20: 455–458.

Hoeffding

. A combinatorial central limit theorem. Ann Math Stat 1951; 22: 558–566.

Westfall

Young

. Resampling-based multiple testing: examples and methods for p-value adjustment. New York: John Wiley & Sons, 1993.

Good

. Permutation tests: a practical guide to resampling methods for testing hypotheses. 2nd ed.New York: Springer, 2000.

Posch

Proschan

. Unplanned adaptations before breaking the blind. Stat Med 2012; 31: 4146–4153.

Hogg

Fisher

Randles

. A two-sample adaptive distribution-free test. J Am Stat Assoc 1975; 70: 656–661.

Gould

. Interim analyses for monitoring clinical trials that do not materially affect the type I error rate. Stat Med 1992; 11: 55–66.

10.

Gould

Shih

. Sample size re-estimation without unblinding for normally distributed outcomes with unknown variance. Commun Stat A Theor 1992; 21: 2833–2853.

11.

Edwards

. On model prespecification in confirmatory randomized studies. Stat Med 1999; 18: 771–785.

12.

Proschan

Leifer

Liu

. Adaptive regression. J Biopharm Stat 2005; 15: 593–603.

13.

Lehmann

. Testing statistical hypotheses (Third edition, with Romano JP, 2010, is published by Springer). New York: John Wiley & Sons, 1959.

14.

Rosenberger

. The theory of response-adaptive randomization in clinical trials. New York: John Wiley & Sons, 2006.

15.

Wei

. Exact two-sample permutation tests based on the randomized play-the-winner rule. Biometrika 1988; 75: 603–606.

16.

Begg

. On inferences from Wei’s biased coin design for clinical trials. Biometrika 1990; 77: 467–484.

17.

Proschan

Nason

. Conditioning in 2

\times

2 tables. Biometrics 2009; 65: 316–322.

18.

Breiman

. Probability. Reading, MA: Addison-Wesley, 1968.

19.

Staudte

Sheather

. Robust estimation and testing. New York: John Wiley & Sons, 1990.

20.

Lan

Zucker

. Sequential monitoring of clinical trials: the role of information and Brownian motion. Stat Med 1993; 12: 753–765.

21.

Proschan

Lan

Wittes

. Statistical monitoring of clinical trials: a unified approach. New York: Springer, 2006.

22.

Pawitan

Hallstrom

. Statistical interim monitoring of the Cardiac Arrhythmia Suppression Trial. Stat Med 1990; 9: 1081–1090.

23.

Simon

. Using randomization tests to preserve type 1 error with response adaptive and covariate adaptive randomization. Stat Probab Lett 2011; 81: 767–772.