Double-blinding of an acupuncture randomized controlled trial optimized with clinical translational science award resources

Abstract

Background

Clinical trial articles often lack detailed descriptions of the methods used to randomize participants, conceal allocation, and blind subjects and investigators to group assignment. We describe our systematic approach to implement and measure blinding success in a double-blind phase 2 randomized controlled trial testing the efficacy of acupuncture for the treatment of vulvodynia.

Methods

Randomization stratified by vulvodynia subtype is managed by Research Electronic Data Capture software’s randomization module adapted to achieve complete masking of group allocation. Subject and acupuncturist blinding assessments are conducted multiple times to identify possible correlates of unblinding.

Results

At present, 48 subjects have been randomized and completed the protocol resulting in 87 subject and 206 acupuncturist blinding assessments.

Discussion

Our approach to blinding and blinding assessment has the potential to improve our understanding of unblinding over time in the presence of possible clinical improvement.

Keywords

Acupuncture randomization blinding randomized controlled trial Research Electronic Data Capture clinical translational science vulvodynia double-blind sham acupuncture

Background

The efficacy of acupuncture is difficult to demonstrate in the absence of a double-blind milieu. We are currently testing a double-blind treatment protocol for the treatment of vulvodynia. Vulvodynia is a chronic pain condition characterized by vulvar pain and painful sexual intercourse (dyspareunia) and has no consistently effective treatments.¹ We provide details of our procedures to achieve an internally valid, double-blind acupuncture efficacy trial with careful attention to blinding of the treatment condition, concealment of treatment allocation, and measurement of blinding success.

Acupuncture experiments have a range of methodological challenges. Most are performed unblinded or single-blinded (subject blind, acupuncturist unblind) and use a range of control conditions including no acupuncture conditions, which may vary in intensity (e.g. physical therapy, usual care, no other care, or attention control), or sham acupuncture using a variety of methods. Evidence shows stronger effects for acupuncture compared to no acupuncture,² yet these designs do not establish that the therapeutic effect is more than a placebo or expectancy effect, a common criticism of acupuncture experiments.^3–5 Sham acupuncture control conditions may use shallow needle insertion at non-acupuncture points or non-penetrating needles, but have shown more similar results to therapeutic acupuncture, possibly because needling or skin stimulation at the alternative sites has some therapeutic effect.² Others have argued that acupuncture in single-blind controlled studies is less effective than routine clinical care due to the use of a base unit that hides needle penetration.⁶ We have developed a rigorous test of acupuncture for reduction of the pain and dyspareunia of vulvodynia using penetrating (verum) and placebo needles that are identical in appearance and are designed to feel the same to subjects and acupuncturists, that is, double-blind needles (Figure 1). The validity of these double-blind needles has been supported in multiple studies, in single-use applications,^7–12 but this is the first study to use this methodology in a multi-needle, multi-session experiment. Our study is designed to test the efficacy of the standardized vulvodynia acupuncture protocol beyond placebo effect and, secondarily, to learn about correlates of blinding.

Figure 1.

A set of double-blind acupuncture needles (DBNs).

Treatment allocation in the context of a randomized trial is the process of assigning subjects to either a treatment or control condition and the allocation sequence is the randomized order of each subject assignment. Selection bias may occur when research staff create conditions that influence which subjects are enrolled and the sequence of their assignment to a treatment arm. To prevent this bias, methodologists recommend that subjects are randomized after enrollment and that the allocation sequence is concealed from staff.¹³ In a double-blind placebo controlled trial, the blinding and concealment of treatment allocation can occur through the same process and both are critical to the integrity of the study. In accordance with the consolidated standards of reporting trials (CONSORT),¹⁴ we present our process for blinding and treatment allocation as well as the level of blinding for persons involved in this research. These details have rarely been reported in methodology sections of published RCTs.^14–16

While it is accepted that double-blinding is ideal in an experimental design, the measurement of blinding success is controversial.^17–19 For many conditions and therapies, blinding to treatment arm may be confounded with treatment efficacy. As a subject improves, the subject and clinician may correctly assume the subject received an active treatment. Measurement of blinding success was part of the CONSORT checklist but was removed in the 2010 revision due to this dilemma.^14,19 Our study dually tests an acupuncture treatment protocol and the use of double-blind needles in a multi-session trial. Therefore, we have taken a discovery approach to understanding blinding success through a multi-pronged assessment for subjects and acupuncturists that will allow us to learn about possible correlates of unblinding. We present an overview of our protocol, methods to conceal treatment allocation and maintain a double-blind protocol, and our approach to measuring the success of blinding.

Methods

Study overview

Our study was designed to achieve equipoise for all non-specific effects of our treatment protocol. All 80 subjects expected to complete the study are acupuncture naïve at time of enrollment. Subjects randomized 1:1 to either penetrating or placebo needles, receive a 13-needle, twice-weekly, 10-session, 5-week acupuncture treatment protocol. Subjects and acupuncturists are instructed to speak only if safety is compromised in an attempt to remove the therapeutic effects of their relationship. A research associate monitors the fidelity of the acupuncture treatment protocol at all treatment sessions. Data collection and management are facilitated using the Research Electronic Data Capture (REDCap), hosted at the University of Illinois at Chicago.^20,21 This study was approved by the University of Illinois at Chicago Institutional Review Board. Successful completion of our research will allow us to study the methodology of double-blinding in a multi-session multi-needle efficacy trial and to test the efficacy of an acupuncture protocol for the treatment of vulvodynia pain.

Randomization protocol

We developed our randomization and blinding procedures to support the internal validity of our study. The on-site statistician oversees the labeling of needle packages, which arrive from the manufacturer in Japan tagged with the treatment arm identifier (1 or 2; Figure 2). The statistician removes the tag and labels the packet with a unique code, described below.

Figure 2.

Needle package contents for each subject.

Randomization is implemented using the REDCap randomization module for its ability to manage stratified randomization and because we could adapt it to mask group allocation. We chose to randomize within strata defined by diagnostic subtype, generalized vulvodynia, and provoked vestibulodynia, due to its prognostic importance.²² We wanted to achieve complete masking of group allocation to ensure that our clinical and research staff did not observe patterns across subjects that might unblind them. For example, if staff were aware of group allocation (e.g. treatment arm 1), they might notice that subjects in that arm seem to improve compared to the other arm and deduce it is the penetrating needle arm. If this occurs, staff have the potential to introduce an expectancy bias in their interactions with future subjects assigned to arm 1. Therefore, we went beyond masking that arm 1 might be the penetrating needle arm, thus preventing staff from ascertaining any treatment arm identifiers entirely.

The REDCap randomization module executes a user-defined randomization model. It requires the project statistician to upload an allocation sequence developed using other software and generates assignments accordingly with the click of a button. Typically, in a randomized controlled trial, subjects are assigned to two or three groups, but since we adapted the module to mask allocation, we created 176 “groups,” one for each possible subject accounting for attrition and unknown strata proportions. Instead of REDCap assigning one of the two treatment arms, we had it assign one of the 176 “groups” which represent unique codes for each individual enrolled. We called this code a randomization identifier (ID) that is linked to the treatment arm in a spreadsheet stored outside of the REDCap system. This randomization ID serves as the study ID for the remainder of the study.

We developed the allocation sequence using Stata²³ with permuted blocks of consistent size including twice as many potential assignments than our target sample size. After copying this sequence into a spreadsheet, we added a diagnostic subtype indicator in equal proportions and a randomly ordered list of IDs. This master worksheet, which included treatment arm allocations (1 and 2), was saved in a secure location with access restricted to the two study statisticians. An example of this master worksheet is shown in Table 1. We used the diagnostic subtype and ID columns from our master worksheet to set up the randomization module in REDCap. We used ID instead of the treatment arm indicator, which functioned as desired because we set up 176 “groups.” Thus, when a staff person randomizes a subject, the ID code is revealed rather than the treatment arm. The staff selects the needle package with the matching code, which contains the subject’s needles for their full course of acupuncture.

Table 1.

Master worksheet sample for randomization allocation schedule.

Generated by Stata				Manually added^a
Arm	Block	Random number	Sequence number	Generalized subtype	Randomization ID
2	1	0.0238564	1	1	140
1	1	0.1413895	2	1	173
1	1	0.1841138	3	1	116
2	1	0.971364	4	1	145
2	2	0.0461595	1	1	180
1	2	0.4007082	2	1	136
2	2	0.7773784	3	1	141
1	2	0.9007107	4	1	115
2	43	0.2526355	1	0	183
1	43	0.3804519	2	0	134
2	43	0.6840113	3	0	104
1	43	0.9718139	4	0	150
1	44	0.1008327	1	0	133
2	44	0.5935789	2	0	179
2	44	0.754083	3	0	160
1	44	0.8610591	4	0	181

REDCap: Research Electronic Data Capture.

Two arm 1 and two arm 2 assignments were randomly shuffled within each block by creating a column with random numbers, then sorting the random number from smallest to largest within each block. Half of the blocks were assigned to each diagnostic subtype. A randomization ID was inputted to be linked to each treatment arm assignment.

Columns used for REDCap allocation template.

By design, the principal investigator, research staff, and acupuncturists are blinded to any treatment arm identifier, and allocation of assignment is concealed. The statistician is aware of a treatment arm identifier (1 or 2) but blinded to its meaning. Our off-site statistician is the only person not blinded to treatment arm. Consequently, he manages communication with study subjects following completion of the study (e.g. emails to inform the subject about treatment received and to offer free acupuncture sessions to those receiving placebo).

Measurement of blinding success

Both subjects and acupuncturists are asked to guess the type of needles used without a “don’t know” option and to rate their confidence on a 0–10 scale. Subjects answer these questions twice, after their first and after their final acupuncture session via a self-administered survey using a tablet computer. The first assessment allows us to examine subjects’ blinding before sustained treatment effects are evident. The second assessment, after 10 acupuncture sessions, will allow us to examine the association of treatment effects with beliefs about treatment received.

Our approach for the assessment of acupuncturist blinding capitalizes on our 10-session protocol. We use multiple acupuncturists to administer the study treatments. Due to the logistics of scheduling 10 sessions per subject, multiple acupuncturists may treat each subject over the 5-week course of treatment. Acupuncturists are asked the blinding questions after the subjects’ 1st and 10th sessions and each time they see a subject for the first time.

Statistical analyses

Once data collection is completed, statistical methods will be used to assess blinding. We will compute a weighted guess score coding a correct guess as +1 and an incorrect guess as −1, which will be multiplied by the confidence rating yielding a possible range of −10 to +10, where 0 indicates no confidence in the guess regardless of its accuracy. For subject responses, we will calculate a separate blinding index for each treatment arm using Bang’s method, adapted for our guess score, which yields a value from −1 interpreted as opposite guessing (100% incorrect guesses) to +1, which is complete unblinding (100% correct guessing).²⁴ Bang’s Blinding Index is a descriptive measure that can be compared with findings from other literature reviews published using this approach.^25,26 Mixed effect linear regression models will be used to understand correlates of the guess score measure for subjects and acupuncturists in separate analyses.²⁷ Our data will contain multiple records per subject, and random coefficients will be used to account for respondents’ influence on their repeated measurements over time. Treatment arm by time interactions will be a key predictor in our modeling. Other candidate variables for these analyses are shown in Table 2.

Table 2.

Possible predictors of unblinding for exploratory analyses.

Variable	Description
Treatment arm	Penetrating versus non-penetrating placebo needles
Session number	Session 1–10 of treatment protocol that is associated with the guess, treated as categorical fixed effect
Recent pain rating	Patient’s worst pain in the past 24 h. For 10th session, this rating will be indicative treatment response.^a
Acupuncturist’s experience	Years of experience
Protocol experience	Number of treatments acupuncturist provided within the study in total
Session # of acupuncturist–subject pair	Number of sessions acupuncturist has treated the subject^a
Body mass index	Acupuncturist may note difference with needle insertion if subject has low versus higher body fat.
Month	Months of data collection over the duration of the study^a
Adverse events	Subject feeling like they will “pass out” or the treatment is too strong

Time-varying measures that will be tested as average and deviation scores as appropriate.

Results

We are currently 22 months into our data collection period and have screened 238 women and enrolled and randomized 58 to participate in the study. We have completed the protocol with 48 subjects and 9 dropouts. We employ six part-time acupuncturists. The results presented below include the lessons learned from our randomization approach, our adherence to the blinding measurement protocol, and the current numbers of blinding assessments associated with the 48 subjects who have completed the study.

Randomization

Our randomization model was designed to assign subjects stratified by two diagnostic subtypes of vulvodynia. This function has been implemented without issue. So far, one randomization-related problem arose on one occasion during the study period. The REDCap system was unavailable when a new subject entered the study and needed to be randomized. Our solution was to randomly choose a needle package so that treatment could be initiated as usual. Afterwards, the statistician reassigned the package chosen to the next randomization identifier in the sequence once REDCap was back online and reallocated the original packet number for future use. This required careful documentation and correction to the master worksheet. A backup plan was also made should this problem recur. In our study, the outage and our workaround for randomization was more easily corrected than if we had used REDCap to assign the treatment arm directly. It should be noted that the randomization sequence in REDCap cannot be edited once the module is set up and in production.

Blinding assessments

We examined the blinding assessment data for missing values to judge adherence with our protocol for the 48 subjects who completed the protocol. We recognized in the early months of enrollment that the collection of the blinding assessment after the first session was overlooked for several subjects. In contrast, no blinding assessment was missing from the 10th session. Assessing the circumstances, we determined that all missing assessments were not filled out due to the complexity of intake tasks to be completed at the first visit. The research specialist self-corrected and developed a reminder system so that the questions were asked routinely. We assume these missing assessments are consistent with the missing at random mechanism because they are associated with a procedural error in the first months of data collection. Consequently, we will estimate our mixed effect models using full-information maximum likelihood, which will utilize all available data and produce less biased than complete case analysis.^28,29

At present, we have 39 (81%) subjects with two blinding assessments and 9 (19%) with one. Our acupuncturist assessments were collected following the appropriate acupuncture sessions: 1st, 10th, and every session when an acupuncturist treated a subject for the first time. These data were collected with 94% adherence to the protocol. For the same 48 subjects described above, we have 206 assessments distributed across the 10 sessions as shown in Figure 3.

Figure 3.

Number of acupuncturist blinding assessments by visit (48 subjects, 9 acupuncturists, 206 assessments).

Discussion

The methods described here reflect a team effort to carefully design and implement a rigorous and internally valid acupuncture double-blind RCT. Although we encountered minor issues along the way, we are confident that our study will be useful in testing the efficacy of our acupuncture protocol for the relief of vulvodynia pain compared to a valid placebo condition. Our use of the REDCap randomization module to assign a unique randomization code has worked well to conceal the allocation schedule and mask treatment arms from staff. These procedural steps have been easy to implement and work reliably. We believe that we have created the best conditions to maintain blinding to treatment allocation and group assignment, but to achieve acupuncture double-blinding and to successfully measure it, over a 10-session, 5-week treatment may be difficult. Statistical approaches to quantify the degree of unblinding have been developed and used.^{18,24,25,30–33} Other researchers have used methods for analysis and interpretation when blinding has been breached.^34,35 Our approach is to discover if there are limits to blinding that occur over time, with experience, and with symptom relief.

We recognize some limitations in our approach. Our research specialist, though blind to condition, is not asked blinding questions. Anecdotally, she has observed situations that are potentially unblinding (e.g. reddening of the skin at the needle insertion site, more pain reaction with needle insertion, needles that fall over after placement). These situations are infrequent and do not occur consistently with the same subjects or acupuncturists. Time will tell if they lead to unblinding due to experience.

In sum, the methods employed here and lessons learned may be helpful to others in developing rigorous acupuncture studies. In addition to our primary aim, to evaluate efficacy for the treatment of vulvodynia pain, this study has promise for adding to our understanding about acupuncturists’ blinding with non-penetrating placebo needles. It will also provide a longitudinal opportunity to look at subject blinding before and after a clinical effect may occur.

Footnotes

Acknowledgements

ClinicalTrials.gov Identifier: NCT03364127.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: N.T. and the Educational Foundation of Hanada Gakuen possess a US patent 6575992B1, a Canadian patent CA 2339223, a Korean patent 0478177, a Taiwanese patent 150135, a Chinese patent ZL00800894.9 (Title: Safe needle, placebo needle and needle set for double-blinding), and two Japanese patents 4061397 (Title: Placebo needle, and needle set for double-blinding) and 4315353 (Title: Safe needle) on the needles described in this manuscript. N.T. is a salaried employee of the Educational Foundation of Hanada Gakuen. Drs. Y.Y., J.M.S., and A.D.S. report grants from the National Institutes of Health, National Institute of Nursing Research. Dr. D.J.W. reports grants from the National Institutes of Health, National Cancer Institute and ownership of eNursing llc, a company with no ownership of tools used in this study. No other author has any competing interests.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This publication was made possible by the grant number R01 HD091210 from the National Institutes of Health, National Institute of Child Health and Human Development (NICHD). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NICHD. The final peer-reviewed manuscript is subject to the National Institutes of Health Public Access Policy.

ORCID iD

Judith M Schlaeger

References

Bornstein

Goldstein

Stockdale

, et al. 2015 ISSVD, ISSWSH, and IPPS consensus terminology and classification of persistent vulvar pain and vulvodynia. J Sex Med 2016; 13(4): 607–612.

Vickers

Vertosick

Lewith

, et al. Acupuncture for chronic pain: update of an individual patient data meta-analysis. J Pain 2018; 19(5): 455–474.

Bausell

Lao

Bergman

, et al. Is acupuncture analygesia an expectancy effect? Eval Health Prof 2005; 28: 9–26.

McGeeney

. Acupuncture is all placebo and here is why. Headache 2015; 55(3): 465–469.

Colquhoun

Novella

. Acupuncture is theatrical placebo. Anesth Analg 2013; 116(6): 1360–1363.

Kim

Lee

Alraek

, et al. Acupuncture in sham device controlled trials may not be as effective as acupuncture in the real world: a preliminary network meta-analysis of studies of acupuncture for hot flashes in menopausal women. Acupunct Med 2020; 38(1): 37–44.

Takakura

Yajima

. A double-blind placebo needle for acupuncture research. BMC Complement Altern Med 2007; 7: 31.

Takakura

Takayama

Kawase

, et al. Double-blind acupuncture needle: a potential tool to investigate the nature of pain and pleasure. ISRN Pain 2013; 2013: 825751.

Takakura

Takayama

Kawase

, et al. Double-blind acupuncture needling: does patient reaction reveal needle authenticity? Med Acupunct 2008; 20(3): 169–174.

10.

Takakura

Takayama

Kawase

, et al. Double blinding with a new placebo needle: a validation study on participant blinding. Acupunct Med 2011; 29(3): 203–207.

11.

Takakura

Yajima

. A placebo acupuncture needle with potential for double blinding: a validation study. Acupunct Med 2008; 26(4): 224–230.

12.

Takakura

Yajima

. Analgesic effect of acupuncture needle penetration: a double-blind crossover study. Open Med 2009; 3(2): e54–e61.

13.

Altman

Schulz

. Statistics notes: concealing treatment allocation in randomised trials. BMJ 2001; 323(7310): 446–447.

14.

Moher

Hopewell

Schulz

, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010; 340: c869.

15.

Polit

. Blinding during the analysis of research data. Int J Nurs Stud 2011; 48(5): 636–641.

16.

Polit

Gillespie

Griffin

. Deliberate ignorance: a systematic review of blinding in nursing clinical trials. Nurs Res 2011; 60(1): 9–16.

17.

Sackett

. Commentary: measuring the success of blinding in RCTs: don’t, must, can’t or needn’t? Int J Epidemiol 2007; 36(3): 664–665.

18.

Kolahi

Bang

Park

. Towards a proposal for assessment of blinding success in clinical trials: up-to-date review. Community Dent Oral Epidemiol 2009; 37(6): 477–484.

19.

Schulz

Altman

Moher

, et al. CONSORT 2010 changes and testing blindness in RCTs. Lancet 2010; 375(9721): 1144–1146.

20.

Harris

Taylor

Minor

, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Inform 2019; 95: 103208.

21.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap): a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42(2): 377–381.

22.

Harlow

Stewart

. A population-based assessment of chronic unexplained vulvar pain: have we underestimated the prevalence of vulvodynia? J Am Med Womens Assoc 2003; 58(2): 82–88.

23.

Stata statistical software: release 15 (computer program) . College Station, TX: StataCorp, 2017.

24.

Bang

Davis

. Assessment of blinding in clinical trials. Control Clin Trials 2004; 25(2): 143–156.

25.

Moroz

Freed

Tiedemann

, et al. Blinding measured: a systematic review of randomized controlled trials of acupuncture. Evid Based Complement Alternat Med 2013; 2013: 708251.

26.

Freed

Assall

Panagiotakis

, et al. Assessing blinding in trials of psychiatric disorders: a meta-analysis based on blinding index. Psychiatry Res 2014; 219(2): 241–247.

27.

Hedeker

Gibbons

. Longitudinal data analysis. Hoboken, NJ: John Wiley & Sons, 2006.

28.

Enders

. Applied missing data analysis. New York: Guilford Press, 2010.

29.

Graham

. Missing data analysis: making it work in the real world. Annu Rev Psychol 2009; 60: 549–576.

30.

Bang

. Random guess and wishful thinking are the best blinding scenarios. Contemp Clin Trials Commun 2016; 3: 117–121.

31.

Bang

Park

. Blinding in clinical trials: a practical approach. J Altern Complement Med 2013; 19(4): 367–369.

32.

Bang

Davis

. Assessment of blinding may be inappropriate after the trial: response to Henneicke-von-Zepelin and Harri Hemili’s comment. Contemp Clin Trials 2005; 26(4): 514–515.

33.

Park

Bang

Cañette

. Blinding in clinical trials, time to do it better. Complement Ther Med 2008; 16(3): 121–123.

34.

Arandjelović

. A new framework for interpreting the outcomes of imperfectly blinded controlled clinical trials. PLoS ONE 2012; 7(12): e48984.

35.

Chow

Shao

. Analysis of clinical data with breached blindness. Stat Med 2004; 23(8): 1185–1193.