Abstract
Purpose:
Safety flag (SF) protocols are increasingly used in adolescent substance use research to protect of minor participants. This report examines the relationship between different thresholds for reporting participant substance use and study attrition.
Methods:
Data were analyzed from 2 concurrent adolescent studies that used an identical SF protocol with the exception of the threshold for heavy episodic drinking (HED). The first study recruited participants from a primary care adolescent medical clinic and the threshold for clinical intervention for past 3-month heavy episodic drinking was 10+ drinks on a single occasion. The second study recruited youth with chronic medical conditions from subspecialty pediatric clinics and used a lower threshold of 3+ to 5+ drinks on a single occasion. Generalized estimating equations were used to assess associations between SF type and subsequent attrition.
Results:
The baseline analytic sample size was 921 (10+ threshold = 487, 3+ to 5+ threshold = 434). No significant relationship between SF type and attrition was observed in either cohort.
Conclusion:
Positive SFs were not associated with significant differences in participant attrition. These results suggest that researchers can use different SF thresholds to accommodate adolescent populations with varying levels of risk without compromising study goals.
Keywords
Highlights
Safety flag (SF) protocols protect minor participants in substance use research.
Triggering a safety flag was not associated with study attrition in adolescent cohorts.
SF thresholds can be tailored to varying adolescent risk levels without compromising recruitment.
Background
The rising number of fatal adolescent drug overdoses and current youth mental health crises make studies on youth substance use and psychiatric illness increasingly important.1,2 Researchers studying high-risk adolescent behaviors are often challenged with protecting both participant privacy and safety. Institutional review boards require protections in place for minor participants; behaviors that place a young person at acute risk of harm may require informing an adult, even if the information was captured as part of a confidential study report. Safety flags (SFs), or designated thresholds for clinical intervention that establish clear, actionable guidelines on when researchers must disclose confidentially collected research information based on responses to survey questions, are a tool that researchers use to identify high-risk study participants.3-5
While certain safety strategies used in adolescent substance use research appear to bias study recruitment (eg, requisite parental consent), 6 our group previously found that use of SF protocols for depression and substance use behaviors did not negatively impact overall study recruitment. 5 Similarly, another group researching heavy alcohol use among college students noted no association between triggered SFs and participation in follow-up surveys. 3 However, it has not yet been demonstrated whether the use of lower SF thresholds in high-risk populations (eg, youth with chronic medical conditions) might correlate with greater participant dropout. If SF protocols include actionable steps that might lead to disclosure of an adolescent participant’s high-risk behavior, this could result in participant attrition and compromise study goals. Comparing participation retention between 2 study protocols that used different SF thresholds for disclosing heavy episodic drinking (HED), this article investigates whether use of a lower SF threshold (resulting in a higher number of positive SFs) is associated with greater rates of attrition by adolescent participants.
Methods
This report is a secondary analysis of safety protocol outcomes from 2 separate research studies that incorporated the same SF protocol with variant thresholds for HED. The first study (10+ threshold) recruited adolescents aged 14 to 18 years presenting to an adolescent medicine primary care clinic for routine care between February 3, 2016 and January 16, 2018 to participate in a longitudinal study of substance use and related problems. Participants were surveyed at baseline and contacted for follow-up surveys at 6 and 12 months. Results from this study, including detailed methods, have been published elsewhere.5,7,8
The second study (3+ to 5+ threshold) recruited a cohort of youth aged 14 to 18 years with chronic medical conditions presenting for care at 1 of 3 subspecialty pediatric clinics between May 11, 2017, and November 20, 2018, to participate in a randomized clinical trial of a preventive intervention about alcohol use. Participants randomized to the intervention group viewed a brief intervention focused on reducing alcohol use while those randomized to the control group received usual care (ie, no preventive intervention). Participants completed assessment batteries at baseline and were contacted for follow-up surveys at 6 and 12 months. Results of this study have also been published elsewhere.9,10
Both studies were approved by the Boston Children’s Hospital Institutional Review Board with waivers of parental permission. During the consent procedure, research assistants explained that survey data would be kept confidential, unless a participant’s response indicated that they were at “high risk of harm,” in which case information would be shared with the participant’s healthcare provider.
SF Protocol
SF thresholds were set for assessments of HED, illegal drug use, combined use of drugs and alcohol (sedative/pain medications and stimulants), and depression. Threshold levels were determined by the research team in conjunction with hospital clinicians, and the exact wording of these questions can be found in the “Assessment Battery” section. For both studies, the threshold for a depression SF was a Patient Health Questionnaire-2 (PHQ-2) score ≥3. Any illegal drug use, as well as use of any alcohol combined with sedatives, pain medications, or stimulants also triggered an SF in both studies. The SF threshold for heavy alcohol consumption differed between studies. Heavy episodic drinking was considered 10 or more drinks on a single occasion in the study performed within the primary care clinic, and “binge level alcohol consumption” (ranging from 3+ to 5+ drinks dependent on participant age and gender 14 ) was used in the study performed within the specialty care clinics due to elevated risks associated with alcohol consumption in youth with chronic medical conditions.11-13
If a participant triggered an SF on the baseline assessment battery, the participant was notified immediately and the participant’s healthcare provider was contacted by the study team. For the 2 follow-up surveys, positive SFs were identified by RAs checking survey results on REDCap. Both the participant and the participant’s healthcare provider were notified about the positive SF within 48 hours via a phone call and/or email. The SF protocol did not require notification of the minor participant’s parent or guardian. This was consistent with the standard practice of adolescent medicine in the hospital system, where adolescent disclosure of high-risk behavior is kept confidential excepting cases of imminent, significant harm to self or others.
Assessment Battery
For both studies, baseline assessments were completed in person on a tablet computer with a privacy film. Links to follow-up surveys were sent by email or text message and completed by adolescents outside of the clinic. Demographic information was collected from all participants, including age, gender, race/ethnicity, number of parents in the home, and highest level of parental education. Both studies assessed participants for symptoms of depression, substance use, and alcohol use via the measures listed below:
HED: In the past 3 months have you consumed on 1 occasion (1) 3/4/5 or more drinks containing alcohol (number varied depending on age and gender) or (2) 10 or more drinks containing alcohol? (yes/no). In the study using the 3+ to 5+ threshold, HED was defined as >3 drinks for 9- to 13-year-olds, >4 drinks for boys and >3 drinks for girls 14 or 15 years of age, and as >5 drinks for boys and >3 drinks for girls 16 or 17 years of age. 14 The threshold of 10+ drinks on 1 occasion was based on prior definitions of high-intensity drinking in other studies of adolescent alcohol consumption.15,16
Illegal Drugs: In the past 12/6 months, how many times have you used illegal drugs (such as cocaine, Ecstasy, or Molly)? In the past 12/6 months how many times have you used inhalants (such as nitrous oxide)? In the past 12/6 months, how many times have you used herbs or synthetic drugs (such as salvia, “K2,” or bath salts)? (1 = never, 2 = once or twice, 3 = monthly, 4 = weekly or more). These questions were derived from Screening to Brief Intervention Screen. 17 Any response other than “none” was considered “use of illicit drugs.”
Combined Alcohol and Sedatives/Pain Medications: In the past 3 months, have you combined alcohol and sedatives (such as Xanax, Klonopin, Valium, etc.) or alcohol and pain medications (such as Vicodin, Percocet, OxyContin etc.)? (yes/no). 18
Combined Alcohol and Stimulants: In the past 3 months, have you combined alcohol and prescription stimulant medications (such as Ritalin, Adderall, etc.)? (yes/no). 18
Depression: The PHQ-2 was used to assess depressive symptoms. 19 PHQ-2 scores ≥3 indicate need for further evaluation.
Attrition: Participants who did not complete surveys at the next adjacent timepoint were considered “drop outs.”
Analyses
Data cleaning removed observations with missing values on the SF criteria items at each time point. We assessed the associations stratified by cohort, due to differences in the study design and threshold for SFs. Generalized estimating equations (GEEs) with a logit link were used to assess the association between each type of previous SF and adjacent attrition with an interaction term used to stratify results within cohorts while adjusting for time (model 1), as well as time, age, and gender at baseline (model 2). Because the outcome is relatively rare, we limited the covariates to avoid overfitting, focusing on age and gender – both of which have been associated with attrition.20,21 The robust sandwich estimator of variance was used to account for correlated measurements within each participant from multiple data collection time points. In a sensitivity analysis to ascertain the association, we used GEEs with a log link and binomial distributions to estimate the risk ratio. The data were managed and analyzed using SAS 9.4 (SAS Institute Inc., Cary, NC, USA) and R version 4.3.1 (R Foundation for Statistical Computing, Vienna, Austria). 22 Statistical precision was determined with 95% confidence intervals.
Results
The baseline analytic sample size was 921 (10+ threshold = 487, 3+ to -5+ threshold = 434) after excluding participants who missed SF-related questions (n = 51) or who submitted more than 1 initial baseline survey (n = 3). The mean age for each cohort was 16.0 years, with both study samples identifying as predominantly female (10+ threshold = 64.7%; 3+ to 5+ threshold = 50.9%). Several sociodemographic and clinical baseline variables did vary significantly between the 2 cohorts (P < .05); participants recruited from the primary care cohort were more likely to identify as female and a racial/ethnic minority, report non-college educated parents and a single-parent household, greater GAD-2 and PHQ-2 scores, and report past-year alcohol, cannabis, and non-medical prescription drug use. There was no significant difference in the number of participants who reported a history of heavy episodic alcohol use in the prior 3 months on the baseline assessment (Table 1).
Baseline Sample Characteristics. a
Abbreviations: PHQ-2, Patient Health Questionnaire-2; GAD-2, Generalized Anxiety Disorder 2-item; HED, heavy episodic drinking.
Mean (SD); n (column %). b Includes American Indian or Alaska Native, Native Hawaiian or other Pacific Islander, multiple races, or other race. c Three plus drinks (ages 14-17)/4+ drinks (age >17) for females; 4+ drinks (ages 14-15)/5+ (ages >15) for males.
The flowchart in Figure 1 shows the number of respondents in each cohort at every timepoint, as well as how analytic sample sizes were determined. Of the 524 respondents in the baseline primary care cohort (10+ threshold), 71 (13.5%) did not return for the 6-month survey, and an additional 41 participants did not respond at the 12-month timepoint. However, 27 participants who missed the 6-month follow-up survey returned to complete the 12-month survey, increasing the number of total respondents in the primary care cohort to 439 at the 12-month follow-up.

CONSORT flow diagram of recruitment and attrition, by cohort.
In the specialty care cohort (3+ to 5+ threshold), 23 of the baseline 451 respondents did not complete the 6-month survey, and an additional 22 participants did not respond at the 12-month follow-up. As with the primary care cohort, respondents who missed the 6-month survey returned to complete the 12-month survey (n = 9), increasing the number of respondents to 415 in the specialty care cohort at the 12-month follow-up.
In the primary care cohort, 34 respondents were removed from the baseline sample and 27 from the 6-month sample because they did not answer the SF questions. Three duplicate respondents from the primary care cohort were removed from both the baseline and 6-month analytic samples. In the specialty care cohort, 17 respondents did not complete the baseline SF questions and 18 did not complete the 6-month SF questions.
At baseline, 141 participants had a positive SF (10+ threshold = 84/487, 17.2%; 3+ to 5+ threshold = 57/434, 13.1%). Of these, 9 (10+ threshold = 8/84, 9.5%; 3+ to 5+ threshold = 1/57, 1.8%) dropped out of the study entirely (Table 2). Regardless of SF type, no evidence was found of an association between SFs and attrition at the next assessment timepoint in either study cohort, and there appeared to be no difference in attrition rates within cohorts (Table 3). A sensitivity analysis using relative risk ratios was performed to ascertain the magnitude of the association, and similar results were demonstrated (Supplemental Appendix Table 1).
Safety Flags Across Both Timepoints by Cohort.
Abbreviation: HED, heavy episodic drinking.
HED (10+ drinks threshold or 3+ to 5+ threshold), use of alcohol with sedatives or pain medications, or stimulants/illicit drug use.
Association Between Previous Safety Flag Status and Adjacent Survey Attrition. a
Abbreviations: AOR: adjusted odds ratio; CI: confidence interval; HED: heavy episodic drinking.
Adjacent survey attrition is defined as non-response to adjacent surveys without re-enrollment in further follow-up surveys up to 12 months. b The model (model 1) includes the predictor and the outcome, adjusting for time. The association between the outcome and previous substance use-related safety flag or mental health-related safety flag was assessed separately. c The model (model 2) includes the predictor and the outcome, adjusting for time, gender, and age at baseline. The association between the outcome and previous substance use-related safety flag or mental health-related safety flag was assessed separately.
In both cohorts, percentages of youth with college-educated parents were significantly lower at the 12-month timepoint (10+ threshold: 52.6% vs 30.8%, P < .001; Supplemental Appendix Tables 2 and 3); in the primary care cohort (10+ threshold), the percentage of Hispanic/Latino youth was higher at 12 months (33.3% vs 47.4%, P = .04), while the percentage of female participants was lower (66.7% vs 53.8%, P = .03; Supplemental Appendix Table 3). In the specialty care cohort (3+ to -5+ threshold), there was no relationship between study arm assignment and attrition (Supplemental Appendix Table 2).
Discussion
This innovative study examines the relationship between positive SFs and retention of adolescent research participants, and our results support the use of SFs to assess and intervene upon high-risk behaviors in adolescent research, even if more restrictive safety thresholds are required to protect vulnerable populations. Our finding of no clear association between SFs and attrition at the next study timepoint suggests that SF protocols with variable thresholds may be useful in the study of substance use in medically complex adolescent populations.
Furthermore, although substance use screening is a recommended part of routine healthcare for all adolescents, rates of screening vary considerably in practice.23,24 Many clinicians are unaware of, or uncomfortable using, standardized and validated screening tools,23,25 which may miss high-risk use. Thus, participating in a research study protocol that uses validated screens could benefit participants by identifying high-risk substance use and bringing it to the attention of a medical professional. 17 While our study did not assess whether SF protocols might offer an opportunity for additional screening for – and intervention upon – adolescent high-risk behaviors, future studies on this topic should explore this possibility in their methodological design.
Finally, while these findings provide novel data on SF protocols for adolescent HED, there are few published studies on use of SFs within existing literature. 5 Future work should explore how SF protocols might affect research and clinical care across diverse high-risk behaviors, and across different populations of youth. For example, rising rates of suicidality and drug use within specific racial, ethnic, gender, and sexual minority groups26-29 necessitate recruiting participants from these vulnerable groups. However, mistrust of the healthcare system is high among racial/ethnic and gender/sexual minority groups, especially among youth who identify with more than 1 minority population. These adolescents may be particularly reluctant to enroll in studies that ask about engagement in illegal activities and may be more likely to drop out if when SF thresholds are lower. It will be important for researchers to determine whether SF protocols might disproportionately promote attrition of those high-risk youth for whom this research is most needed.
Limitations
These results should be interpreted with several caveats. First, our study protocol resulted in only a small number of SFs, increasing the possibility of a type II error, and a difference in attrition rates being missed. Additionally, although our study population was diverse, it was recruited from a single medical institution, limiting generalization of our findings to youth outside the geographical area and reach of participating clinics. Study methodology also did not collect comprehensive data on reason for attrition from those participants who dropped out of the study, nor data estimating “baseline” attrition in a study of youth substance use without an SF protocol (due to concerns about safety of minor participants). Thus, we are unable to conclude that attrition was caused by an SF, or that attrition would not have occurred without an SF occurring.
Finally, participants enrolled in specialty care clinics do not represent all high-risk adolescent populations who might benefit from SF protocols using lower thresholds for substance use. While identifying (and intervening upon) high-risk substance use is standard practice in many adolescent health clinics (both specialty and primary care), 30 we did not collect comprehensive data on how participants’ healthcare providers managed notifications of positive SFs. We could not know the frequency with which positive SFs resulted in the healthcare provider notifying a participants’ parent or guardian about the result, or whether a positive SF directly benefitted any participant by facilitating an intervention upon unsafe behaviors. Participants with chronic medical conditions (such as those in our 3+ to 5+ drink threshold cohort) often recognize the potential health ramifications from use of drugs and alcohol, 31 so may be more accepting of disclosure of their substance use than youth without chronic medical conditions.
Similarly, there is some research demonstrating that substance use-related SFs may impact participant behavior differently depending upon demographic characteristics (eg, gender or sex). 4 Compared to those in the primary care cohort, our specialty-care clinic participants were less likely to be female gender, endorsed better baseline mental health and were more likely to report college-educated parents and a 2-parent household; female gender and parental education were also sociodemographic factors associated with overall study dropout. These factors could influence how a minor participant reacts after an SF, and require further study in the same way that racial/ethnic and gender-minority populations merit additional attention, as noted above.
Conclusion
SF protocols that provide confidentially collected study information to a healthcare provider may be a useful strategy to optimize the safety of adolescent study participants who report high-risk behaviors without compromising study recruitment goals.
Supplemental Material
sj-docx-1-saj-10.1177_29767342261450960 – Supplemental material for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research
Supplemental material, sj-docx-1-saj-10.1177_29767342261450960 for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research by Meredith Gansner, Elissa R. Weitzman, Machiko Minegishi, Melissa Brogna, Lydia A. Shrier, Katharine C. Garvey, Fatma Dedeoglu, Laurie N. Fishman and Sharon Levy in Substance Use & Addiction Journal
Supplemental Material
sj-docx-2-saj-10.1177_29767342261450960 – Supplemental material for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research
Supplemental material, sj-docx-2-saj-10.1177_29767342261450960 for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research by Meredith Gansner, Elissa R. Weitzman, Machiko Minegishi, Melissa Brogna, Lydia A. Shrier, Katharine C. Garvey, Fatma Dedeoglu, Laurie N. Fishman and Sharon Levy in Substance Use & Addiction Journal
Supplemental Material
sj-docx-3-saj-10.1177_29767342261450960 – Supplemental material for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research
Supplemental material, sj-docx-3-saj-10.1177_29767342261450960 for Participant Attrition Associated With Longitudinal Use of Safety Flag Protocols in Adolescent Substance Use Research by Meredith Gansner, Elissa R. Weitzman, Machiko Minegishi, Melissa Brogna, Lydia A. Shrier, Katharine C. Garvey, Fatma Dedeoglu, Laurie N. Fishman and Sharon Levy in Substance Use & Addiction Journal
Footnotes
Ethical Considerations
This study was conducted in accordance with the Declaration of Helsinki. The Institutional Review Board at Boston Children’s Hospital approved this study (IRB-P00021649).
Consent to Participate
Written informed assent was required for participation. During the consent procedure, research assistants explained that survey data would be kept confidential, unless a participant’s response indicated that they were at “high risk of harm,” in which case information would be shared with the participant’s healthcare provider.
Author Contributions
MG, SL, and ERW designed the study. MG and SL drafted the initial article. LAS, KCG, FD, and LNF collaborated in the study design and provided clinical support in data collection. MB collaborated in the study design and analysis. MM carried out data analysis. All authors reviewed and revised the article. All authors approved the final article as submitted and agree to be responsible for all aspects.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute of Drug Abuse (K23; Grant #1K23DA055916-01A1) and the Conrad N. Hilton Foundation (Grant #20140273). Supporting organizations had no further role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Meredith Gansner reports financial support was provided by National Institute of Drug Abuse. Sharon Levy reports financial support was provided by The Conrad N. Hilton Foundation. Sharon Levy reports a relationship with ARMR that includes consulting or advisory. Elissa R. Weitzman reports a relationship with ARMR that includes consulting or advisory. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article.
Data Availability Statement
Data can be made available upon request.
Clinical Trials Registry Site and Number
Boston Children’s Hospital NCT02803567.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
