Safety and Efficacy of a Digital Therapeutic for Substance use Disorder: Secondary Analysis of Data from a NIDA Clinical Trials Network Study

Abstract

Background: Traditional treatments for substance use disorders (SUDs) rely heavily on face-to-face interactions, which pose substantial limitations for patients. A clinical trial of a digital therapeutic (DT), delivering behavioral therapy demonstrated safety and efficacy in a population including patients with opioid use disorder (OUD) not treated with buprenorphine, which is not a guideline-recommended approach. This study re-analyzed the data excluding patients with OUD to more closely approximate real-world patient populations. Methods: Secondary analysis of patients with substance use disorders related to alcohol, cannabis, cocaine, or other stimulants (n = 399, patients with OUD excluded) from a previously-published randomized controlled trial. Patients received 12-weeks of outpatient treatment-as-usual (TAU; n = 193) or TAU with reduced counseling plus a DT (n = 206) providing computerized cognitive behavioral therapy and contingency management. Primary outcomes were abstinence in weeks 9–12 and retention in treatment. Results: The 399 patients in this analysis (206 in the DT group and 193 in the TAU group) reported substance use disorders related to alcohol, cannabis, cocaine, or other stimulants (e.g., methamphetamines). Demographic and baseline characteristics including age, sex, race, education, and reported primary substance use disorder were balanced between treatment groups. Abstinence was significantly higher in the DT group compared to TAU (40.3 vs. 17.6%; p < 0.001) as was retention in therapy (76.2 vs. 63.2%, p = 0.004). Intergroup adverse event rates were not significantly different (p = 0.68). Conclusions: The results demonstrate that use of a DT safely increased abstinence (reduced substance use) and retention in treatment among patients with substance use disorders related to alcohol, cannabis, cocaine, or other stimulants (including methamphetamines).

Keywords

Prescription digital therapeutic PDT reSET reSET-O therapeutic education system TES

Background

Substance use disorder (SUD) is a chronic, relapsing disease with devastating consequences on the social, psychological, and financial health of patients, their families, and their communities. 1

Despite rising substance use and substance-related harms, treatment for SUDs is inadequate. SUD affects up to 41 million people in the United States, and yet the most recently available data show that only about 6.5% of these people ever receive SUD treatment. 2 Although there are effective, evidence-based behavioral treatments for SUD (with, or without pharmacological therapy) 3 most patients do not receive behavior-based therapies due to a lack of specialty facilities and/or trained healthcare providers. 4 – 6 For those who do receive behavioral treatment, clinical efficacy is eroded by inconsistent delivery, quality, and fidelity across healthcare providers, 7 as well as high attrition rates among providers and limited supervision. 8

Digital therapeutics (DTs) are software-based disease treatments that may safely expand access to evidence-based behavioral treatments for SUDs and improve outcomes. 9 DTs may also enhance the therapeutic relationship by giving clinicians real-time information about their patients and their progress in treatment.

The Therapeutic Education System (TES) is a DT for SUD that patients accessed via a browser on an internet-connected device. A commercial version of TES (reSET®) is comprised of equivalent therapeutic content but is accessed on mobile devices and includes a clinician interface for monitoring and communicating with patients. Currently reSET is the first, and only, FDA-authorized prescription digital therapeutic for patients with cocaine, cannabis, or stimulant (including methamphetamines) use disorder. 10

FDA authorization of reSET was based on a set of analyses evaluating the 2014 pivotal randomized trial of TES, which showed improved abstinence and retention in treatment among 507 patients reporting substance use disorders related to alcohol, cannabis, opioids, or cocaine and other stimulants. 11

Three previous studies had evaluated the digital therapeutic in patients with opioid use disorder (OUD) in combination with medications for OUD (MOUD). 12 – 15 The clinical trial discussed here sought to evaluate an all-comer population of patients with SUDs, most of whom were using more than one substance. In the study, patients reporting primary OUD did not receive MOUD because medication would have been a confounder in comparisons with other patients who were not receiving pharmacotherapy. Since the initial publication, the use of buprenorphine has increased substantially for patients with OUD due to its demonstrated efficacy, 16 ^, 17 hence the population of individuals indicating opioids as their primary substance in the original study may not be fully representative of current patient populations.

The secondary analysis presented here extends prior investigations by delineating the safety and efficacy of TES in the subsample of patients on which FDA authorization of reSET is based: those who reported their primary substance as alcohol, cannabis, cocaine, or other stimulants (e.g., methamphetamines). The interaction of abstinence status at study entry and during treatment was explored to determine efficacy of TES among patients who were either abstinent or non-abstinent at baseline.

Methods

Study design

The methodology of the clinical trial has been described previously. 11 ^, 18 An overview emphasizing methods pertinent to the current secondary analysis is provided here. The trial was conducted at 10 sites in the U.S. affiliated with the National Drug Abuse Treatment Clinical Trials Network. 19 The sites offered intensive community-based treatment for SUD. Study procedures were reviewed and approved by an institutional review board for each trial site. All enrolled patients provided written informed consent. Study data were collected between April 2010 and September 2012 (ClinicalTrials.gov number: NCT01104805).

Patients

Enrolled patients were 18 years or older who were accepted for outpatient SUD treatment at participating sites and were within their first month of treatment. 18 Eligible patients could self-report any substance use problem, reported drug use at least once in the prior month, and could have any number of prior addiction and/or psychiatric treatment episodes. Patients who indicated alcohol as the primary substance causing them problems were included only if they reported use of an additional substance. A total of 507 patients were randomized.

The data described here includes the cohort of patients (n = 399) with primary alcohol, cannabis, cocaine, or other stimulant use disorder. The analysis excluded patients with primary OUD for the reasons previously described.

Interventions

Patients were randomly assigned (1:1 ratio) to receive either treatment-as-usual (TAU control group) or reduced TAU plus the TES digital therapeutic (DT group). Patients in the TAU group received 12 weeks of the standard intensive outpatient treatment offered by each collaborating treatment program. TAU consisted of a combination of individual and group face-to-face counseling sessions (on average 4–6 hours per week across programs, with a minimum of 2 hours per week) and pharmacotherapy for some patients. Patients in the DT group received 12 weeks of the DT plus TAU with reduced face-to-face counseling (approximately 2 fewer hours of counseling per week). The TES program was primarily accessed via desktop computers located on-site in participating clinics; however, participants had the option to access TES offsite. Individual participant access patterns show that 39.8% of participants accessed TES outside of the clinic during the study.

The DT consisted of 32 core education modules and 30 supplemental modules with each module designed to last 20–30 minutes. 11 Core modules focus on building basic cognitive behavioral and relapse prevention skills, and provide education about risky behaviors and preventing infections transmitted either sexually or via shared needles. Supplemental modules seek to improve psychosocial functioning (e.g., managing relationships, building communication skills, and improving time management) and provide in-depth training on preventing or living with infections. Patients receiving the DT were recommended to complete at least 4 modules per week with a targeted completion of all 32 core modules over the first eight weeks, followed by repetition of previously completed core modules or completion of select supplemental modules over the subsequent 4 weeks.

Patients randomized to the DT also earned contingency management incentives based on objective evidence of abstinence (i.e., negative urine drug screens and breath alcohol tests) and for module completion. A computerized prize bowl system delivered earned incentives. Half of the rewards were virtual and included positive reinforcement messages such as “good job” and half of the rewards were tangible prizes ($1, $20, or $100), with higher-value rewards occurring with less frequency. Patients in the TAU group did not receive contingency management incentives. 13 ^, 15

Statistical analysis

Power and sample size determinations of the full analysis set have been described. 11 Abstinence (achieving no substance use) during weeks 9–12 and retention in treatment (time to last face-to-face contact) were co-primary outcomes.

Substance use was measured twice per week and defined as a binary measure of abstinence or non-abstinence in half-week intervals based on self-report, urine drug tests, and breath alcohol tests. Missing assessments were imputed as positive for substance use. Odds of abstinence (no substance use) were determined using Generalized Estimating Equations (GEE) with factors for treatment, time, and treatment × time interaction with subject as a random factor. The analysis was repeated with baseline abstinence status and baseline abstinence status × treatment × time interaction included as additional factors in the model. This method was selected because it provides a framework for accounting for correlated assessments within a participant across time.

Retention was analyzed as time-to-event data using the Kaplan–Meier method to estimate the percentage of patients retained over time. Retention distribution between the treatment groups was compared using a log-rank test. For patients who completed 12 weeks of treatment, time to withdrawal was censored at 84 days.

Compliance was assessed by calculating the number of counseling sessions attended relative to the number of counseling sessions prescribed.

Safety endpoints included comparison of the number of reported adverse events between treatment groups. Adverse events were monitored from signing of the informed consent until end of study participation.

Reported P values for demography, compliance, and safety comparisons are based on two-sample t-tests. All analyses used the intent-to-treat model (within the subpopulation studied) and were implemented using SAS Version 9.2 statistical software.

Results

Demographics

The 399 patients in this analysis (206 in the DT group and 193 in the TAU group) reported substance use disorder related to alcohol, cannabis, cocaine, or other stimulants (e.g., methamphetamines). Within this group, 101 in the DT group and 91 in the TAU group were not abstinent at baseline. Demographic and baseline characteristics including age, sex, race, education, and reported primary substance use disorder were balanced between treatment groups (Table 1).

Table 1

Demographics and Baseline Characteristics.*

Outcome	rTAU + reSET (n = 206)	TAU (n = 193)
Mean age at screening (± standard deviation)	37.1 (10.87)	35.4 (10.97)
Female sex – no. (%)	73 (35.4%)	85 (44.0%)
Race
White	98 (47.6%)	94 (48.7%)
Black or African American	65 (31.6%)	45 (23.3%)
Asian	5 (2.4%)	7 (3.6%)
Native Hawaiian or Pacific Islander	7 (3.4%)	5 (2.6%)
Multiracial	20 (9.7%)	29 (15.0%)
American Indian or Alaska native	2 (1.0%)	1 (0.5%)
Other	9 (4.4%)	12 (6.2%)
Education
Less than high school diploma	43 (20.9%)	45 (23.3%)
High school diploma or equivalent	134 (65.0%)	117 (60.6%)
More than high school diploma	29 (14.1%)	31 (16.1%)
Primary substance used
Alcohol	58 (28.2%)	46 (23.8%)
Cocaine	53 (25.7%)	49 (25.4%)
Stimulants	33 (16.0%)	36 (18.7%)
Cannabis	54 (26.2%)	60 (31.1%)
Other	8 (3.9%)	2 (1.0%)

There were no significant differences in any of these characteristics (p > 0.05).

Pivotal trial results

Results from analysis of the full cohort (N = 507) in the pivotal trial showed that, compared with patients in the TAU group, those in the digital therapeutic group had a lower dropout rate (HR 0.72, 95% CI 0.57, 0.92) and a greater abstinence rate (OR 1.62, 95% CI 1.12, 2.35). 11 This effect was more pronounced among patients who had a positive urine drug or breath alcohol screen at study entry (n = 228) (OR 2.18, 95% CI 1.30, 3.68). 11

Abstinence (no substance use)

The predicted probability of abstinence during the last 4 weeks of treatment (i.e., weeks 9-12) was 40.3% for the DT group vs. 17.6% for the TAU group (p = 0.0004). The odds of abstinence tripled with the DT (OR 3.17, 95% CI 1.68–5.99) (Table 2).

Table 2

Model-Based Abstinence Rates.

Abstinence outcome	rTAU + reSET (n = 206)	TAU (n = 193)	Odds ratio (95% confidence interval)	p-Value
Primary outcome of percentage of study participants abstinent at weeks 9–12	40.3%	17.6%	3.17 (1.68, 5.99)	<0.001
Number of participants abstinent at baseline	n = 105	n = 102
Percentage of study participants abstinent at baseline and abstinent at weeks 9–12	65.1%	45.7%	2.22 (1.15, 4.27)	0.02
Number of participants not abstinent at baseline	n = 101	n = 91
Percentage of study participants not abstinent at baseline and abstinent at weeks 9–12	16.1%	3.2%	5.74 (1.99, 16.60)	0.001

Among patients abstinent at baseline, the predicted probability of abstinence during weeks 9–12 was 65.1% in the DT group vs. 45.7% in the TAU group (OR 2.22, 95% CI 1.15–4.27; p = 0.02). The predicted probability of abstinence in the last 4 weeks of treatment was substantially lower among those not abstinent at baseline, but were significantly higher in the DT group: 16.1 vs. 3.2% (p < 0.001) (Table 2).

Analyses of abstinent half-weeks across the treatment period demonstrated a significant impact of treatment. Patients in the DT group had a mean of 4.9 consecutive abstinent half-weeks vs. 2.8 in the TAU group (p = 0.001) and had a mean of 10.9 total abstinent half-weeks vs. 8.6 in the TAU group (p = 0.002).

Among those abstinent at baseline, the DT group had a mean of 8.6 consecutive abstinent half-weeks vs. 6.7 in the TAU group (p = 0.001) and had a mean of 15.7 total abstinent half-weeks vs. 12.9 in the TAU group (p = 0.007). Among those non-abstinent at baseline, those receiving the PDT had a mean of 1.6 consecutive abstinent half-weeks vs. 0.4 in the TAU group (p = 0.02) and had a mean of 7.9 total abstinent half weeks vs. 4.2 in the TAU group (p < 0.001).

There was lack of an interaction between treatment and site (p = 0.6095) suggesting that the benefit of TES on abstinence was not mitigated by the site at which a patient was treated.

Treatment retention

Significant improvements in treatment retention (i.e., time to dropout) were observed in the DT group compared with the TAU group (76.2 vs. 63.2%, HR 0.59, 95% CI 0.41–0.85; p = 0.004).

Although patients receiving the DT were prescribed 2 fewer hours of in-person counseling sessions, the total number of hours spent attending in-person sessions was comparable between groups. Participants randomized to TAU spent an average of 3.7 hours per week attending in-person sessions whereas those randomized to the DT spent an average of 3.3 hours per week (p = 0.08).

Safety

The observed adverse events were of the type and frequency anticipated in a large population of patients with SUD (e.g., indicators of cardiovascular, gastrointestinal, and nutritional diseases). 20 – 23 No statistically significant difference in adverse event rates (15.5% in the DT group vs. 14.0% in TAU group, p = 0.68) or number of suicide-related events (3.4 vs. 2.1%, p = 0.55) were noted (Table 3). Except for suicide-related events, none of the other reported adverse events occurred in more than 2% of patients in either treatment group. No adverse events in the DT group were determined by investigators to be related to the therapeutic.

Table 3

Frequency of Reported Adverse Events.

Adverse events	rTAU + reSET (n = 206)	TAU (n = 193)	p-Value
Number of study participants with adverse events	32 (15.5%)*	27 (14.0%)	0.68
Number of study participants with suicide-related events	7 (3.4%)	4 (2.1%)	0.55

One event of stress of moderate severity following a positive urine drug test in a patient in the rTAU + reSET was considered related to the study procedure and was not considered related to the study intervention (reSET). This event was resolved by allowing the participant to continue in the study without providing urine samples.

Discussion

The secondary analyses presented here of results among patients reporting alcohol, cannabis, cocaine, or other stimulants (e.g., methamphetamine) as their primary substance use disorder, provided the basis for FDA-authorization of reSET, the first PDT for patients with SUD. Patients with OUD were excluded from this secondary analysis because the patients in the original trial did not receive recommended medication-assisted treatment for OUD and had poor outcomes. 24 These data are important as they clearly describe efficacy and safety outcomes in the SUD patient population for whom reSET is indicated, where poly substance use was common (i.e., 91% of patients reported using more than one substance).

In the present analysis, patients randomized to the DT TES were more likely to achieve and maintain abstinence (greater than 2-fold improvement) and had improved treatment retention compared to TAU. The safety findings demonstrate no statistically significant increase in adverse events in the DT group compared to TAU. The abstinence data among those not abstinent at baseline (and, thus, plausibly a more difficult-to-treat population) were particularly encouraging with a greater than 5-fold improvement in abstinence rates among the DT group compared to patients receiving TAU.

Polysubstance use is common in this patient population and was reported by 91% of participants, as mentioned above. A prior secondary analysis explored the impact of primary substance use disorder on abstinence outcomes, showing a significant impact of DT treatment for individuals with stimulant use disorder users compared to TAU, and improvements for individuals with alcohol and cannabis use disorders compared to TAU that were promising, but did not reach significance. 24 Individuals in the digital therapeutic group with opioid use disorder did not demonstrate improvement in abstinence compared to TAU. 24 Given that the treatment modalities utilized by TES (e.g., CBT/CRA and CM) are well established as effective for a broad range of SUDs, the findings reported here are consistent with the literature supporting the efficacy of these modalities, and the data may be interpreted as generalizable across substance use disorders related to alcohol, stimulants, and cannabis

Participation in SUD treatment is key to successful treatment outcomes. 25 ^, 26 These analyses suggest that the DT encouraged participation in treatment: the aggregate number of in-person counseling session hours were similar between groups, despite a reduction in the recommended amount by 2 hours per week for participants randomized to the prescription digital therapeutic. This, in turn, suggests that DTs could play a role as a “clinician-extender” to expand and improve treatment. These data are consistent with a real-world study showing high levels of engagement and improvement in outcomes among 3144 patients using the related PDT reSET-O, (indicated for patients with OUD on buprenorphine therapy), such as reductions in emergency room visits and inpatient hospitalizations among patients treated with the PDT. 27 ^, 28

An urgent need exists for therapeutic approaches that overcome logistical limitations and provide patients with high-quality behavioral SUD treatments they can access remotely, a need that has grown acute with the COVID-19 pandemic 29 ^, 30 but which will remain even after the pandemic has faded.

Limitations

The data on which these analyses are based were obtained approximately 10 years ago, and the authors recognize that substance use patterns and substances themselves may have evolved over time. In addition, although patients who received the DT had superior outcomes compared to those who received TAU, patients in the DT group received contingency management rewards for negative urine drug screen and module completion as part of using the digital therapeutic. It is not possible, therefore, to determine how much of the positive results reported here are attributable to the specific therapeutic mechanisms of action/components of the therapeutic and how much are attributable to contingency management.

This study was not designed or powered to evaluate differences between individual substance use disorders, nor was designed or powered to assess the impact of polysubstance use on outcomes. Differences by site were observed, as is expected in any multi-site trial, however, there was no significant interaction of site by treatment, as reported previously. 11

Summary

This study demonstrates the safety and efficacy of a digital therapeutic for SUD among patients reporting substance use disorders related to alcohol, cannabis, cocaine, or other stimulants. Digital therapeutics offer consistent, asynchronous delivery of high-quality neurobehavioral treatment for SUD that patients can access 24/7 without geographical limitations (once the program is downloaded). By expanding access to proven treatments, DTs have the potential to safely improve treatment outcomes in patients grappling with the difficult challenges of SUD.

Footnotes

Acknowledgments

We thank the patients and their families, the trial coordinators, and the counselors. Editorial assistance was provided by Karishma Manzur, PhD of Lenimen Consulting, Inc., Kate Jones of Global MedCom Consulting, LLC, and Stephen Braun and Nicole Enman of Pear Therapeutics Inc.

Disclosure statement

Drs. Maricich and Luderer are employees of Pear Therapeutics, Inc. Dr. Campbell has no conflicts of interest to disclose. Dr. Nunes is a consultant of University of Arkansas and Alkermes outside of the submitted work. Mr. Botbyl is a consultant for Pear Therapeutics Inc. No potential conflict of interest was reported by the author(s).

Author contributions

YAM contributed to the conception and design of the analysis, and assisted with review and revision of the manuscript. EVN and ANCC contributed to the design and conduct of the trial and assisted with conceptualization and execution of the data analysis. JDB performed the analysis and contributed to the writing and editing of the manuscript. HFL contributed to conception and design of the analysis and writing and editing of the manuscript.

References

National Academies of Sciences Engineering and Medicine. Medications for Opioid Use Disorder Save Lives. Washington, DC: The National Academies Press 2019.

Substance Abuse and Mental Health Services Administration. Key substance use and mental health indicators in the United States: results from the 2020 National Survey on Drug Use and Health (HHS Publication No. PEP21-07-01-003, NSDUH Series H-56). https://www.samhsa.gov/data/. Accessed October 28, 2021.

Miller

, Fielin

, Rosenthal

, Saitz

The ASAM Principles of Addiction Medicine. 6th ed. Philadelphia, PA: Wolters Kluwer; 2018.

Moore

, Fazzino

, Garnet

, Cutter

, Barry

DT.

Computer-based interventions for drug use disorders: a systematic review. J Subst Abuse Treat. 2011;40(3):215–223.

Kiluk

, Sugarman

, Nich

A methodological analysis of randomized clinical trials of computer-assisted therapies for psychiatric disorders: toward improved standards for an emerging field. Am J Psychiatry. 2011;168(8):790–799.

Marsch

, Carroll

, Kiluk

BD.

Technology-based interventions for the treatment and recovery management of substance use disorders: a JSAT special issue. J Subst Abuse Treat. 2014;46(1):1–4.

Institute of Medicine. Improving the Quality of Health Care for Mental and Substance-Use Conditions. Washington, DC: The National Academies Press; 2006.

Lappan

, Brown

, Hendricks

PS.

Dropout rates of in-person psychosocial substance use disorder treatments: a systematic review and meta-analysis. Addiction 2020;115(2):201–217.

Food & Drug Administration. FDA permits marketing of mobile medical application for substance use disorder. https://www.fda.gov/news-events/press-announcements/fda-permits-marketing-mobile-medical-application-substance-use-disorder. Accessed January 26, 2021.

10.

Center for Devices and Radiological Health. De Novo Summary for reSET (DEN160018). Washington, DC: US Food and Drug Administration; 2017.

11.

Campbell

, Nunes

, Matthews

Internet-delivered treatment for substance abuse: a multisite randomized controlled trial. Am J Psychiatry. 2014;171(6):683–690.

12.

Bickel

, Marsch

, Buchhalter

, Badger

GJ.

Computerized behavior therapy for opioid-dependent outpatients: a randomized controlled trial. Exp Clin Psychopharmacol. 2008;16(2):132–143.

13.

Christensen DR , Landes

, Jackson

Adding an internet-delivered treatment to an efficacious treatment package for opioid dependence. J Consult Clin Psychol. 2014;82(6):964–972.

14.

Marsch

, Guarino

, Acosta

Web-based behavioral treatment for substance use disorders as a partial replacement of standard methadone maintenance treatment. J Subst Abuse Treat. 2014;46(1):43–51.

15.

Maricich

, Bickel

, Marsch

, Gatchalian

, Botbyl

, Luderer

HF.

Safety and efficacy of a prescription digital therapeutic as an adjunct to buprenorphine for treatment of opioid use disorder. Curr Med Res Opin 2021;37(2):167–173.

16.

Tseregounis

, Gasper

, Henry

SG.

Trends in buprenorphine to treat opioid use disorder in California, 2012 to 2018: Medicaid outpaces the rest of the state. J Addict Med. 2021;15(5):425–428.

17.

Roberts

, Saloner

, Dusetzina

SB.

Buprenorphine use and spending for opioid use disorder treatment: trends from 2003 to 2015. Psychiatr Serv. 2018;69(7):832–835.

18.

Campbell

, Nunes

, Miele

Design and methodological considerations of an effectiveness trial of a computer-assisted intervention: an example from the NIDA Clinical Trials Network. Contemp Clin Trials. 2012;33(2):386–395.

19.

Weisner

, Ray

, Mertens JR , Satre

, Moore

Short-term alcohol and drug treatment outcomes predict long-term outcome. Drug Alcohol Depend. 2003;71(3):281–294.

20.

Helzer

, Pryzbeck

TR.

The co-occurrence of alcoholism with other psychiatric disorders in the general population and its impact on treatment. J Stud Alcohol. 1988;49(3):219–224.

21.

Grant

, Stinson

, Dawson

Prevalence and co-occurrence of substance use disorders and independent mood and anxiety disorders: results from the National Epidemiologic Survey on Alcohol and Related Conditions. Arch Gen Psychiatry. 2004;61(8):807–816.

22.

Regier

, Farmer

, Rae

Comorbidity of mental disorders with alcohol and other drug abuse. Results from the Epidemiologic Catchment Area (ECA) Study. JAMA 1990;264(19):2511–2518.

23.

National Institute on Drug abuse. Health consequences of drug misuse. National Institute on Drug abuse. https://www.drugabuse.gov/related-topics/health-consequences-drug-misuse. Published 2017. Accessed November 26, 2019.

24.

Cochran

, Stitzer

, Campbell

, Hu

, Vandrey

, Nunes

EV.

Web-based treatment for substance use disorders: differential effects by primary substance. Addict Behav. 2015;45:191–194.

25.

Hser

, Evans

, Huang

, Anglin

DM.

Relationship between drug treatment services, retention, and outcomes. Psychiatr Serv. 2004;55(7):767–774.

26.

Marchand

, Oviedo-Joekes

, Guh

, Brissette

, Marsh

, Schechter

MT.

Client satisfaction among participants in a randomized trial comparing oral methadone and injectable diacetylmorphine for long-term opioid-dependency. BMC Health Serv Res. 2011;11:174.

27.

Maricich

, Xiong

, Gerwien

Real-world evidence for a prescription digital therapeutic to treat opioid use disorder. Curr Med Res Opin 2021;37(2):175–183.

28.

Velez

, Colman

, Kauffman

, Ruetsch

, Anastassopoulos

Real-world reduction in healthcare resource utilization following treatment of opioid use disorder with reSET-O, a novel prescription digital therapeutic. Expert Rev Pharmacoecon Outcomes Res. 2021;21(1):69–76.

29.

Rosenbaum

, Lucas

, Zandrow

Impact of a shelter-in-place order during the COVID-19 pandemic on the incidence of opioid overdoses. Am J Emerg Med. 2021;41:51–54.

30.

Wainwright

, Mikre

, Whitley

Analysis of drug test results before and after the US declaration of a national emergency concerning the COVID-19 outbreak. JAMA 2020;324(16):1674.