Co-Design of a Digital Health Tool for Use by Individuals With Opioid Use Disorder: App4Independence (A4i-O)

Abstract

Background:

Opioid use disorder (OUD) has arguably the highest mortality rate of mental health conditions; opiate-related deaths are identified as the number one cause of accidental deaths in Canada and the United States. Specialized care for OUD is often described as lacking, fractured, and with frequent periods of disengagement. Digital health strategies may support connection to evidence-based resources even during periods of disengagement. However, sustained engagement in digital interventions remains a barrier, and as such, experts recommend using co-design approaches to develop interventions.

Methods:

The current study outlines the results from a qualitative co-design project that engaged 6 lived experts and 8 clinical experts in a series of focus groups and interviews to adapt an existing intervention for use in OUD. Focus groups and interviews were recorded and transcribed before undergoing thematic analysis. This co-design process is the first stage of a larger project that will lead to the development of a novel digital health intervention for OUD populations.

Results:

Transcripts underwent thematic analysis, and themes were divided into Crosscutting Themes, Feasibility and Engagement, and Specific Features. Each theme was divided into specific subthemes, which were reviewed by the design team and informed the design of the digital health platform. Key resulting directions included creating a psychologically safe digital space, curating resources for OUD as a multifaceted condition, and being mindful of barriers to implementation from both lived and clinical expert perspectives. Specific features are discussed in detail in the article.

Conclusion:

Lived experts and clinicians strongly supported integrating digital tools into OUD care. Ongoing work is needed to better understand the role of technology in existing OUD structures as well as the implementation of key features such as digital peer support and creating effective and safe social connections. This study also validates co-design as an essential step in digital health development.

Keywords

digital health interventions health technology opioid use disorder co-design

Highlights

Opioid use disorder (OUD) has been in the spotlight in recent years due to its complexity and associated mortality and health impacts. However, specialized services remain fractured and characterized by periods of disengagement from care. Integrated technological interventions may help support access to and engagement in specialized services and resources. Across the field of digital mental health, there is a concern about technology fit and sustained engagement.

This project utilizes a codesign framework to develop a digital intervention comprised of a patient-facing smartphone app and a clinician-facing web portal for individuals in treatment for OUD. Participants included lived/living and clinical experts (n = 14) engaged in a series of focus groups and interviews.

Results indicate the need for multifaceted, personalized resources to address complex and heterogeneous conditions such as OUD. Results additionally indicate perceived stigma and judgement as key barriers for individuals with OUD using services and resources. For clinicians, sustaining appropriate boundaries while supporting therapeutic rapport with regard to patient safety monitoring was a key concern.

Last, results outline key features and functions for the intervention that are promising with regard to facilitating engagement (eg, involvement of peer support) and supporting recovery.

Introduction

Background

Individuals with opioid use disorder (OUD) represent a particularly vulnerable group. Current estimates place the prevalence of OUD in the United States between 6.7 and 7.6 million people,¹ representing a more than 3-fold increase since 2013.² Likewise, in the United Kingdom, opioid-related hospitalizations have increased by 49% between 2008 and 2018.³ Furthermore, opioid-related deaths have steadily increased since the escalation in the availability of highly potent, illicitly manufactured synthetic opioids, like Fentanyl, in 2012 to 2013. Since then, opioid-related deaths have surpassed all other causes of accidental deaths in both Canada and the United States.^4,5

Furthermore, OUD is a complex condition. There are high rates of comorbidity (30%-50%) with other types of mental health conditions^6,7 and co-occurring substance use (78%).⁸ Comorbid chronic medical conditions are also common, particularly for those using intravenously, like HIV, hepatitis C, and other serious infections.⁹ In addition, this population faces a high prevalence of homelessness, which is associated with increased risk risks in all areas.¹⁰

Compounding these effects is that this population is underserved by specialized concurrent services, leading to decreased treatment access and engagement.¹¹ An estimated 43% of individuals with OUD in the United States go without mental health or substance use care.¹² A lack of access to concurrent treatment approaches creates a negative feedback loop where people may self-medicate due to comorbid conditions, complicating recovery.⁶ Coupled with poor access to treatment is variable care engagement over time.¹³ As such, specialized services require flexibility in periods of disengagement.

Flexibility and extended access to resources may be where digital mental health strategies leveraging personal devices can support recovery for individuals with OUD. Despite popular misconceptions, the majority (up to 94%) of individuals with OUD own smartphones, use the internet daily (67%), and engage in health information seeking via the internet (88%).¹⁴ Additional considerations regarding long-term engagement with digital tools may be required for individuals experiencing homelessness or poverty/low income who still show high cellphone ownership (95%) but also experience high rates of device (56%) and phone number (55%) turnover.¹⁵

Commercially available technologies for OUD have largely focused on assessment-based technologies to support clinicians in predicting/preventing overdose and seldom have undergone efficacy testing.¹⁶ However, technologies capable of delivering specialized OUD resources are steadily gaining traction.¹⁷ In a recent review of technology-based interventions for OUD (n = 20), approximately 70% showed significant improvements in opioid abstinence (n = 10) and treatment engagement (n = 4).¹⁸ The majority of interventions included in this review were internet/computer-based accessed in-person at the clinic, with a minority leveraging personal technology such as cell phones (ie, smartphone apps and text messaging).¹⁸ Thus, though these studies report promising preliminary results, most digital tools for OUD do not substantively increase the scale of OUD treatment. Many are also positioned as adjunctive to treatment with varying levels of clinical integration and additional psychosocial resources available,¹⁸ limiting their potential impact beyond traditional care. Furthermore, sustained engagement with digital tools is an ongoing concern in OUD, with leading technologies reporting 47% attrition despite compensation for module completion.¹⁹ Nonuser-centered design may affect disengagement,²⁰ with calls for increased involvement of lived/living experts in intervention design.

Co-Design

Co-design is an approach to intervention design that places equal importance on academic and lived/living (“lived/ing”) expertise.²¹ It prioritizes co-creation with lived/living experts and other knowledge users (ie, clinicians, policymakers, management, leadership, etc).²² Especially within stigmatized and vulnerable groups, such as those with OUD,^23-25 co-design as an approach may help overcome barriers to participation, including a perceived power imbalance in psychiatry, stigma, and fear of judgment.²⁶ Active collaboration throughout development allows for a better fit between the end user, digital intervention, and context²⁰ to improve uptake, adoption, and engagement. However, many published co-design studies do not include information on how industry partners use and incorporate feedback into the finished product.

The Current Study

The current study outlines the procedures and findings from the initial co-design process of an iterative procedure for App4Independance for OUD (A4i-O), a new version of a validated app used to support individuals with a schizophrenia spectrum disorder that is being modified to support individuals in recovery from opioid addiction.

Methods

App4Independence

The App4Independence (https://www.a4i.me) or A4i is a validated digital health tool with a smartphone application paired with a clinician web-based platform. A4i was initially designed for individuals who experience psychosis. A4i has undergone an initial open pilot trial with early findings suggesting that A4i is feasible with excellent rates of sustained user engagement, positive qualitative feedback, and positive outcomes in multiple symptom domains.²⁷ A larger randomized control trial²⁸ and 2 clinical validation investigations are underway.

A4i architecture is flexible and easily adaptable to conditions such as OUD. A4i is equipped with tools and resources spanning several areas of mental health and can easily accommodate new information. Moreover, A4i is personalized, and content can be specifically tailored to concerns endorsed by the user. The original scaffolding was used as a foundation for A4i-O with the psychosis-specific content removed. Like its predecessor, A4i-O is designed for individuals currently in treatment. Foundational scaffolding is comprised of a home feed to share resources and experiences, sections for specific tools and resources, the ability to leave notes for a care provider of topics to discuss at the next appointment, reminders, and urgent care resources. Beyond this, the following co-design process informed the content, structure, specific features, and overall design of A4i-O. The intended purpose of the A4i-O is to support and facilitate engagement in existing evidence-based care and extend access to evidence-based resources outside of clinic appointments. The A4i-O app is not intended to deliver stand-alone OUD care.

Following these initial focus groups, the development plan for A4i-O includes a small sample beta test followed by an iterative design phase to implement feedback from the beta test. The purpose of the beta test is to evaluate the functioning of A4i-O and identify areas of improvement. During this phase, knowledge users will be engaged regarding app design and implementation, including our lived/living expert advisory board, key administrators from various tertiary and community care settings, and policymakers at the municipal and provincial levels, if possible. Based on these informant meetings, the app will undergo another round of iterative design before our larger clinical integration pilot study. This pilot study will focus on the integration of the A4i-O technology into existing care settings to ascertain a more accurate picture of how A4i-O may support individuals in care for OUD. Subsequent steps for A4i-O will be determined after these preliminary investigations.

Participants

We engaged lived/living and clinical experts in the co-design of A4i-O. We focused primarily on planned users of the technology for this initial design process to prioritize user-technology fit, as fit has been highlighted as a key determinant of user engagement. Once useability and fit have been established, key policymakers and administration will be involved in a later iterative design process prior to implementation trials. We decided a step-wise engagement of different key knowledge users would preserve and prioritize end users’ voices, especially given the inherent power differential between patients and clinicians, and policymakers and administration. Participants were recruited through professional and patient engagement networks. Participants were contacted by leaders in these networks through email advertisements, and participants self-referred to the study. Participants were reimbursed $50.00 CAD for their participation in each focus group or interview that they participated in. This research was reviewed and approved by the Research Ethics Board at the Centre for Addiction and Mental Health in Toronto, Ontario (REB# 042/2022)

Based on factor analysis²⁹ and binomial logic models,³⁰ between 4 to -6 participants/interviews are satisfactory to reach saturation, especially with participants who demonstrate a high level of knowledge on the topic. Recent focus group guidelines additionally recommend between 4 and 8 focus groups³¹ with an upper limit of 10, as too many may lead to a lack of clarity in coding.³² We included 14 participants in total who participated in 6 focus groups and 10 interviews. As such, we believe we have included a sufficient sample to be considered at saturation, providing support for the transferability of our results. However, true transferability is unable to be estimated at the time of study design and can only be properly evaluated using qualitative meta-synthesis methods in comparison with other findings from other studies.

Data Collection

Interested potential participants self-referred to the study, underwent an informed consent process via videoconferencing or phone by research staff, and provided basic demographic information. Participants took part in a series of 3 focus groups with a target sample size of 6 to 8 participants each, lasting approximately 90 minutes each. In the event participants could not attend a focus group, they were offered an interview. Interviews lasted between 60 and 90 minutes each and followed the same question guide as the focus groups. Both were video and audio recorded. Each transcript was reviewed for accuracy, corrected verbatim, and de-identified by a research analyst.

Focus groups and interviews were structured such that questions began on broad topics and became more specific. Topics of the focus groups included:

Use of Technology Broadly and for Mental/Physical Health: In the first focus group, we explored the types of apps participants currently use in general and for mental and physical health and why.

Brainstorming App Features: In the second focus group, we explored more specifically what types of app functions, features, and information would be helpful/unhelpful.

Prototype Reveal: In the final focus group, a design specialist from our industry partner joined as a co-facilitator and presented a paper prototype of the app features based on the information provided in the first 2 focus groups to gather feedback on the prototype.

The first 2 focus groups occurred within 2 to 3 weeks of each other, and the third focus group occurred after approximately 5 to 6 to allow the design team to build a prototype.

Co-Design Best Practices

Several co-design best practices in digital mental health^26,33,34 were followed. First, we engaged 2 types of knowledge users. Second, we offered multiple means of participation, including videoconferencing, phone, or in-person and provided technological support as needed. In addition, we offered both group and individual participation to accommodate the schedules and participant needs and provided multiple types of compensation (ie, gift cards, visa cards, cash). Third, we prioritized data transparency, including the type of information requested, its purpose, data storage, and dissemination of results. We kept participants apprised of delays or difficulties in the project and the expected timelines. Most importantly, we were transparent about potential limitations to implementing design suggestions and openly discussed these in the focus groups and interviews. Fourth, we embodied an equal partnership in the data collection process. To foster collaborative brainstorming, we conducted focus groups conversationally using accessible language (ie, avoiding technical jargon) rather than in a typical question-answer format. Above all, we embodied respect and validation for the expertise of lived/living experts and clinicians.

Thematic Analysis

Braun and Clarke’s inductive thematic analysis method³⁵ is an iterative process consisting of 6 steps: (1) becoming familiar with the data, (2) generating codes, (3) generating themes, (4) reviewing themes, (5) defining and naming themes, and (6) locating exemplars. The first author generated initial codes and themes based on an informal review of the transcripts by the research analyst, which was reviewed for congruence by the research analyst and the last author. Authors JND and LTG reviewed transcripts for themes and exemplars, and the final coding was reviewed by the last author. Themes were based on codes that reached saturation, defined as majority endorsement in each focus group (>50%).

Commercial Design

All feedback provided by lived experts and care providers was considered to formulate an overall roadmap of how an app may support individuals with OUD in recovery. However, the collected information was prioritized based on frequency, potential impact, and feasibility. The frequency in which specific topics/requests were identified was an indicator of the importance of the information, which also had implications for the level of potential impact of suggested features/resources. Feasibility was evaluated from a coordination perspective (ie, how often the information would require updating, the ease of acquiring the information/resource, etc) and a technological perspective (ie, how well the feature could be integrated with the existing architecture).

Results

Participants

A total of 6 lived/living experts with OUD and 8 care providers participated in the series of focus groups and interviews (see Figure 1 for details). Two lived/living experts who provided informed consent to participate in the study were lost to follow-up despite efforts to connect. Of note, all of our lived/living experts identified as white, and all but one participant identified as women (n = 5). The other identified as a man, and we did not have representation from gender/sexual minority groups. The age of our living experts ranged from 33 to 65 (mean age = 37). Other important identities and experiences included 2 participants who had navigated homelessness while in recovery, one who was a new mother, another who was a mother with a son in recovery, and another living in a rural area. Furthermore, our lived/living experts had all served on focus groups and advisory panels previously, demonstrating a high level of knowledge regarding not only their personal experiences with OUD but also the OUD service system, treatment options, and barriers to care.

Figure 1.

Interviews and focus group structure.

Care providers/clinicians included social workers, addiction nurses, clinical managers, and psychiatrists from various settings, including a tertiary hospital, community addiction clinics, and outpatient community mental health clinics in urban and rural Ontario. All our clinical experts identified as women, and all but 2 identified as white (n = 6). The other 2 ethnic identities endorsed included South Asian and Mixed (Latin). The age of our clinical experts ranged from 30 to 45 (mean = 36), and the number of years working with individuals with OUD ranged from 4 to 20 (mean = 9.14).

Themes Identified

Given the overarching research question regarding app design, the resulting themes were divided into 3 categories: Crosscutting Themes, Feasibility and Engagement, and Specific Features. Crosscutting themes emerged as relevant to most, if not all, aspects of app design. These broad themes included (1) personalization and autonomy, (2) transparency, (3) stigma and judgement-free digital space, (4) simplicity in design, (5) OUD as a multifaceted disorder requiring multifaceted recovery, and (6) inviting appearance. Overwhelmingly mentioned was the need for a personalized app experience. Participants stressed that OUD is a complex and heterogeneous condition that requires a personalized response. Participants additionally described OUD as an oft-misunderstood condition and voiced a contextual history of mistrust connected to felt judgement and stigma. To this end, participants felt creating a “safe digital space” free from judgement and stigmatizing language was crucial. Many participants reported that failing to create sense of safety would lead to disengagement. This meant not “pushing” a particular recovery path (ie, abstinence vs harm reduction), using validating language and steering clear of “toxic positivity,” and prioritizing data transparency and autonomy over the information shared with the clinician portal.

Additionally critical, was the need for users to understand the role of clinicians, peer support, and industry partners (ie, access/use of information and supporting users) and the limitations of app functions (ie, not a crisis app). Regarding themes of simplicity and multifunctionality, participants felt that an app that can address multiple facets of OUD while remaining simple to use (ie, understandable instructions, easily navigable) would be most effective. Appearance considerations included ensuring that colors and design do not convey an overly clinical digital space but are not excessively gamified/immature. This approach avoids condescension while still offering rewards/reinforcement for reaching goals. For more fulsome definitions of crosscutting themes, see Table 1.

Table 1.

Overarching/Crosscutting Themes.

Themes/subthemes	Definition	Exemplar quote(s)
1. Personalization and autonomy	Ensuring individuals have control over the types of information being tracked and shared with clinicians, when and how often they are asked to provide information, their goal setting, and personalized skills/information available.	[LE5] “we have to allow the person to identify their own goal.” [LE1] “Making sure people know upfront . . . that the choice is their choice”—referring to harm reduction vs abstinence.
2. Transparency	Ensuring individuals understand what information is being asked, the purpose, and who has access to that information (eg, a data privacy statement that is brief and in lay language). Role of Clinicians, Industry Partners, Peer Support: Outline clearly the roles and responsibilities of each person involved.	[CP2] “So there can be an explanation about why we might ask somebody to track and benefits of doing so, but giving the option not to so they feel comfortable.” Throughout conversations with lived and clinical experts, the need to clearly outline the roles of all involved was emphasized.
3. Stigma and judgement free	Ensuring no direct or indirect pressure on the app to engage in specific treatment approaches (ie, abstinence, alcoholics/narcotics anonymous). Avoiding implying/suggesting blame. Avoiding “toxic positivity” (ie, “of course you can do it, just try”) and using validation.	[LE4] “There’s a lot of stigma that revolves around addiction. So, with the people, it’s a very tight-knit group because we don’t judge each other, you know, we don’t, we’re not here to worry about what, what drug you did or didn’t do? I’m here to make sure that you’re happy and you have a smile on your face, you know.”
4. Simplicity	Ensuring the app is easily navigated and intuitive, avoiding jargon/overly clinical language, and providing accessibility settings.	[LE1] “the simpler, the better, the app should be like, not too hard to navigate.” [LE5] “. . .making sure it’s accessible and it’s not ugly, but not, clinical looking.”
5. OUD is multifaceted and requires a multifaceted approach	Participants highlighted that OUD is about more than overdose prevention. They underscored the importance of paying attention to comorbid mental health concerns like mood, sleep, exercise, emotion regulation, etc, as well as more holistic views of needs and recovery, including housing, clothing, food, healthy recreation, and social connection. Participants additionally consistently highlighted comorbidity and polysubstance use.	[LE5] “I don’t think there’s any other apps besides like overdose prevention ones where peers are on there, like, so I think that will be cool.” [LE4] “list of resources and there was everything it was like, the top things like food clothing, ID, shelter housing.”
6. Inviting appearance	Make design choices that support the message above. Consider spacing, accessibility, and colors, in nonclinical designs that are also not condescending or stigmatizing.	[CP7] “Design a calming interface. . .think about the colour scheme and type of reminders. . .don’t want the app screaming at people.”

Abbreviation: OUD, opioid use disorder; LLE, lived/ing experts; CP, clinical providers.

Themes of feasibility and engagement explored (1) barriers and (2) facilitators of engagement for both lived/living and clinical experts, as well as (3) practical considerations such as location of resources, updating information, and safety, related to potential impacts on the uptake of the app. Common barriers to app engagement for lived/living experts included concerns about intermittent access to devices and Wi-Fi, complicated design, costs associated with use, privacy concerns, and stigma related to disclosures of substance use. Facilitators of engagement for lived experts included reward systems and peer support involvement as digital navigators. Regarding anonymity, participants felt strongly about protecting user identities to create a safe space; however, they simultaneously feared that the anonymity would hinder a sense of community and social connectedness. Participants ideated that other ways to create a sense of community could be to enable the users to “follow” each other’s accounts and comment on one another’s posts so that, despite the anonymous format, they could feel like they were interacting with known individuals. There was no clear consensus in this area.

The most common barrier that clinical experts reported was concerns about monitoring safety on the app and whether missed information would damage therapeutic rapport or lead to clients not seeking appropriate services in crisis situations. Facilitators for clinical experts included brief/graphical descriptions in the clinical portal to facilitate quick understanding of reported information, being able to export those reports to PDF to share with the circle of care, and making sure the functionality of the apps is appropriate to the type of clinical setting/workflow. Clinicians stressed to be an appropriate fit the app needed to reflect contextual factors such as the clinic services (ie, prescription vs case management), stage of change (ie, contemplation vs. recovery), and severity of symptoms. Clinicians saw the severity of symptoms as both a psychological barrier (ie, symptoms themselves) and a practical barrier (ie, those with more severe symptoms may also be street-involved, leading to inconsistent access to devices and Wi-Fi).

Also, considering the range of potential comorbid experiences associated with OUD, participants advocated for moving beyond typical crisis support for suicidal ideation and overdose to include education on additional risks of intravenous drug use and wound care, drug testing options, how to find and use naloxone kits, as well as clearly communicating limitations of the app in crises (ie, this is not a communications-based app). For further details, see Table 2.

Table 2.

Themes of Feasibility and Engagement.

Themes/subthemes	Definition	Exemplar quote(s)
Barriers to engagement	Barriers for individuals with OUD: • Intermittent access to Wi-Fi/cell service, device turnover, etc. • If there were costs associated with the app, especially hidden costs. • Overly complicated design or not enough options for personalized resources. • Privacy and stigma related to disclosure.	[LE5] “A lot of people [with OUD] don’t have phones or like if they do have phones, they might not have the internet or data.” [CP6] “I think the biggest barrier would probably be like the privacy and like . . . if they use drugs, they might not want to disclose that.” [CP8] “Personally . . . I don’t love anonymous threads . . . I don’t really know how much of a sense of community anonymous threads create”
Barriers to engagement	Barriers for clinicians: • Too much responsibility on the app (eg, expected to be involved in the app outside of client appointments in terms of communication/digital navigation). • Expectations for clinicians to use the app/portal regularly, fears of damaged rapport if something is missed.	[CP8] “I don’t know that there is the capacity of staffing right now that people would regularly be checking that, and then I also think it could potentially create like liability and like, safety concerns that people are in there, like really not doing good and no one’s seeing that.” [CP6] “it would also feel really [bad] if you were a client. and tracking all these hard things and . . . there was [not] any kind of response.”
Engagement facilitators	Facilitators for Individuals with OUD: • Reward systems (eg, encouragement, congratulatory badges, etc) • Peer support: involvement of individuals with lived expertise fills key knowledge gaps and may help create a validating/normalizing digital space.	[LE4] “. . . then if you do participate at the end, you get points and then you can use it towards your purchases. So, it’s an incentive to be healthier, you know.” [CP7] “Yeah, absolutely, I think it could be a great idea if the peers providing the support are embedded and trained, that would be awesome.”
Engagement facilitators	Facilitators for clinicians: • Having graphical/brief descriptions of client data available may facilitate discussion at appointments and help boost the efficiency of discussions about symptoms to focus more on skills/therapy. • Having exportable versions of client summaries to append to their chart. • Thinking about the appropriateness of the app for different settings	[CP1] “I think if there’s a way for it to be able to provide us with like a quick summary or report that would be helpful.” [CP5] “The only issue on our end is most of these patients who want to come in and say, “Hi, I’m doing good. I don’t want to change anything get their prescription and walk back out the door, so I don’t even think a good majority of them would even care to discuss like their app use or anything with the clinician.”
Practical considerations	(a) Location: resources are often very different depending on the local/provincial healthcare system.	[CP1] “I don’t know if this is all possible, but local resources, like community, you know, if, if somebody is struggling where could they walk into or where they go.”
	(b) Updating information: regularly updating the resources and tools available to ensure evidence-based information and that external/linked resources are still available.	[CP7] “I also find that a lot of the information becomes outdated really quickly and my patients needs may be very different. So, I’ll direct them to specific resources myself.”
	(c) Safety: ensuring that the app has a safety plan and provides adequate safety information beyond suicidal ideation and crisis resources to include topics of naloxone kits, overdose, drug testing options, safe consumption sites, wound care, and risk of medical conditions like infections (eg, HIV, hepatitis C) and transparency regarding limitations of the app with respect to safety.	[CP5] “Even if there was like a crisis tab or something so that it was only like it was all sort of in one spot when someone felt like they were in crisis, or it could even be like prompted to someone based on any mood charting.” [CP7] “. . . we use the term clinically safety plan, it’s very much like if you are suicidal, if you are overdosing, if you are blank. . .”

Abbreviation: OUD, opioid use disorder; LLE, lived/ing experts; CP, clinical providers.

Participants provided feedback on several specific app features, including (1) psychoeducation, (2) peer support, (3) mental wellness tracking, (4) journaling, (5) community feed/message board, (6) reminders, (7) evidence-based skills and strategies, and (8) clinician portal. Participants highlighted that they felt there was a high level of misinformation being circulated regarding OUD and OUD treatment and, thus, felt it was important to include an educational component to address topics like pathways to care (ie, harm reduction vs abstinence models), opioid-agonist treatment, safe consumption, and so on. Related to the crosscutting themes of creating a safe digital space, participants thought it essential to include peer support as potential “digital navigators” to support users in using the apps and finding the right resources.

Participants also highlighted using tools to facilitate self-reflection and self-discovery, such as wellness tracking and journaling. Participants identified a range of wellness indicators to better inform their understanding of the connection between various aspects of their mental health and substance use. They emphasized the use of journaling prompts to highlight this connection further. A significant point of interest for psychoeducation, skills, and self-reflection tools included information on other areas of wellness (eg, mood, sleep, recreation, social support, physical health, etc). For further details on specific features considered, see Table 3.

Table 3.

Themes Outlining Specific Features.

Themes/subthemes	Definition	Exemplar quote(s)
1. Psychoeducation	Topics: (a) Harm reduction vs abstinence: myth-busting and how to access care. (b) Client rights when it comes to accessing care. (c) Safe substance use (eg, information on overdosing, infections, drug testing, safe environments, etc) (d) How urge-triggers, mood, sleep, and other mental wellness factors interact with substance use and what skills to use. (e) Promoting healthy recreation and social connection.	[CP1] “. . .wound care is a big one. So, looking after the injection sites and things like that, um, and how to, yeah, how to best take care of that, or rotating veins around your body. That type of thing and—educational pieces like, I guess hepatitis as well. So, and different communicable diseases that would be probably beneficial to have some education around.” [LE2] “How your mood connects to like [substance use] through a flowchart or worksheet.”
2. Peer Support	Role: As a community resource connection or digital resource navigator to help individuals connect to the support they are looking for in a validating and normalizing way. Both lived experts and clinicians asserted that the role must be clear as “non-clinical” supports. There was ongoing debate in both groups about synchronous vs asynchronous communication.	[LE4] “If there’s a way for somebody to be able to be someone’s, like, oh not necessarily assistant, but like to navigate people through, not to stay with the permanently, but to show them the ropes initially, and then that way they can continue on.” [LE2] “So maybe if there’s staff available, like, you know, for emergency things that you guys can take over, right away because I mean as a peer support or sort of role certain people might not want to take on those kinds of like heavier things.”
3. Mental Wellness Tracking	Topics thought helpful to track over time: (a) Urges to use, context/triggers, skills use, & outcomes/ consequences (b) Mood (c) Sleep (d) Exercise (e) Positive recreation (f) Social connection (g) Subjective goal progression Function: participants saw this as a means to see trends over time that may help both individuals (ie, self-reflection) and their clinicians better understand factors related to use and recovery.	[LE5] “I just know that lots of people find it helpful to track their substance use and then to also kind of get feedback based on that, that uses CBT models to help cope with cravings and urges to use.” [CP2] “If the tracker can focus not just on negative things. . .” [CP3] “. . .clients usually find really helpful is it [to be] kind of be graphed and they can sort of see progress.” [LE2] “if you can see like a graph at the end of the week or whatever, so you can work on improving or you can see improvement.”
4. Journaling	Function: There were 3 functions of journaling suggested by participants: 1. Free Journaling: allow individuals to write their reflections down about whatever they find helpful. 2. Guided Journaling: allow individuals to work through structured worksheets about therapeutic topics (eg, thought records, urge tracking, pros and cons, etc). 3. Prompted Public Posting: providing a public prompt on the community feed or message board that allows individuals to share with others (eg, what is something you’re proud of this week?). 4. JoyKit: a place for app users to collate pictures, songs, quotes, and notes to the self as reminders to visit when they feel like using substances.	[LE2] “[Add a] journal entry sort of reflection piece so that they can keep for their personal viewing or whatever. So like, if a goal didn’t get met, maybe a question would be like, what do you feel like needs to happen in order for you to achieve or reach that goal.” [LE3] “. . .incorporate a gratitude journal . . . [with] options for input, like what are you thankful for today? And then list three things.” [LE2] . . . “choose [which entries] to share on the feed”
Community feed/message board	Function: To create a space where individuals can share about their substance and mental health recovery. It is meant to be a positive, encouraging space where members can post about topics that they want support or resources about, or things that are going well for them, resources they’ve used and liked, sharing milestones, etc. Many participants requested the ability to interact with others’ posts such as liking a post, commenting on a post, or being able to follow someone’s story through an anonymous profile (ie, just see a collection of their posts). This was an important aspect to many, as they described that OUD recovery can be isolating and many don’t understand, so a sense of connection and community is important.	[LE5] “. . . a community page or like a way that people can talk about, like, what’s working for them.” [CP3] “I do feel like people really appreciate hearing from others and their experiences, but also kind of just sharing all their different coping strategies.” [LE2] “So what that makes me think of is like, open sort of a forum component to the feed. That would be kind of like a forum base and post and comment or whatnot. With some level of moderation to it, but to be able to interact.”
Reminders	Types of Reminders: (a) Appointments (b) Medication (daily) and prescription refills (c) Goals (eg, did you go for a walk today?) (d) Events and community resources (ie, calendar function, there’s a food drive on this day, etc) (e) Reminders should be controlled by the individual in terms of frequency, topic, time of day, and have alerts/pop-ups.	[LE4] “Definitely! My whole life revolves around reminders.” [LE2] “You could have a feature on it too about like when you hope to achieve this goal by, so it kind of gives you a bit of a timeline and accountability and then maybe a reminder halfway through.” [LE1] “a daily list of chores and reminders.”
Skills and strategies	Facilitate access to evidence-based skills and strategies for managing cravings/urges to use, emotion regulation, distress tolerance, anxiety, depression, self-efficacy/mastery, positive relationships, SMART goal setting, and other comorbid mental health symptoms.	[CP2] “The DBT skills, I think are very, useful, we use them across the board . . . especially with this population trauma . . . and a lot of depression.” [CP1] “using CBT for challenging unhelpful thought patterns, recognizing cognitive distortions. Well, yeah, just definitely within the DBT skills, lots of distress tolerance skills when it comes to urge surfing and trying to ride that wave.”
Clinician portal	Functions: • Tracking use to inform treatment decisions. • Tracking moods, sleep, and other factors that might be impacting use to find helpful strategies in session. • Consider the circle of care and who might need access. • Consider personalizing the information that is getting tracked and giving users the option to decide which information is shared.	[CP7] “You know. Cause it can also help us make decisions on like medication, right? Like, if we track it for like three months and we’re like, “Oh, you’re using three days before your medication” . . . maybe we could move it back by three days and see if that would be helpful for the cravings, so you don’t end up using.”

Abbreviation: OUD, opioid use disorder; LLE, lived/ing experts; CP, clinical providers; SMART, Specific Measurable Attainable Relevant Time-bound.

Design Implementation

Not every feature suggested during the focus groups was possible to implement in the first iteration of the app. Prioritized features were determined collaboratively with our industry partner and informed by the focus groups. Given the rapid pace of technology development and evaluation, features that were deemed necessary by participants and leveraged the existing app architecture were prioritized. The urge-tracking feature utilized the foundations of the existing structure and was highlighted by lived/living experts and clinicians as having a significant potential impact. Participants also advocated for a feature that could remind app users of their motivations for recovery, given the fluctuating nature of OUD and relapse. To this end, the development team leveraged current features to create the JoyKit, wherein app users can visit personally curated pages of pictures, quotes, and music that remind them why their recovery is important to them. The social feed, which also delivers personalized information and resources, was easily adapted to provide information on OUD instead of resources related to schizophrenia. Furthermore, broader mental health resources on topics that are common to both OUD and schizophrenia, like recreation, social network building, anxiety, and depression, are recycled in this version of the app. Goal setting, reminders, and tracking are also existing features that were updated with information relevant to OUD. The remaining features outlined in Table 3 have planned implementation throughout the beta testing phases.

It was also crucial that patient privacy and safety were prioritized on the app. Safety was prioritized by building an app training program whereby the limitations of the app as a crisis response tool are clearly outlined. These limitations are clearly stated in the “urgent resources” tab of the app (ie, this app should not be used as a primary means of urgent care and cannot be used to contact assistance directly). The urgent resources tab includes information on emergency department locations, hotlines, recognizing an overdose, and instructions on locating and using naloxone. In addition, participants are informed that the app is connected to a clinician portal that, with their permission, allows their clinician to see the information they have entered into the app, such as notes on topics they would like to in the app discuss at their next clinical appointment. It is highlighted that their clinician does not receive these notes in real-time and may not be monitored regularly. Last, before posts go “live” on the feed, they are moderated by research staff, allowing them to reach out to users if there are concerns regarding risk, privacy/identifying information, or potentially triggering content.

Discussion

This study provided the results of the initial co-design process examining the extension of an existing app for individuals with schizophrenia for use by individuals with OUD. There is a dearth of technology available that supports recovery with OUD-specific resources and supports. Results highlight several critical areas of consideration when designing digital tools to support OUD recovery from overarching principles to specific functions and features. Perhaps most relevant and worth consideration broadly by researchers are the crosscutting themes and the context behind them.

Noteworthy was that each lived/living expert shared their experience with social and provider stigma during their recovery journey and advocated for a judgement and stigma-free digital space. Both provider and social in-group stigma have been identified as key factors related to service disengagement.³⁶ Relatedly, lived experts and clinicians alike reported that creating a digital environment that facilitates social connectedness and acceptance is an essential factor for fortifying engagement with the digital tool and is a strategy that has been identified across several areas of digital mental health.^37,38 However, results from this study emphasize the tension between establishing a sense of social connection and protecting users’ health information, with no clear direction regarding anonymity in digital spaces. Regardless of preference, many digital tools used for healthcare purposes fall under the purview of healthcare privacy laws, which may supersede user preference.

A specific feature highlighted in our results related to social connectedness is the inclusion of peer support as digital navigators. This commentary aligns with an increasing emphasis on the need for digital navigators integrated into care teams to support the effective implementation of digital health tools.³⁹ Peer support has been hypothesized as one specific strategy to address disengagement through shared lived experiences to facilitate a sense of belonging and hope.⁴⁰ Our participants additionally highlighted the importance of lived/living expertise in helping individuals navigate digital and community resources. “Who better to guide you on a path than someone who has walked it before?” (-PT001). There was an ongoing debate among participants regarding the need for synchronous communication with peer support. Asynchronous digital peer support tools have been used with some preliminary success but require more rigorous analysis.⁴¹ In our discussion on this topic, some (n = 2) participants felt strongly that the role of peer support should be asynchronous communication for 2 reasons: (a) it would allow the peer supporter time to consult and gather evidence-based resources when necessary and (b) it would underscore users’ understanding that the peer supporters would not be constantly available and that the app does not replace other crisis resources like call/text lines.

Also thought to impact user engagement is the perceived fit of the intervention to personal needs, including how well the app can be integrated into day-to-day activity and the severity of depressive symptoms.³⁸ This resonates with the feedback from participants regarding the need for a personalized digital space and resources. Participants described OUD as heterogeneous. The challenge with personalized digital strategies is the breadth of the resources required to adequately address potential areas of concern while curbing the associated complexity. Simplicity and ease of use are routinely described as paramount in digital health design.⁴² Yet, it would seem apparent that clinical and lived/living experts saw integrating multiple functions as central to engagement and utility.

Furthermore, integration into the clinical context was essential for clinicians in our study to implement technological tools successfully. Lack of clinical integration is associated with low uptake and is identified as a barrier to implementation.⁴³ Considerations in this regard include understanding the clinical culture, the setting infrastructure, and how technology may affect staff workflow.⁴³ Thus, it is important to consider design elements of digital interventions well suited to the existing technologies and care structures (eg, case management vs prescription management vs residential care models, etc.). Other significant concerns clinicians in our study highlighted as barriers to implementation were questions regarding crisis resources, synchronous communication, and medical-legal responsibility. Clinicians reported concern regarding synchronous communication and workflow, such that they worried they would miss an urgent message from a client and either damage rapport or result in client jeopardy. As such, the unanimous preference among clinicians was asynchronous communication, where patient-users could leave short notes about instances they want to remember to discuss at the next appointment, with a clear disclaimer present that the portal is not monitored as most reported they would only open the dashboard live in session. Surprisingly, this was the only concern regarding workflow that was coded to saturation. Some clinicians opined that the clinician dashboard may even improve appointment efficiency. However, again, that code did not reach saturation.

Future Directions and Limitations

The results of this study should be considered in light of multiple limitations. First, we did not formally evaluate the co-design process or collect formal feedback from participants regarding how well we collected and integrated the insights provided. Co-design best practices recommend a formal review to gauge how well the resulting technology reflects the feedback provided from the participant’s perspective. In addition, our co-design project is limited to only 2 types of knowledge users, whereas best practices suggest engaging multiple types of knowledge users, including policy-makers and management, where appropriate. Given the need for seamless clinical integration, including institutional decision-makers and policymakers in the design of the A4i-O app would be helpful and is planned for later stages of design prior to implementation trials. Future directions for this and other design projects should include a more diverse sample of knowledge users in design and a stepped or iterative design framework. Iterative design frameworks would allow for ongoing co-design processes through advisory boards comprised of lived/living experts, clinicians, social supports, institutional decision-makers, policymakers, and various industry partners.

Second, the lack of diverse identities represented in this study may limit the transferability of results. Our sample lacks gender, racial, and ethnic diversity among providers and lived experts. This representation challenge occurred despite recruitment through a number of sources for both lived/living experts (ie, a large tertiary care hospital, community substance use clinics, Rapid Access to Addiction Medicine (RAAM) clinics, lived/living expert advisory networks) and providers (ie, professional networks, community organizations, and the aforementioned sources for lived/living experts). Staying true to the voluntary nature of co-design and wanting to avoid feelings of coercion or pressure to participate, we allowed participants to self-refer to the study rather than use purposive sampling. With regard to gender, we had more men initially interested in the study; however, they were lost to follow-up despite attempts to contact them. Despite this limitation, we believe that the general themes presented here are likely present and important across identities of individuals experiencing OUD-related concerns (eg, themes highlighting the importance of stigma-free spaces, social connection, and judgement-free evidence-based content) and still highlight important foundational design principles for design within this population. A considerable degree of diversity among participants with regard to professional roles among providers and diversity across recovery and life circumstances among lived/living experts may also offset representation concerns to some degree. Last, consideration is warranted that women are markedly underrepresented in clinical research on substance use treatments,⁴⁴ despite evidence that opioid use may be more common in adult⁴⁵ and adolescent⁴⁶ women (even though mortality rates are higher in men). This represents a critical gap in the literature, which, as other qualitative studies become aggregated, may help support overall representation within the field. In addition, we have not come across any evidence in the literature that suggests gender differences in how clinicians provide substance use treatment. Future iterations of this project and other technologies, however, should examine intersectional perspectives regarding barriers and facilitators to engagement, stigma, specific content, and language/terminology as important nuances may be missed in this sample. For instance, though no gender differences in overall willingness to use technology for substance use recovery have been found,⁴⁷ some research suggests that gender-specific resources may be helpful⁴⁸ and is likely the same for culturally adapted technological resources. An area of special consideration regarding representation is the investigation of barriers and facilitators of engagement and access, in addition to identity-specific resources. Future studies should consider purposive sampling methods to ensure a more representative group. Our study faced several recruitment challenges relying on self-referral sampling methods, including a prolonged recruitment phase and difficulties identifying genuine participants, all of which resulted in a less diverse sample.

Conclusion

Individuals with lived/living experience of OUD and care providers highlighted several promising avenues regarding the functionality of digital tools in this area, how they might be effectively implemented (eg, peer support digital health navigators), and essential considerations affecting uptake and engagement. Opioid addictions represent a public health crisis with major barriers related to care access, a gap that digital interventions may partially bridge, with co-design being a critical part of the development process. Future work will explore how we move from focus-group-derived input to a feedback cycle involving technology testing and design iteration.

Footnotes

Acknowledgements

The authors would like to thank the lived/living and clinical experts who participated in this project. This work would not be possible without your contributions.

Author Contributions

The first author, Jessica D’Arcey, is a doctoral research trainee under the supervision of the last author, Dr Sean A. Kidd. Jessica led the data collection, analysis, interpretation, and manuscript authorship and preparation. Leah Tackaberry-Giddens is a research analyst who supported the data collection and significantly contributed to the manuscript authorship and preparation. Drs Matthew Sloan and Lena Quilty are co-investigators on this project and contributed to the design of the project, interpretation of results, and to manuscript authorship and preparation. Wenjia Zhou is the A4i project design lead at Memotext, our industry partner, and Sana Junaid is a design support analyst. Both Wenjia and Sana contributed to the data collection and industry-led sections of the manuscript. As the principal investigator, Dr Sean A. Kidd devised the project, led its design, oversaw the data collection and analysis, and significantly contributed to the interpretation of results and manuscript preparation.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The technology behind the core architecture of the platform, A4i Schizophrenia, was developed and owned by the joint for-profit venture A4i. The company Memotext owns 50% of this company, the Centre for Addiction and Mental Health owns 35% of this company, and Dr Sean A. Kidd owns 15% of this company. COI has been reviewed by legal professionals, and a mitigation plan has been implemented.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Centre for Addiction and Mental Health, Discovery Fund. The first author is also supported by a Vanier Scholarship through the Canadian Institute for Health Research (CIHR).

Compliance,Ethical Standards,and Ethical Approval

This research was reviewed and approved by the Research Ethics Board at the Centre for Addiction and Mental Health (REB# 42/2022). This project was conducted in compliance with ethical standards.

ORCID iD

Jessica N. D’Arcey

References

Keyes

Rutherford

Hamilton

, et al. What is the prevalence of and trend in opioid use disorder in the United States from 2010 to 2019? Using multiplier approaches to estimate prevalence for an unknown population size. Drug Alcohol Depend Rep. 2022;3:100052. doi:10.1016/j.dadr.2022.100052

Shoff

Yang

Shaw

BA.

Trends in opioid use disorder among older adults: analyzing Medicare data, 2013–2018. Am J Prev Med. 2021;60(6):850-855. doi:10.1016/j.amepre.2021.01.010

Friebel

Maynou

Trends and characteristics of hospitalisations from the harmful use of opioids in England between 2008 and 2018: population-based retrospective cohort study. J R Soc Med. 2022;115(5):173-185. doi:10.1177/01410768221077360

Centre for Disease Prevention and Control. Drug overdose deaths. Published Decmber, 2022. https://www.cdc.gov/nchs/nvss/drug-overdose-deaths.htm

Government of Canada. Health Infobase: Opioid- and stimulant-related harms in Canada. Published September, 2023. https://health-infobase.canada.ca/substance-related-harms/opioids-stimulants/

Santo

Campbell

Gisev

, et al. Prevalence of mental disorders among people with opioid use disorder: a systematic review and meta-analysis. Drug Alcohol Depend. 2022;238:109551. doi:10.1016/j.drugalcdep.2022.109551

Novak

Feder

Ali

Chen

Behavioral health treatment utilization among individuals with co-occurring opioid use disorder and mental illness: evidence from a national survey. J Subst Abuse Treat. 2019;98:47-52. doi:10.1016/j.jsat.2018.12.006

Mahoney

Winstanley

Lander

, et al. High prevalence of co-occurring substance use in individuals with opioid use disorder. Addict Behav. 2021;114:106752. doi:10.1016/j.addbeh.2020.106752

Gomes

Kitchen

Tailor

, et al. Trends in hospitalizations for serious infections among people with opioid use disorder in Ontario, Canada. J Addict Med. 2022;16(4):433-439. doi:10.1097/ADM.0000000000000928

10.

McLaughlin

Carrero

Bain

Chatterjee

Opioid use disorder treatment for people experiencing homelessness: a scoping review. Drug Alcohol Depend. 2021;224:108717. doi:10.1016/j.drugalcdep.2021.108717

11.

Livingston

JD.

A Framework for Assessing Structural Stigma in Health-Care Contexts for People With Mental Health and Substance Use Issues. Mental Health Commission of Canada; 2020.

12.

Ali

Dubenitz

Suicidal behavior, opioid use disorder, and behavioral health treatment: Prevalence and correlates among adults in the United States 2015–2018. J Subst Abuse Treat. 2021;130:108413. doi:10.1016/j.jsat.2021.108413

13.

Rhee

Rosenheck

RA.

Use of drug treatment services among adults with opioid use disorder: rates, patterns, and correlates. Psychiatr Serv. 2019;70(11):992-999. doi:10.1176/appi.ps.201900163

14.

Masson

Chen

Levine

Shopshire

Sorensen

JL.

Health-related internet use among opioid treatment patients. Addict Behav Rep. 2019;9:100157. doi:10.1016/j.abrep.2018.100157

15.

Rhoades

Wenzel

Rice

Winetrobe

Henwood

No digital divide? Technology use among homeless adults. J Soc Distress Homeless. 2017;26(1):73-77. doi:10.1080/10530789.2017.1305140

16.

Nuamah

Mehta

Sasangohar

Technologies for opioid use disorder management: mobile app search and scoping review. JMIR Mhealth Uhealth. 2020;8(6):e15752. doi:10.2196/15752

17.

Miller-Rosales

Morden

Brunette

Busch

Torous

Meara

ER.

Provision of digital health technologies for opioid use disorder treatment by US health care organizations. JAMA Netw Open. 2023;6(7):e2323741. doi:10.1001/jamanetworkopen.2023.23741

18.

Kiburi

Ngarachu

Tomita

Paruk

Chiliza

Digital interventions for opioid use disorder treatment: a systematic review of randomized controlled trials. J Subst Abuse Treat. 2023;144:108926. doi:10.1016/j.jsat.2022.108926

19.

Kawasaki

Mills-Huffnagle

Aydinoglo

Maxin

Nunes

Patient- and provider-reported experiences of a mobile novel digital therapeutic in people with opioid use disorder (reSET-O): feasibility and acceptability study. JMIR Form Res. 2022;6(3):e33073. doi:10.2196/33073

20.

van Gemert-Pijnen

Kelders

Kip

Sanderman

EHealth Research, Theory and Development. Routledge; 2018. doi:10.4324/9781315385907

21.

Hardy

Wojdecka

West

, et al. How inclusive, user-centered design can improve psychological therapies for psychosis: development of SlowMo. JMIR Ment Health. 2018;5:e11222.

22.

Orlowski

Matthews

Bidargaddi

, et al. Mental health technologies: designing with consumers. JMIR Hum Factors. 2016;3(1):e4. doi:10.2196/humanfactors.4336

23.

Cheetham

Picco

Barnett

Lubman

Nielsen

The impact of stigma on people with opioid use disorder, opioid treatment, and policy. Subst Abuse Rehabil. 2022;13:1-12. doi:10.2147/SAR.S304566

24.

Janulis

Ferrari

Fowler

Understanding public stigma toward substance dependence. J Appl Soc Psychol. 2013;43(5):1065-1072. doi:10.1111/jasp.12070

25.

Paquette

Syvertsen

Pollini

RA.

Stigma at every turn: health services experiences among people who inject drugs. Int J Drug Policy. 2018;57:104-110. doi:10.1016/j.drugpo.2018.04.004

26.

Schouten

Kip

Dekkers

, et al. Best-practices for co-design processes involving people with severe mental illness for eMental health interventions: a qualitative multi-method approach. Des Health. 2022;6(3):316-344. doi:10.1080/24735132.2022.2145814

27.

Kidd

Feldcamp

Adler

, et al. Feasibility and outcomes of a multi-function mobile health approach for the schizophrenia spectrum: APP4Independence (A4I). PLoS One. 2019;14(7):e0219491. doi:10.1371/journal.pone.0219491

28.

Kidd

McKenzie

Wang

Agrawal

Voineskos

Examining a digital health approach for advancing schizophrenia illness self-management and provider engagement: protocol for a feasibility trial. JMIR Res Protoc. 2021;10(1):e24736. doi:10.2196/24736

29.

Romney

Weller

Batchelder

. Culture as consensus: A theory of culture and informant accuracy. Am Anthropol. 1986;88(2):313-338.

30.

Galvin

. How many interviews are enough? Do qualitative interviews in building energy consumption research produce reliable knowledge? J Build Eng. 2015;1:2-12. doi:10.1016/j.jobe.2014.12.001

31.

Hennink

Kaiser

Weber

. What influences saturation? Estimating sample sizes in focus group research. Qual Health Res. 2019;29(10):1483-1496. doi:10.1177/1049732318821692

32.

Carlsen

Glenton

. What about N? A methodological study of sample-size reporting in focus group studies. BMC Med Res Methodol. 2011;11(1):26. doi:10.1186/1471-2288-11-26.

33.

Aryana

Brewster

. Design for mobile mental health: exploring the informed participation approach. Health Inform J. 2020;26(2):1208-1224. doi:10.1177/1460458219873540

34.

Vial

Boudhraâ

Dumont

. Human-centered design approaches in digital mental health interventions: exploratory mapping review. JMIR Ment Health. 2022;9(6):e35591. doi:10.2196/35591

35.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-100.

36.

McCradden

Vasileva

Orchanian-Cheff

Buchman

DZ.

Ambiguous identities of drugs and people: a scoping review of opioid-related stigma. Int J Drug Policy. 2019;74:205-215. doi:10.1016/j.drugpo.2019.10.005

37.

Dewa

Lawrance

Roberts

, et al. Quality social connection as an active ingredient in digital interventions for young people with depression and anxiety: systematic scoping review and meta-analysis. J Med Internet Res. 2021;23(12):e26584. doi:10.2196/26584

38.

Borghouts

Eikey

Mark

, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Internet Res. 2021;23(3):e24387. doi:10.2196/24387

39.

Macrynikola

Nguyen

Lane

Yen

Torous

The digital clinic: an innovative mental health care delivery model utilizing hybrid synchronous and asynchronous treatment. NEJM Catal Innov Care Deliv. 2023;4(9). doi:10.1056/CAT.23.0100

40.

Fortuna

Brooks

Umucu

Walker

Chow

PI.

Peer support: a human factor to enhance engagement in digital health behavior change interventions. J Technol Behav Sci. 2019;4(2):152-161. doi:10.1007/s41347-019-00105-x

41.

Fortuna

Venegas

Umucu

Mois

Walker

Brooks

JM.

The future of peer support in digital psychiatry: promise, progress, and opportunities. Curr Treat Options Psychiatry. 2019;6(3):221-231. doi:10.1007/s40501-019-00179-7

42.

Baldwin

Singh

Sittig

Giardina

TD.

Patient portals and health apps: pitfalls, promises, and what one might learn from the other. Healthc (Amst). 2017;5(3):81-85. doi:10.1016/j.hjdsi.2016.08.004

43.

Connolly

Kuhn

Possemato

Torous

Digital clinics and mobile technology implementation for mental health care. Curr Psychiatry Rep. 2021;23(7):38. doi:10.1007/s11920-021-01254-8

44.

Gunn

Pankowska

Harris

Helsing

Battaglia

Bagley

SM.

The representation of females in clinical trials for substance use disorder conducted in the United States (2010–19). Addiction. 2022;117(10):2583-2590. doi:10.1111/add.15842

45.

McHugh

DeVito

Dodd

, et al. Gender differences in a clinical trial for prescription opioid dependence. J Subst Abuse Treat. 2013;45(1):38-43. doi:10.1016/j.jsat.2012.12.007

46.

Bagley

Gai

Earlywine

Schoenberger

Hadland

Barocas

JA.

Incidence and characteristics of nonfatal opioid overdose among youths aged 11 to 24 years by sex. JAMA Netw Open. 2020;3(12):e2030201. doi:10.1001/jamanetworkopen.2020.30201

47.

Gilbert

Saathoff

Russell

Brown

Gender differences in lifetime and current use of online support for recovery from alcohol use disorder. Alcohol Clin Exp Res. 2022;46(6):1073-1083. doi:10.1111/acer.14827

48.

Stinson

Wolfson

Poole

Technology-based substance use interventions: opportunities for gender-transformative health promotion. Int J Environ Res Public Health. 2020;17(3):992. doi:10.3390/ijerph17030992