Organizational factors impacting the implementation of a digital mental health tool in Alberta's mental health care of youth and young adults

Abstract

With mental health concerns on the rise among youth and young adults (age 12–24), increased mental health options include virtual care, apps and online tools, self-management and tracking tools, and digitally-enabled coordination of care. These tools may function as alternatives or adjuncts to face-to-face models of care. Innovative solutions in the form of digital mental health (dMH) services not only provide support, resources and care, but also decrease wait times and waitlists, increase access, and empower youth. However, organizational factors may impact the extent of dMH interventions are that accepted, used, and sustained in clinical settings. This qualitative study explores organizational barriers and facilitators surrounding the implementation of a digital platform (Innowell), which uses measurement-based care (MBC) to track youth progress and outcomes. Data was collected from 154 mental health care providers participating in 23 focus groups across Alberta, drawing on school and community settings, specialized mental health services, and primary care networks. A thematic analysis revealed the following: barriers included incompatibility with current systems and workflows, lack of inter-organizational collaboration, time commitment, perceived sustainability and lack of digital literacy. Facilitators included positive attitudes towards using dMH to optimize clinical practices by empowering youth and improving continuity of care, transitions in care, and quality of care, as well as workplace culture and leadership. The study highlights a critical need for decision makers and clinical leaders to address organizational factors by integrating training and support, establishing interoperability between digitized and in-person healthcare systems, and leveraging support for MBC and youth-centred care.

Keywords

Digital mental health (dMH)measurement-based care (MBC)mental health care organizational barriers youth and young adults

Introduction

Rates of poor mental health among youth and young adults (aged 12–24 years) have been a growing concern among mental health care providers and researchers in recent years. The prevalence of mental health issues such as anxiety, depression, and mood disorders has increased among young people.^1,2 The COVID-19 pandemic further exacerbated these issues, with social isolation, disrupted routines, and uncertainty causing increased stress and anxiety among youth^3–5; symptoms of anxiety increased from 11.6% to 20.5% and depressive symptoms increased from 12.9% to 25.2%.⁵ Additionally, a high prevalence of sleep disorders (44%) and post-traumatic stress symptoms (48%)⁶ were reported. Youth are at an increased risk for developing mental health issues due to developmental, neurological, and hormonal changes that characterize this life stage.^7–9 These changes occur in the context of academic pressures, social media influence, economic instability, and stigma surrounding mental health. These factors compound stress and further contribute to the onset of mental health concerns,¹⁰ underscoring a need to provide mental health support, resources, and services to bolster youth well-being.

Traditional models of mental health services offered exclusively in-person are associated with system-level challenges such as long waiting lists¹¹ and shortages of mental health professionals.¹² Geographic barriers (i.e. living in remote or rural areas) may also prevent youth from accessing timely and appropriate support.^13,14 Digital mental health (dMH) solutions have emerged as a way to deliver innovative mental health services that address the gaps in traditional service delivery.^15–17 dMH entails offering mental health services and resources through digital modes, such as mobile apps, online therapy platforms, websites, and telehealth services.^16,18 Integrating dMH into existing health systems could be a cost-effective way to address the shortage of mental health professionals,¹⁹ enhance continuity of care,²⁰ expand access to mental health services,²¹ increase communication and trust between clients and health professionals,^22,23 reduce stigma, and empower youth.^17,24 Furthermore, embedding measurement-based care (MBC) into dMH enables ongoing evaluation of symptoms and progress tracking. This can inform treatment decisions and give autonomy to clients as they engage with their mental health care,²⁵ as well as enable providers to identify potential issues earlier and offer timely intervention.^26,27

While MBC and dMH have the potential to enhance the quality of mental health care, organizational barriers can hinder its widespread implementation.^28–30 For instance, the regular psychometric assessments required in MBC³¹ are time-consuming.^28,30 Some practitioners struggle to change established clinical practices and workflows required to implement dMH and MBC without adequate leadership support.^29,32 Logistically, integrating MBC and dMH into existing electronic health record systems can be challenging if there is a lack of interoperability.^29,33 Understaffing and a lack of training for clinical staff also pose barriers.^28,29 Privacy and data security present another obstacle, as concerns about confidentiality and protection of personal information can deter clients from engaging with dMH platforms.^24,34 Regulatory and reimbursement challenges may further hinder the integration of dMH and MBC into healthcare systems, making it difficult for healthcare providers to offer these services.²⁹

Conversely, organizational facilitators can support the implementation of dMH innovations. The literature identifies several organizational facilitators including a supportive organizational culture, the suitability of innovations for program fit, simplicity of innovations, and adaptability within mental health care settings.³⁵ Broader contextual factors including government commitment and the political context can also impact a dMH implementation, as well as stakeholder attitudes, stable and enthusiastic leadership, and the prevalence of inter-agency partnerships.³⁶ These factors impact uptake, sustainability, and success of dMH implementations.

Studies exploring mental health care providers’ perspectives on organizational barriers and facilitators that influence the acceptability, uptake, and sustainability of using dMH with youth and young adults in the Canadian context are lacking. The Canadian context presents unique challenges for mental health services, both in meeting the needs of diverse constituents (i.e. immigrant and Indigenous populations) and geographical challenges (i.e. mental health services are often located in large urban centers, far from many large rural and remote communities). Additionally, studies exploring the perspectives of mental health care providers across multiple service settings are limited. Therefore, this study aimed to explore providers’ perspectives regarding organizational factors impacting their use of dMH and MBC among Albertan youth across mental health service settings including specialized mental health services (SMHS), primary care networks (PCN), and school and community-based providers. SMHS are cross-provincial settings that support mental health promotion, intervention, treatment, and recovery. PCNs are clinical networks that use team-based care models within geographic areas. This study used the Theoretical Domains Framework to explore contextual organizational factors impacting a dMH implementation. Providers’ perspectives inform recommendations for alleviating organizational barriers and enabling facilitators to support greater use of dMH and MBC in mental health care settings.

Methodology

This study is part of a larger multi-method research project exploring the implementation of dMH and MBC for youth and young adults in Alberta, Canada. A qualitative descriptive methodology was used to explore pre-implementation perspectives on the barriers and facilitators to implementing dMH. This methodology aims to extensively describe healthcare experiences or events with a central focus on staying close to the data and reporting results using language similar to that of participants’ responses.^37,38

This study is epistemologically aligned to a naturalistic standpoint because we aimed to present an understanding of a phenomenon—in this case, the implementation of a dMH platform—through the perspectives and meanings ascribed by participants in their real-world contexts.³⁹ We used an inductive process to understand and describe the context and the subjective worldviews and meaning-making processes of the participants involved.⁴⁰ Using an ontology-based in relativism, our data analysis prioritized the literal description of data, then progressed towards explaining providers’ perspectives regarding a dMH implementation.

The dMH platform implemented in this project, Innowell, is a web-based MBC platform that allows youth and providers to assess and monitor youth mental health. The Innowell platform is a dMH tool that was co-designed by health professionals and individuals with mental health lived-experience.^25,27 The platform utilizes a personalized approach that incorporates MBC across 20 mental health and substance use measures, including suicidal thoughts and behavior, anxiety, substance use, and social connectedness.³³ These psychometric tools facilitate the evaluation of the young person's mental health symptoms, assist in forming the treatment plan with the youth, and support with monitoring treatment progress.²⁵ The platform also offers youth access to verified digital mental health resources, applications, tools, and crisis support, all tailored to their community. The research group has previously published work on the implementation of this platform in school settings,⁴¹ on mental health providers’ perceptions of using and engaging dMH with youth,⁴² and on the development of an implementation protocol for using dMH in stratified care.⁴³ This study reports on pre-implementation focus group data; participants received a short orientation on the dMH platform before sharing feedback. Full training sessions occurred later during the implementation.

Ethical statement

This research study received ethical approval from the University of Calgary Conjoint Health Research Ethics Boards (CHREB) (“eMH Qualitative Study—Pre-Implementation and Implementation”—REB20-1137). All participants provided informed consent and demographic information online prior to participating in the focus group. Consent was obtained by the research team via REDCap (Research Electronic Data Capture), hosted at the University of Calgary.^44,45 REDCap is a secure web-based software that facilitates data collection. It offers a user-friendly interface for reliable data collection, audit trails to monitor data handling and export processes, automated export functions for data transfers to statistical software, and tools for integrating data with external sources to enhance compatibility and ease of use. In presenting demographic information, categories with fewer than six responses are suppressed to prevent identification.

Data collection

Focus groups were used to collect data from mental health care providers in schools, community settings, Specialized Mental Health Services (SMHS), and Primary Care Networks (PCNs). Focus groups were chosen because this method allows participants to collectively identify key topics and opinions,⁴⁶ yielding a rich blend of perspectives and viewpoints.⁴⁷ A semi-structured focus group guide was developed by the research team; questions focused on participants’ perspectives regarding the implementation of dMH. The design of the focus group guide (Appendix 1) followed the Theoretical Domains Framework (TDF), which is suitable for exploring factors that impact health-related behavior change.⁴⁸ The TDF includes domains such as knowledge, skills, beliefs about capabilities, environmental context and resources, and social influences.⁴⁸

To recruit participants, we shared recruitment materials with site leads and clinical supervisors in participating sites across Alberta. Site leads helped to identify participants who would be involved in using the platform. Participants were introduced to the Innowell platform and given training on how to use the platform. A total of 23 focus groups were conducted virtually from February 2021 to June 2022 with stakeholders in SMHS (6 focus groups, n = 50), PCN (5 focus groups, n = 25), and school and community settings (11 focus groups, n = 62). Participants included social workers, counsellors, psychologists, nurses, administrators, and educators. Focus groups were 90-min in length, with three members from the research and implementation team facilitating each focus group, taking roles of moderating and note-taking. Focus groups were video and audio recorded, with the audio recording being transcribed verbatim by a professional transcriptionist, and checked for accuracy by a member of the research team. During transcription, focus group data were de-identified for participants’ privacy and confidentiality.

Thematic data analysis

We used a primarily inductive approach to analysis using Braun and Clarke's⁴⁹ six stages of thematic analysis: (1) familiarization with data; (2) coding; (3) theme generation; (4) theme review; (5) naming and defining themes; and (6) writing. The research team met weekly from September 2022 to December 2023 during the analysis stage of the research. Coding was undertaken by youth and young adult research assistants (RAs). The RAs included students, interns, and people with lived experience. They worked independently to familiarize themselves with the data and were each assigned a subset of the transcripts to complete a preliminary round of descriptive coding. Codes were discussed with the qualitative research team including experienced qualitative researchers, and each RA completed a secondary round of coding on a second set of transcripts to ensure agreement, reliability, and consistency. Disagreements were discussed as a group during recurring qualitative research meetings until consensus was achieved. We used a reflexive thematic analysis approach,⁵⁰ precluding any need for inter-rater reliability or agreement statistics. The research team grouped the codes into themes through iterative discussion sessions to identify patterns. Throughout the process, the research team kept records of coding and theming decisions. The research team remained reflexive, considering potential bias and questioning assumptions through memoing and group conversations. Preliminary findings were shared with the broader research team to ensure alignment of the study's processes and findings with existing literature.

Research team positionality

The research team consists of a senior research team, young adult RAs, and an implementation team (who provided implementation support and trained providers on the dMH platform). We carefully considered how team members were involved in leading the focus groups and analyzing data. The senior researchers and implementation team were involved in focus group moderation, with the principal investigator, GD, playing a pivotal role in leading discussions while drawing on a clinical background as a social worker. The research team worked together throughout data analysis to share perspectives, recognize their own positions and personal biases, and maintain an objective approach to interpreting data. For example, the team included diverse young adult researchers with direct experiences working in mental health settings; discussions in recurring meetings enriched the team's understanding of youth experiences in mental health settings.

Results

Participant demographics

Most participants (n = 125; 81.2%) identified as female, and most participants self-reported their ethnicity to be European origins (n = 94; 61%), with 83.8% (n = 129) reporting that they were not members of a racialized group. Participants were working in SMHS (n = 49; 31.8%), PCNs (n = 43; 27.9%), school districts (n = 49, 31.8%), and community settings (n = 13; 8.5%). Most participants were between ages 30–49 (n = 54; 69.5%), and the majority had over 11 years of professional experience (n = 91; 59.1%). A complete list of descriptive information about participants is presented in Table 1.

Table 1.

Descriptive demographic information of participants (n = 154).

Variable		n (%)
Gender
	Woman	125 (81.2)
	Man	25 (16.2)
	Other or Not Reported	*
Type of organization
	Specialized Mental Health Services	49 (31.8)
	Primary Care Network	43 (27.9)
	School District	49 (31.8)
	Community-based or municipality	13 (8.5)
Professional role
	Social Worker	53 (34.4)
	Administrator	11 (7.1)
	Counsellor or Therapist	28 (18.2)
	Psychologist	20 (13.0)
	Nurse	20 (13.0)
	Teacher/Educator	7 (4.6)
	Other or Not Reported	15 (9.7)
Age group
	20–29	16 (10.4)
	30–39	54 (35.1)
	40–49	53 (34.4)
	50–59	23 (14.9)
	60 or older	6 (3.9)
	Other or not reported	*
Length of time practicing (years)
	<1–5	34 (22.1)
	6–10	28 (18.2)
	≥11	91 (59.1)
	Other or not reported	*
Member of a visible minority group
	Yes	19 (12.3)
	No	129 (83.8)
	Other or not reported	6 (3.9)
Ethnicity
	Indigenous to North America	7 (4.5)
	Other North American origins	31 (20.1)
	European origins	94 (61.0)
	Asian origins	10 (6.5)
	Other or not reported	27 (17.5)
Level of education
	Some university, college, or trade school	6 (3.9)
	Completed college or trade school	20 (13.0)
	Bachelor's degree from a university	49 (31.8)
	Graduate school	76 (49.4)
	Other or not reported	*

Note: “Other” professional roles included physicians, community connectors, and respondents who did not report a role. “Other” ethnicity responses included Caribbean origins, Latin, Central and South American origins, African origins, and respondents who did not report their ethnicity; numbers do not equal 154 because participants could select multiple responses.

*Unable to report.

Qualitative results

Four organizational barriers and four facilitators were identified through the analysis, informing three overarching themes (see Figure 1). The overarching themes are: (1) Navigating the ‘new e-world’: Training, education, and information to increase knowledge and use of dMH; (2) “All hands in the kitchen”: Learning to work together through coordinated digital and MH systems, and (3) “Change weary” but “nimble”: Shifting mindsets and approaches to care. Within these themes, barriers included obstacles to integrating dMH into MH practice (in terms of clinical limitations, current platform customization, and privacy concerns), self-reported low digital literacy on the part of clinicians, limited inter-organizational communication to collaborate across sectors, and skepticism surrounding the time commitment and implementation's sustainability. Facilitators included the potential for optimizing clinical practices and efficiency, for improving continuity of care and transitions between and across sectors, and for improving quality of care and empowering youth, and positive attitudes towards dMH and MBC.

Theme 1: Navigating the ‘new e-world’: Training, Education, and Information to Increase Knowledge and Use of dMH

Figure 1.

Overview of overarching themes, barriers, and facilitators.

Data informing this theme includes providers’ perceptions on two barriers: obstacles integrating dMH into mental health practices (i.e. concerns about current platform customization, clinical limitations, legal and privacy concerns) and self-reported low digital literacy among providers.

Obstacles integrating dMH into MH practice: “my biggest fear is how to integrate it into workflows”

Providers across sectors highlighted the need for dMH platforms to be flexible in tailoring to their practice's scope. One community provider expressed that it would be problematic to not have “the ability to cut out different assessments or [be] able to customize it, because it would be a huge change for our practice” (P24–2). Providers worried that the large range of assessment domains could result in gathering information they did not feel qualified to collect and interpret. Providers in PCN and SMHS also expressed that clinical policies and procedures related to short-term care plans for youth may limit use of the Innowell platform since youth access is tied to a provider. Providers raised concerns that youth would lose access after completing their sessions with their provider. This led to hesitancy towards enrolling youth on the dMH platform. One provider stated:

… we’re getting pressure from our own system to have short-term therapy. It's going to make it harder for clinicians … to want to enroll these [youth] because we might only have ‘X’ number of sessions and then we can’t keep them open. So, we have to close them and then when we close them … we know that that means you have to end it for the youth also, because you’re no longer connected. (P23.7)

Providers also expressed concerns as to how dMH aligned with their clinical data protection policy, legal privacy obligations, and duty to protect sensitive information. Questions arose, such as “where is this [assessment information] housed? How confidential is it?” (P6.1), “Where is the server? In the cloud? Okay, how is it encrypted?” (P24.2). Lack of awareness of these technical details made providers hesitant to engage in use of dMH out of fear of the legal implications of sensitive data handling. One provider expressed that data privacy considerations regarding sharing youth information limited their willingness to work with providers in other settings using dMH: “It's not that … I don’t want to work with you, it's [that] there's this use act that is legally binding that could affect my ability to practice if I go against the act” (P9.8). Understanding how dMH complied with legal requirements and safeguarded private information was considered a prerequisite before providers would “be able to even go near it” (P24.2).

Self-reported low digital literacy: “stumbling block”

A lack of familiarity with digital modes of mental health care provision emerged as a potential obstacle to implementation. Some providers were concerned that they lacked the digital literacy and technical skills to navigate and integrate new technologies into their mental health practice. One provider stated, “this kind of … e-world and online or electronic resources—this is still new to a lot of clinicians” (P13.1). Some providers described feeling intimidated and discouraged by the platform's features, for example, a provider expressed, “I haven’t seen the whole site yet, but just in the glimpse that I had, ya lost me!” (P6.1). Providers underscored how some healthcare providers “aren’t necessarily as apt to pick up the Internet side of things or the app side of things …The logistics of that I think could be a little bit of a stumbling block” (P18.2).

Theme 2: “All the hands in the kitchen”: Learning to Work together through Coordinated Digital and MH Systems

Within this theme, barriers included limited inter-organizational collaboration and communication and a lack of defined roles between mental health service settings and across adolescent and adult mental health services. Two facilitators were also noted in the potential for improving continuity of care and enabling cross-sector collaboration when transitioning youth between and across healthcare settings, as well as a perceived potential for dMH to optimize clinical practices and efficiency.

Limited collaboration: “With all the hands in the kitchen, who becomes the chef?”

A perceived challenge of using dMH was a lack of inter-provider collaboration across separate service settings when providing care to the same youth, either concurrently or throughout ongoing care across different systems. Participants reported that few mechanisms facilitated information exchange and collaboration to bridge gaps between service contexts and ensure a cohesive approach to patient well-being, which constitutes a barrier to implementing dMH. One provider in SMHS highlighted this issue, explaining that even though their organization's youth and adult mental health services were offered in the same building, “we don’t have necessarily structured communication” (P6.1) to facilitate the transition for clients turning 18 and moving to adult care. A school provider also highlighted this barrier, stating:

…when kids are referred to the clinic and the next day they show back up in school, there seems to be a lack of communication at times. Right? Not knowing what supports are in place and … what level of risk they’re at or how are you going to support that child. (P22.4)

In discussing this gap, some providers voiced that using dMH would require establishing strategic processes for communication between service providers.

A related concern was that the roles and responsibilities of providers across settings would need to be clearly defined if the dMH platform were to be used by multiple service providers with a single client, particularly for youth in distress. A provider stated, “So, if you have multiple caregivers or providers, whose responsibility is it to contact that client if there's an escalation of suicide risk?” (P12.1). An organizational barrier exists in the current lack of communication and defined roles across the sector, particularly regarding imminent suicide risk.

Relatedly, providers noted there could be overlap of the dMH platform with existing Electronic Medical Record (EMR) management systems, leading to redundancies and duplication of work. For example, a provider noted that they would have to reassess their clients in the EMR if they flagged a notification on the Suicidal Thoughts and Behaviors domain, indicating a high score on that measure:

I mean if they’re putting on this [dMH] app that everything is escalating including their risk, then we theoretically should go into [the EMR], do a risk assessment there as well. Right? So, again how much of that information do we use into our own charting Electronic Medical Records charging system?. (P6.1)

Other providers observed that a potential lack of interoperability between the dMH platform and EMR would mean having to “duplicate services” (P2.2) and information across multiple platforms, essentially “using two different tools for the same thing” (P2.2).

Potential to improve continuity of care and transitions between and across sectors

Despite the concerns about communication between settings, providers in PCN and SMHS settings also identified that a dMH platform had the potential to allow youth to “still have a health care provider that is checking in on them through that app” (P2.3) post-discharge, or to use the platform “as part of a discharge plan” (P7.3). Participants shared that dMH would be helpful in improving practices surrounding continuity of care, with one PCN provider stating, “A lot of times after we do the 3–6 sessions … if they’re just not quite ready or they're not doing well, we will refer to [a higher level of care]. So, in the meantime if [dMH] can be kind of that gateway” (P2.5) to monitor progress between in-person service provision. Providers also discussed how dMH could be a valuable tool for youth as they transition in and out of provider care. A school-based provider appreciated the “opportunity to provide something that moves on past high school” (P5.3). Providers hoped that a dMH platform could be used to bridge the gap between high school and post-secondary, a gap in which they had traditionally “lost all that support and there's no easy transition” (P25.1). The potential of dMH to attach a digital mental health tool and record to youth was considered beneficial as providers in a new care setting may have greater access to information on past care and symptoms.

Providers expressed a desire for more collaboration, coordination, and communication between service settings across the sector. For instance, school setting providers felt they were working in silos, and one PCN provider described the lack of communication as “work[ing] in silence” (P18.2). There was some hope that the dMH platform could reduce this, as evidenced through a participant stating, “I’m excited because there's collaboration. There's going to be collaboration within our community” (P5.1). The potential of working together was hoped to “take some strain off of the system as a whole, not just the school or [SMHS] or all of the places that these kids end up—I’m very hopeful that it’ll alleviate some of that pressure” (P14.1).

Potential to optimize clinical practice

Despite the above barriers, providers expected that dMH could enhance clinical practice, organizational efficiency, and triaging. Providers appreciated that using MBC through dMH would allow them to use session time effectively to focus on and track clients’ most urgent needs. For example, a participant noted, “I think having pre- and post- outcomes would actually allow me to have more time to give better care to my clients” (P4.4). Beyond sessions with clients, participants felt the platform would facilitate organizational efficiency within clinics. One provider stated, “Over time it will help us focus on those clients that… need to be seen at the clinic, and the ones that are a little bit lesser needs that can be seen through the e-Mental Health platform” (P2.6). Other providers agreed that the platform could facilitate prioritization of care needs across symptom severity, operating as a triage tool that could direct youth to appropriate services. They discussed using the platform to connect youth to specialists if needed, for instance:

Are we the best place … to provide the support that they actually need?… I had a few eating disorder referrals come to me and after triage it felt like, “You know what, that isn’t appropriate for my service. They need a more specialised, more intense team around them. (P4.3)

Providers perceived that the platform questionnaire results could make referral decisions more objective instead of relying solely on providers’ professional capabilities and judgements about the severity of the young person's mental illness.

Theme 3: “Change Weary” but “Nimble”: Shifting Mindsets & Approaches to Care

Within this theme, a barrier was providers’ skepticism surrounding the longevity and time commitment of a dMH implementation, and two facilitators emerged: the perceived potential to improve quality of care and empower youth, and positive attitudes towards greater use of dMH-enabled MBC.

Skepticism of time commitment and implementation longevity: “a little bit change weary”

Participating providers perceived that implementing dMH would be a time-intensive endeavor amid already demanding schedules and existing workflows. One provider stated, “I’m sure we’ll get the training but it's more the time. It's the timeline to do this. That's the biggest thing. We run a tight ship at this clinic” (P16.3). Beyond the time needed to learn the new platform, monitoring MBC results would be challenging: “So, if we’re doing like a bunch of assessments, I feel like in a fifteen-minute session where I see them once a month, it’d just be too focused on assessments” (P24.2). The sense of obligation to “look at every single assessment that they do and debrief it with them” (P24.2) was considered a barrier, especially in managing large caseloads.

Participants’ skepticism to dMH was also related to uncertainty about the long-term sustainability of the implementation. Participants expressed a sense of change fatigue at the potential for the time investment to not be sustained over the long term. One provider said:

I’m just worried that we’re going to get something really good rolling, or not and then it's going to end… Or is it going to be for real? …I guess I’m a seasoned social worker, and it's nice to see some initiative, but I was also a little bit change-weary in regards to if this is not going to be renewed. (P23.1)

Organizational barriers such as change fatigue and a lack of clarity regarding the longevity of the implementation limit the potential uptake and use of the dMH platform.

Empowering youth and improving quality of care: “the keeper of their own information”

With the abundance of digital resources and tools available to youth, providers appreciated that the dMH platform could provide timely and accurate “evidence-based information” (P4.1) that they could professionally endorse. The abundance of care resources and tools were considered a major asset, with one provider stating, “anytime we can give youth more tools and empower them, I think that's always beneficial” (P17.3). Providers also perceived that the platform would empower youth to be active participants in their care and to “self-advocate” (P5.4). For instance, one provider stated that it was “super empowering [that] the youth can access [the results] themselves, they’re accountable and they have that immediate data or feedback to themselves” (P5.1). Long-term use of the dMH platform could enable tracking progress throughout treatment and across providers, positioning youth as “the keeper of their own information” (P4.5). In this way, the platform offered potential to empower youth with resources, tools, and a sense of autonomy in managing their mental health.

Positive attitudes towards dMH and MBC: “we are nimble”:

Some participants across sites indicated a sense of readiness for a dMH platform. Some organizations embodied a culture of improvement and innovation to support clients’ needs. One provider emphasized the adaptive nature of their workplace culture, describing it as “nimble” and stating that the staff “evolve around the environment” and use resources that are “new and innovative” (P13.1). In another group, a participant described their workplace as “really creative” (P24.3), with management that was “always willing to just try things and see if it works” (P24.3), which had led to two successfully implemented programs in the past. Providers in another focus group mentioned they had previously adjusted their working hours to allow for appointments in high demand time slots such as the lunch hour. Providers with this sense of innovation and flexibility deemed the dMH implementation more feasible, as noted through expressions of interest and enthusiasm for the platform's potential to meet the needs of providers and youth.

Some providers also noted an increase in familiarity with digital tools and platforms given the rapid digitization of their work since the onset of the COVID-19 pandemic. The pandemic catalyzed virtual offerings of mental health services (e.g. appointments), and some providers still worked in hybrid arrangements. For instance, one participant stated, “I think also for the clinic, a lot of us are also still kind of working virtually. Right? Unless we actually have clients to come in to see” (P23.1). Providers had gained familiarity with digital tools in their practice, which was seen in a positive light by some providers, as illustrated by one provider stating, “the one positive that came out of COVID … is doing Zoom and offering that option” (P19.2). The pandemic led to changes in attitudes and beliefs surrounding technology, suggesting an increase in interest in and acceptability of using digital tools.

Discussion

This study aimed to explore providers’ perspectives on organizational factors impacting their use of dMH with youth in Alberta. Their perspectives included barriers and facilitators that could constrain or enable dMH implementations. These results informed three overarching themes aimed at alleviating organizational barriers and enabling facilitators to support the use of dMH in mental health care settings. The first theme is the need for training, education, and information to increase knowledge and use of dMH; this theme addresses the barriers of obstacles to integrating dMH into practice and self-reported low digital literacy among providers. The second theme is the need for coordinated digital and mental health systems across sectors; this theme relates to the barrier of limited inter-organizational communication and the lack of defined roles and responsibilities supporting clients across service settings, and the potential for optimizing clinical practices, improving clinical efficiency, and improving continuity of care and transitions between and across sectors. The third theme is the need to leverage behavioral change towards dMH and MBC; this theme entails addressing skepticism regarding the time commitment and the implementation's longevity and building on positive attitudes towards and the potential to empower youth and improve the overall quality of care.

The need for training, education, and information to increase knowledge and use of dMH

Our findings suggest that providers face challenges integrating dMH into clinical practices and policies and gaining skills for navigating the technical aspects of digital mental health care provision. This indicates a need for training and information to increase providers’ knowledge and capacity to use the platform and integrate MBC into practice. To address this need identified in the pre-implementation stage, training was offered during the implementation both in-person by implementation leads and through online learning modules, which fit flexibly into providers’ schedules and were scalable across sites. Building capacity among designated on-site champions or implementation leaders allowed local leads to field questions specific to clinical policies and procedures. Importantly, equipping senior leadership with change management strategies and support further facilitated the implementation. These strategies were consistent with literature addressing the need for training prior to implementing dMH.^51,52

Participants were concerned about how dMH would align with clinical and professional obligations to protect patient privacy. This finding is consistent with other studies exploring the adoption of technology in mental health care.^53–55 Professional associations such as the Canadian Federation of Mental Health Nurses and the Canadian Counselling and Psychotherapy Association outline standards and best practices regarding the use of technology in mental health care.^56,57 These guides are live documents, with new editions revised and released in recent years establishing current best practices for technology and privacy in dMH; sharing emergent best practices with providers may help to address their concerns regarding the legal stipulations of dMH implementations. Organizations seeking to implement dMH solutions should provide training on the intersection of technology, privacy, and mental health to increase uptake among providers on the role they play in safely handling data and managing caseloads.

In this study, participants also identified the challenge of adjusting to novel online resources for mental health care delivery with sometimes limited digital literacy. While this barrier may reduce naturally over time as a new generation of ‘digital natives’ (i.e. people who grew up with technology) become the main workforce, in the meantime, there is a need to support and develop technical competencies needed to use dMH platforms by providing training on technical skills. This aligns with findings in the literature, such as Bucci et al.'s³⁴ study reporting that a lack of familiarity and confidence in digital abilities impeded the use of online mental health tools used in providers’ practice. Employing digital navigators trained in mental health technology whose role is to support providers in operating and troubleshooting devices and software may be an effective way to address this barrier and increase user engagement.^58,59 Our study suggests there is potential to leverage interest in the uptake of digital tools to increase quality and continuity of care, especially following the pandemic.^60,61 The literature suggests a shift towards adopting technology to meet the needs of clients.⁶² This suggests that supporting dMH adoption through training can bridge digital ability gaps with high-interest levels in dMH implementation.

The need for coordinated digital and mental health systems

While findings in this study suggest that participants deemed there to be barriers in current approaches to inter-organizational communication and collaboration, they also noted the potential for dMH to bridge the gap by enabling cross-sector coordination. As a result, dMH may positively impact clinical practices, clinical efficiency, and continuity of care during transitions between and across sectors. Improving care coordination has also been linked to major decreases in self-harm and hospitalizations.²¹ The findings of this study indicate that dMH could be a promising solution to increase collaboration 1) between and among different mental health service providers within a mental health service and across sectors including from adolescent to adult services, 2) between youth and their providers, and 3) between digital operating systems.

First, participants in this study indicated that mental health care provision is fragmented across organizational silos, a finding mirrored in other mental health contexts.^63–65 There would be increased interest if the platform enabled inviting and collaborating with others (e.g. teachers, doctors, parents) in youth care plans and dMH journeys. Appropriately designed dMH has the potential to support inter-organizational and interdisciplinary care collaboration^66,67 through increased communication between each patient-provider, provider-provider, provider-school, and specialist-family practitioner. dMH supports continuity of care by connecting youth to resources, information, and services, ensuring that they will have timely access to the care they need.⁶⁸

Second, dMH could enable improved continuity of care among clients who receive short-term treatment plans if youth had continued access to their data/tools on the dMH platform. Other studies highlight the benefits of using an online platform to connect youth with providers after discharge from clinical care.^69,70 An Ontario study monitored the implementation of a mobile application called “Be Safe” used by youth after discharge from a psychiatric unit. The platform gave youth access to their personal safety plan and local mental health and addiction resources.⁶⁹ Another study by Montague et al.⁶⁵ further illustrates how online therapeutic modalities can bridge transitions in care through supporting recovery and preventing relapse. Providing youth access to their own data can empower them to share it wherever they go. dMH platforms could be an optimal solution for youth when transitioning between care.

Third, participants in this study indicated that a barrier to integrating dMH was the functional overlap with existing EMR procedures in Alberta. A lack of interoperability would create more work for providers if information cannot be shared between the dMH and EMR systems, requiring manual duplication from one system to another. Other studies report similar barriers to interoperability, including privacy and confidentiality concerns, lack of standardized protocols, and challenges in data integration.^71,72 Increasing interoperability could reduce redundancies and save time through automation.⁷³ There is a need to make dMH innovations congruent with existing digital health records to increase the acceptability and feasibility of their uptake. It is also crucial to align procedures for collecting and recording data with the goal of delivering quality care to clients, rather than recording activities to maintain funding cycles. It will remain a challenge of the 21st century to identify strategies for addressing these complexities and enabling greater communication between digitized health systems.

Leveraging behavioral change towards new paradigms of MH practice using MBC

Our findings indicate a need to address providers’ concerns regarding the time required to learn how to use dMH, which can be facilitated by building on positive attitudes towards dMH and the potential for dMH to empower youth and improve the overall quality of care. Providers’ initial perceptions of the dMH platform included a view that it would be too time-consuming to adopt, especially under increasing pressure to offer short-term care for clients. This trend in MH is a barrier to the implementation and use of MBC systems. This finding aligns with the existing literature in which implementations were met with skepticism for being “overly time consuming” (28, p. 3) or seeming to increase workload.⁵⁵ To that end, supporting successful organizational implementations can include nurturing professional networks among providers^74,75 and involving staff in discussions about the implementation. This was found to lead to ‘role changing practices’ in a longitudinal study looking at data from 517 hospitals.⁷⁶ Additionally, restructuring providers’ workloads to allow time to integrate MBC may increase their uptake. For a successful implementation of the dMH platform, the willingness and adaptability of the professionals involved must be accounted for.

Promisingly, the findings of this study indicate a growing willingness to shift towards digital modalities of dMH that use MBC. Participants valued that dMH empowered youth by enabling them to monitor personal progress using MBC outcomes. Despite the evidence supporting the utility of MBC,^28,77 the reality has been that most services have not previously had access to IT-based systems to support its everyday implementation. Other literature suggests that although providers value monitoring patient care, there remains low adherence to standardized procedures for collecting measures.⁷⁸ Hence, a new service activity (and assumed capacity of services to respond appropriately) presents a challenge when introduced alongside the technology platform. Experiences elsewhere,^15,25 indicate that although desirable, this is very challenging for providers and service organizations.

Providers expected the tools and resources on the platform to increase youth engagement and accountability. This is consistent with findings in a study by Cliffe et al.⁷⁹ indicating that 88.3% of providers surveyed appreciated that dMH enabled easy distribution of resources to youth, especially self-guided tools.³⁴ While this does not guarantee that resources will be used or have an impact among young adults, participants in this study were optimistic that the resources, combined with the ability to track MBC outcomes would optimize the quality of care delivered to clients during in-person sessions. Similar optimism is reported in other studies where tracking client progress through longitudinal data helped inform clinical decision-making, generate treatment plans across a spectrum of mental illness acuity, support targeted care over time, improve efficiency of time used during sessions, and increase the frequency of interactions between clients and providers.^34,80 While providers in some service settings in this study were hesitant to endorse assessments for mental health domains beyond the scope of their practice—consistent with findings by Zhao et al.⁸¹—they recognized the value of identifying youth needs and being able to give referrals to clients whose needs exceeded a provider's capacity.^68,82 It is worth noting that dMH entails a broad range of digital interventions that do not always include MBC. dMH and MBC each have their own implementation challenges. A dMH platform that uses MBC requires thoughtful consideration to deliver the benefits of both.

Strengths and limitations

Using a semi-structured approach in focus groups meant some discussions flowed more openly as participants identified key issues and opinions. A common challenge inherent in focus groups is that certain participants may dominate the discussion sessions and participants may only voice opinions they believe would be accepted by the group, potentially limiting a full spectrum of perspectives. Clinical leads and management were often present during focus groups, which enabled leadership to understand perspectives within their settings, but this may have limited the extent that some participants openly shared views specific to their context. Our sample lacked diversity in race and gender and had low representation from school/community-based settings, which may limit the generalizability. This study also had several strengths, including the involvement of youth research partners in the analysis of focus group data. Also, the recruitment of participants from diverse mental health settings across urban and rural Alberta communities provided a wide range of viewpoints, allowing a comprehensive view of pre-intervention feasibility. Since this data is pre-implementation data, perspectives of participants may change over time; future research directions will focus on evaluating specific organizational factors that impact dMH and MBC implementation through post-intervention data collection. It will also be important to involve the voices of youth participating in dMH implementations as their insights and experiences should shape approaches to dMH moving forward.

Practical recommendations for organizations

The themes identified in this paper provide a practical approach to implementing dMH platforms and innovations in clinical settings with youth experiencing mental health concerns (see Table 2 for the summary of practical recommendations). First, a dMH implementation must be supported by adequate training to equip providers with information to recognize the value of using dMH to really deliver MBC in mental health care. This is a challenging task for services that have not had prior experience with delivering real-time MBC. Training needs include ways to overcome challenges, understand measures in place for privacy and safe data handling, and increase providers’ digital literacy and familiarity using online platforms and tools for mental health care. Training could include workshops or professional development sessions where providers could participate in hands-on sessions to learn how to navigate dMH platforms and explore case studies that integrate dMH into existing clinical workflows or adapt those workflows to respond appropriately to these new real-time sources of information. Online resources, tutorials, and guides could also be accessible for providers to consult when troubleshooting common issues or seeking information on privacy protocols. Mentoring programs could also support providers by pairing them with experienced users who can provide guidance, answer questions, and share best practices. These types of training approaches should teach both technical aspects of dMH as well as contextual considerations such as privacy, ethics, and integration into clinical practice and community.

Table 2.

Practical recommendations for organizations.

Recommendation	Examples
Provide training to equip providers with information about the value of dMH to deliver MBC.	Workshops, PD sessions, opportunities for hands-on exploration of the dMH platform and resources through demo accounts and clinical examples. Online resources, tutorials, and guides (i.e. practical use of MBC, how to invite and onboard clients’ to the platform, how to discharge clients from the platform). Mentorship programs for dMH.
Consider the fit of the dMH platform in context.	Adjust providers’ workloads to accommodate the implementation. Optimize cross-sectoral communication to improve care coordination. Engage providers in discussions about how their clients would be best served by the dMH platform. Remain flexible in how the dMH platform best fits the needs of the organization.
Build on the interest of providers to use digital modes and implement MBC.	Promote and embed leaders who are champions of dMH. Support communal learning and sharing of ideas, including strategies for addressing challenges and success stories. Highlight alignment of the dMH platform / MBC with providers’ approaches to care (e.g. client-centered, solution-focused, etc.).

Second, when organizations adopt a dMH approach or platform, they should consider how it fits within their context in terms of the mental health services provided, the existing dMH records in place, and the likelihood that the intervention could enable inter-organizational communication and collaboration in supporting care for patients moving between service settings. This will aid in optimizing clinical workflows, modifying existing practices and cross-sectoral efficiency, and will help providers see value in using the platforms. Finally, leadership and management in settings hoping to integrate dMH should consider ways to build upon the interest among providers to use digital modes of MH care and really implement MBC effectively. Leveraging the momentum and interest of providers through enthusiastic leadership, nurturing a flexible and innovative workplace, adjusting providers’ schedules and caseloads to account for the learning curve of an innovation, and enabling opportunities for sharing the learning process as a clinic, can greatly enhance the successful adoption of dMH platforms.

Conclusion

This study aimed to understand mental health care providers’ perspectives on the organizational factors surrounding the implementation of a dMH platform across Alberta in school and community settings, SMHS, and PCNs. The diverse perspectives highlighted in our findings demonstrate the crucial need to consider how organizational factors limit or strengthen implementation efforts. Decision-makers in organizations must implement organizational-level strategies to support the success of dMH and MBC-focused interventions, rather than leaving it up to healthcare providers to navigate individually. Organizational strategies, such as providing training, optimizing clinical policies to support new workflows, adapting existing work practices enhancing inter-organizational partnerships and collaboration, and establishing enthusiastic and supportive leadership, all have the potential to facilitate successful and sustained implementation of dMH interventions in practice.

Footnotes

Acknowledgements

The authors extend their gratitude to the dedicated research assistants who supported this project: Brea McLaughlin, Simron Sidhu, and Katelyn Greer. The authors also acknowledge the collaborative efforts of the team responsible for implementing the electronic mental health platform for facilitating data collection and providing feedback throughout the project. Finally, the successful completion of this project was made possible thanks to the support and involvement of many stakeholders and participants whose voices informed the direction of the research.

Contributorship

Study conception and design: GD. Data collection: GD, EB, KB, KM, MF, and LS. Analysis and interpretation of results: GD, KB, JHG, EB, MB, KP, SD, LN, and LV. Draft manuscript preparation: MB, KP, SD, LV, KB, EB, JHG, and GD. All authors reviewed the results and approved the final version of the manuscript.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethical approval

This research study received ethical approval from the Conjoint Health Research Ethics Board at the University of Calgary (“eMH Qualitative Study—Pre-Implementation and Implementation”—REB20–1137). All participants provided written informed consent via an online consent form.

Funding

The authors disclose receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Alberta Children's Hospital Foundation; Alberta Health; the Partnership for Research and Innovation in the Health System (PRIHS); and CIHR (Operating Grant: SPOR iCT Rewarding Success).

Guarantor

ORCID iDs

Marianne Barker

Julia Hews-Girard

Emilie Bassi

Jason Gondziola

Gina Dimitropoulos

Haley LaMonica

Appendix 1—Guiding questions

Guiding questions
Knowledge	What do you know about online digital Mental Health (dMH) interventions and apps/tools for youth/young adults? Probe: What knowledge is needed to increase user acceptance, engagement and comfort with using dMH platform? What beliefs about dMH interfere with engaging youth? What information/training would you find helpful to increase and better engage your use of dMH interventions with youth in your clinical practice? How would you envision dMH being used in both intake assessments and ongoing treatment planning? What information/training do you need to implement dMH interventions for information sharing and communication with other providers within your organization/agency? Probe: Are there other options, e.g. using during case rounds? What information/training do you need to implement dMH interventions for information sharing and communication with providers across different systems such as primary care networks, SMHS, and other youth serving organizations to improve clinical care? What information/training do you need to help you use dMH to support the transfer of care from adolescent to adult programs?
Skills	Ask questions from preceding sections but probe about skills.
Professional Role	What do you know about your profession's stance on the use of dMH? Do you perceive any professional/ethical issues related to dMH? Do you have any thoughts about privacy, confidentiality, and patient safety when using dMH? Are you on a multidisciplinary team? If so, how might different roles/ professions interact with/influence the implementation of dMH?
Beliefs about Capabilities	How would you characterize your digital readiness to incorporate dMH? How would you characterize your service's (or organization's) digital readiness to incorporate dMH? What impacts your confidence level? What raises it, lowers it, maintains it?
Outlook	What value could you see dMH bringing to your practice with youth and their caregivers? What would make you want to incorporate dMH into your practice? Are there youth/young adult populations who have been disproportionately affected by COVID-19? What about certain mental health services? Are there opportunities to address these gaps with dMH?
Intentions	What do you believe is the primary intention for using dMH in your organization/agency/service with youth? What do you believe is the intention of your organization/agency/service for using dMH with other providers from different systems?
Environmental	What factors in your organization might influence your use of dMH? What unintended/intended consequences might result from using dMH? What are the barriers/facilitators of using dMH during a pandemic? What are the downsides/benefits of using dMH during a pandemic? Does your agency serve youth that may have specific barriers to accessing/ using dMH, for example youth with cognitive or physical disabilities?
Social Influences	What role do you hope your supervisor or leaders will play in providers using dMH with youth? Do your colleagues influence your likeliness of using dMH? What influence do you have on your supervisor or leader in how dMH is used with youth/young adults? Does your relationship with the community in which you practice have any influence on the use of dMH? If so, how? What unique factors in your community might influence the use of dMH? Probe: Geographic location, culture, internet connectivity, other contextual variables. Do you experience pressure that influences your use of dMH? If so, what types of pressure? How do you deal with this?
Emotion	What feelings come up when you think about implementing dMH into your scope of practice? How are you feeling about participating in this dMH project at this time?
Behavioral Regulation	Do you currently have a way of measuring progress or change for youth in your services? What are the key metrics? How could dMH help to track and demonstrate progress? How could dMH help us track the degree to which organizations, agencies, and services are collaborating?

References

Keyes

Gary

O’Malley

, et al. Recent increases in depressive symptoms among US adolescents: trends from 1991 to 2018. Soc Psychiatr Epidemiol 2019 Aug 1; 54: 987–996.

Wiens

Bhattarai

Pedram

, et al. A growing need for youth mental health services in Canada: examining trends in youth mental health from 2011 to 2018. Epidemiol Psychiatr Sci 2020; 29: e115.

Loades

Chatburn

Higson-Sweeney

, et al. Rapid systematic review: the impact of social isolation and loneliness on the mental health of children and adolescents in the context of COVID-19. J American Acad Child & Adol Psych 2020; 59: 1218–1239.e3.

Madigan

Vaillancourt

Dimitropoulos

, et al. A systematic review and meta-analysis: child and adolescent healthcare utilization for eating disorders during the COVID-19 pandemic. J American Acad Child Adol Psych 2024; DOI: https://doi.org/10.1016/j.jaac.2024.02.009.

Racine

McArthur

Cooke

, et al. Global prevalence of depressive and anxiety symptoms in children and adolescents during COVID-19: a meta-analysis. JAMA Pediatr 2021; 175: 1142–1150.

Mazidi

, et al. Prevalence of mental health problems among children and adolescents during the COVID-19 pandemic: a systematic review and meta-analysis. J Affect Dis 2021; 293: 78–89.

Jaworska

MacQueen

. Adolescence as a unique developmental period. J Psychiatry Neurosci 2015; 40: 291–293.

Kessler

Amminger

Aguilar-Gaxiola

, et al. Age of onset of mental disorders: a review of recent literature. Cur Opin Psych 2007; 20: 359–364.

Malla

Shah

Iyer

, et al. Youth mental health should be a top priority for health care in Canada. Can J Psychiatry 2018; 63: 216–222.

10.

Acharya

Jin

Collins

. College life is stressful today – emerging stressors and depressive symptoms in college students. J American Coll Health 2018; 66: 655–664.

11.

Kowalewski

McLennan

McGrath

. A preliminary investigation of wait times for child and adolescent mental health services in Canada. J Can Acad Child Adolesc Psychiatry 2011 May; 20: 112–119.

12.

Moroz

Angelo

, et al. Mental health services in Canada: barriers and cost-effective solutions to increase access. Healthc Manage Forum 2020; 33: 282–287.

13.

Fante-Coleman

Jackson-Best

. Barriers and facilitators to accessing mental healthcare in Canada for black youth: a scoping review. Adolescent Res Rev 2020; 5: 115–136.

14.

Kourgiantakis

Markoulakis

Hussain

, et al. Navigating inequities in the delivery of youth mental health care during the COVID-19 pandemic: perspectives of youth, families, and service providers. Can J Public Health 2022; 113: 806–816.

15.

Davenport

Cheng

VWS

Iorfino

, et al. Flip the clinic: a digital health approach to youth mental health service delivery during the COVID-19 pandemic and beyond. JMIR Ment Health 2020; 7: e24578.

16.

Ellis

Augustsson

Grødahl

, et al. Implementation of e-mental health for depression and anxiety: a critical scoping review. J Comm Psychol 2020; 48: 904–920.

17.

Kruzan

Fitzsimmons-Craft

Dobias

, et al. Developing, deploying, and evaluating digital mental health interventions in spaces of online help- and information-seeking. Procedia Comp Sci 2022; 206: 6–22.

18.

Strudwick

Sockalingam

Kassam

, et al. Digital interventions to support population mental health in Canada during the COVID-19 pandemic: rapid review. JMIR Ment Health 2021; 8: e26550.

19.

Graham

Weissman

Mohr

. Resolving key barriers to advancing mental health equity in rural communities using digital mental health interventions. JAMA Health Forum 2021; 2: e211149.

20.

Iorfino

Piper

Prodan

, et al. Using digital technologies to facilitate care coordination between youth mental health services: a guide for implementation. FrontHealth Serv 2021; 1: 745456.

21.

Iorfino

Occhipinti

Skinner

, et al. The impact of technology-enabled care coordination in a complex mental health system: a local system dynamics model. J Med Int Res 2021; 23: e25331.

22.

Andreassen

Trondsen

Kummervold

, et al. Patients who use E-mediated communication with their doctor: new constructions of trust in the patient-doctor relationship. Qual Health Res 2006; 16: 238–248.

23.

Dilkes-Frayne

Savic

Carter

, et al. Going online: the affordances of online counseling for families affected by alcohol and other drug issues. Qual Health Res 2019; 29: 2010–2022.

24.

Wies

Landers

Ienca

. Digital mental health for young people: a scoping review of ethical promises and challenges. Front Dig Health 2021; 3, Available from: https://www.frontiersin.org/articles/10.3389/fdgth.2021.697072.

25.

Iorfino

Cross

Davenport

, et al. A digital platform designed for youth mental health services to deliver personalized and measurement-based care. Front Psych 2019; 10, Available from: https://www.frontiersin.org/articles/10.3389/fpsyt.2019.00595.

26.

Cárdenas

Bartels

Cruz

, et al. Perspectives, experiences, and practices in the use of digital information technologies in the management of depression and alcohol use disorder in health care systems in Colombia. Qual Health Res 2020; 30: 906–916.

27.

Ospina-Pinillos

Davenport

Iorfino

, et al. Using new and innovative technologies to assess clinical stage in early intervention youth mental health services: evaluation study. J Med Internet Res 2018; 20: e259.

28.

Cuperfain

Hui

Berkhout

, et al. Patient, family and provider views of measurement-based care in an early-psychosis intervention programme. BJPsych Open 2021; 7: e171.

29.

Lewis

Boyd

Puspitasari

, et al. Implementing measurement-based care in behavioral health: a review. JAMA Psychiatry 2019; 76: 324.

30.

Murphy

Khan

Sun

, et al. Needs, gaps and opportunities for standard and e-mental health care among at-risk populations in the Asia pacific in the context of COVID-19: a rapid scoping review. Int J Equity Health 2021; 20: 161.

31.

Aboraya

Nasrallah

Elswick

, et al. Measurement-based care in psychiatry—past, present, and future. Innov Clin Neurosci 2018 Nov 1; 15: 13–26.

32.

Oslin

Hoff

Mignogna

, et al. Provider attitudes and experience with measurement-based mental health care in the VA implementation project. PS 2019; 70: 135–138.

33.

LaMonica

Iorfino

Lee

, et al. Informing the future of integrated digital and clinical mental health care: synthesis of the outcomes from project synergy. JMIR Ment Health 2022; 9: e33060.

34.

Bucci

Berry

Morris

, et al. They are not hard-to-reach clients. we have just got hard-to-reach services. Staff views of digital health tools in specialist mental health services. Front Psych 2019; 10: 1–14. Available from: https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2019.00344.

35.

Zurynski

Ludlow

Testa

, et al. Built to last? Barriers and facilitators of healthcare program sustainability: a systematic integrative review. Implementation Sci 2023; 18: 62.

36.

Stroul

Manteuffel

. The sustainability of systems of care for children’s mental health: lessons learned. J Behav Health Serv Res 2007; 34: 237–259.

37.

Neergaard

Olesen

Andersen

, et al.

Qualitative description – the poor cousin of health research?

BMC Med Res Methodol 2009; 9: 52.

38.

Sandelowski

. Whatever happened to qualitative description? Res Nurs Health 2000; 23: 334–340.

39.

Bradshaw

Atkinson

Doody

. Employing a qualitative description approach in health care research. Global Qual Nurs Res 2017; 4: 233339361774228.

40.

Bingham

Witkowsky

Vanover

, et al. Analyzing and interpreting qualitative data: after the interview. In: Vanover

Mihas

Saldaña

(eds) Deductive and inductive approaches to qualitative data analysis. Sage Publications, Inc.: Thousand Oaks, CA, 2022, pp.133–146.

41.

Dimitropoulos

Bassi

Bright

, et al. Implementation of an electronic mental health platform for youth and young adults in a school context across Alberta, Canada: thematic analysis of the perspectives of stakeholders. JMIR Ment Health 2024; 11: e49099.

42.

Bassi

Bright

Norman

, et al. Perceptions of mental health providers of the barriers and facilitators of using and engaging youth in digital mental health enabled measurement-based care. Digital Health 2024; 10: 20552076241253093. DOI: https://doi.org/10.1177/20552076241253093.

43.

Dimitropoulos

Lindenbach

Potestio

, et al. Using a rapid learning health system for stratified care in emerging adult mental health services: protocol for the implementation of patient-reported outcome measures. JMIR Res Protoc 2024; 13: e51667.

44.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)—A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inf 2009; 42: 377–381.

45.

Harris

Taylor

Minor

, et al. The REDCap consortium: building an international community of software platform partners. J Biomed Infor 2019; 95: 103208.

46.

Kitzinger

. Focus groups. In: Qualitative research in health care. 3rd ed. Williston, VT: BMJ Books, 2006, pp.21–31.

47.

Tausch

Menold

. Methodological aspects of focus groups in health research: results of qualitative interviews with focus group moderators. Global Qualit Nurs Res 2016; 3: 2333393616630466.

48.

Cane

O’Connor

Michie

. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implementation Sci 2012; 7: 37.

49.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

50.

Braun

Clarke

. Toward good practice in thematic analysis: avoiding common problems and be(com)ing a knowing researcher. Int J Trans Heal 2023; 24: 1–6.

51.

Iorfino

Piper

Prodan

, et al. Using digital technologies to facilitate care coordination between youth mental health services: A guide for implementation. Front Heal Serv 2021; 1: 1–7.

52.

Silfee

Williams

Leber

, et al. Health care provider perspectives on the use of a digital behavioral health app to support patients: qualitative study. JMIR Formative Research 2021; 5: e28538.

53.

Braun

Drüge

Hennemann

, et al. Acceptance of E-mental health services for different application purposes among psychotherapists in clinical training in Germany and Switzerland: secondary analysis of a cross-sectional survey. Front Digit Health 2022; 4: 840869.

54.

Pierce

Perrin

McDonald

. Pre-COVID-19 deterrents to practicing with videoconferencing telepsychology among psychologists who didn’t. Psychol Serv 2022; 19: 157–166.

55.

Schueller

Washburn

Price

. Exploring mental health providers’ interest in using web and mobile-based tools in their practices. Int Interv 2016; 4: 145–151.

56.

Canadian Counselling and Psychotherapy Association [Internet]. Guidelines for the Uses of Technology in Counselling & Psychotherapy. Available from: https://www.ccpa-accp.ca/chapters/technology-counselling/. 2024.

57.

CFMHN [Internet]. Canadian Standards of Practice for Psychiatric-Mental Health Nursing 5th Edition — 2023. Available from: https://www.cfmhn.ca/standards. 2024.

58.

Chen

Lane

Burns

, et al. The digital navigator: standardizing human technology support in app-integrated clinical care. Telemed e-Health 2024; 30: e1963–e1970. Available from: https://www.liebertpub.com/doi/10.1089/tmj.2024.0023.

59.

Wisniewski

Gorrindo

Rauseo-Ricupero

, et al. The role of digital navigators in promoting clinical care and technology integration into practice. Digital Biomarkers 2021; 4: 119–135.

60.

Appleton

Williams

Juan

NVS

, et al. Implementation, adoption, and perceptions of telemental health during the COVID-19 pandemic: systematic review. J Med Int Res 2021; 23: e31746.

61.

Ağirkan

. The reflections of COVID-19 pandemic to counselling: bibliometric analysis of online psychological counselling research. British J Guid Counsell 2023; 51: 558–573.

62.

Békés

Aafjes-van Doorn

Luo

, et al. Psychotherapists’ challenges with online therapy during COVID-19: Concerns about connectedness predict therapists’ negative view of online therapy and its perceived efficacy over time. Front Psych 2021; 12, Available from: https://www.frontiersin.org/articles/10.3389/fpsyg.2021.705699.

63.

Flood

Thomas

. Fragmented Law & Fragmented Lives: Canada’s Mental Health Care System [Internet]. Rochester, NY; 2017 Jul [cited 2024 Feb 7]. Report No.: 3662667. Available from: https://papers.ssrn.com/abstract=3662667.

64.

Halsall

Manion

Iyer

, et al. Trends in mental health system transformation: integrating youth services within the Canadian context. Health Manage Forum 2019; 32: 51–55.

65.

Cohen

Londoño

Klodnick

, et al. There wasn’t any build up to it, it was just like, now you have to go over.” disengagement during the child to adult community mental health service transition. J Adol Res 2023; 38: 666–696.

66.

Ries

. Innovation in healthcare, innovation in law: does the law support interprofessional collaboration in Canadian health systems? Osgoode Hall Law J 2016; 54: 87–124.

67.

Schadewaldt

McInnes

Hiller

, et al. Views and experiences of nurse practitioners and medical practitioners with collaborative practice in primary health care – an integrative review. BMC Fam Pract 2013; 14: 1–11.

68.

Capon

Hickie

McKenna

, et al. Characterising variability in youth mental health service populations: a detailed and scalable approach using digital technology. Australas Psychiatry 2023; 31: 295–301.

69.

Gregory

Sukhera

Taylor-Gates

. Integrating smartphone technology at the time of discharge from a child and adolescent inpatient psychiatry unit. J Can Acad Child Adolesc Psychiatry 2017; 26: 45–50.

70.

Montague

Varcin

Simmons

, et al. Putting technology into youth mental health practice: young people’s perspectives. SAGE Open 2015; 5: 2158244015581019.

71.

Cifuentes

Davis

Fernald

, et al. Electronic health record challenges, workarounds, and solutions observed in practices integrating behavioral health and primary care. J Amer Board Family Med 2015; 28: S63–S72.

72.

Rothstein

Tovino

. Privacy risks of interoperable electronic health records: segmentation of sensitive information will help. J Law Med Ethics 2019; 47: 771–777.

73.

Madden

Lakoma

Rusinak

, et al. Missing clinical and behavioral health data in a large electronic health record (EHR) system. J American Med Inf Assoc 2016; 23: 1143–1149.

74.

Provan

Sebastian

. Networks within networks: service link overlap, organizational cliques, and network effectiveness. Acad Manag J 1998; 41: 453–463.

75.

Di Vincenzo

Iacopino

. Catching the new’: exploring the impact of professional networks on innovative work behavior in healthcare. Creat Innov Manag 2022; 31: 141–151.

76.

Nembhard

Morrow

Bradley

. Implementing role-changing versus time-changing innovations in health care: differences in helpfulness of staff improvement teams, management, and network for learning. Med Care Res Rev 2015; 72: 707–735.

77.

Bourghouts

Eikey

Mark

, et al. Barriers to and facilitators of user engagement with digital mental health interventions: systematic review. J Med Int Res 2021; 23: e24387.

78.

Jensen-Doss

Becker Haimes

Smith

, et al. Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Adm Policy Ment Health 2018; 45: 48–61.

79.

Cliffe

Croker

Denne

, et al. Clinicians’ use of and attitudes towards technology to provide and support interventions in child and adolescent mental health services. Child Adol Mental Health 2020; 25: 95–101.

80.

Bradford

Rickwood

. Adolescent’s preferred modes of delivery for mental health services. Child Adol Mental Health 2014; 19: 39–45.

81.

Zhao

Stadnick

Ceballos-Corro

, et al. Facilitators of and barriers to integrating digital mental health into county mental health services: qualitative interview analyses. JMIR Formative Res 2023; 7: e45718.

82.

Chong

Hickie

Cross

, et al. Digital application of clinical staging to support stratification in youth mental health services: validity and reliability study. JMIR Formative Res 2023; 7: e45161.