Abstract
BACKGROUND
The problem, condition or issue
Clinical supervision is increasingly being recognized as a core professional competency within the mental health field (Brosan, Reynolds, & Moore, 2008). Supervision is also considered an essential component of modern effective health care systems (Kadushin, 2002) and training programs for mental health professionals (Berger & Mizrahi, 2001; Milne, Sheikh, Pattison, & Wilkinson, 2011; Watkins, 2011). The terms mental health professional is used here to refer to all clinicians working in the mental health field, including psychotherapists, counsellors, social workers, psychologists, psychiatrists, nurses, pastoral counsellors and couple/marriage and family therapists as well as trainees in each of these professions. Similarly, the term mental health work refers to the clinical work for each of these professions, which includes, but is not limited to the provision of direct psychosocial interventions (i.e., non-pharmacological interventions), such as mental health therapy, couple and family counselling, psychotherapy and psychosocial interventions designed to improve, enhance, maintain socio-emotional, psychological, behavioural, interpersonal functioning and/or reduce risk factors, such as psychiatric symptoms and/or addictions. Despite the culture of evidence-based practices in mental health settings, the practice of supervision of mental health professionals lags behind in its use of evidence-informed practices (Schoenwald et al., 2009). Further exacerbating the problem is that most supervisors assume, perhaps erroneously, that the supervision they provide is effective (Kilminster & Jolly, 2000).
There are at least three clinical reasons (Milne, 2009) to develop a better understanding of supervision: one, effective supervision may be essential to enhance and maintain mental health professionals’ competencies; two, supervision can increase fidelity to evidence-based treatment models; and third, effective supervision can reduce unnecessary interventions (e.g., mental health professionals making referrals to multiple services when contraindicated or failing to disengage services when goals are achieved) and reduce waitlist times and health care costs. Although the quality of the evidence emerging from outcome research on the effectiveness of supervision is limited (Schoenwald, Sheidow, & Chapman, 2009; Waller, 2009) due to methodological issues, there is indication that supervision results in improved patient care outcomes (Bambling, King, Raue, Schweitzer, & Lambert, 2006; Bradshaw, Butterworth, & Mairs, 2007; Callahan, Almstrom, Swift, Borja, & Heath, 2009; Milne, Aylott, Fitzpatrick, & Ellis, 2008; Watkins, 2011) and that it acts as a quality assurance mechanism (Schoenwald et al., 2009). Without supervision, the quality control of mental health work depends on the ability of mental health professionals to self-evaluate their competencies (Hansen et al., 2006). Self-evaluations prove to be difficult with early career and lower skilled clinicians who are found to typically over-rate their competencies (Vallance, 2005), which can have negative implications for patient outcomes and safety.
The intervention
Clinical supervision is an ongoing supportive learning process for clinicians of all levels to develop, enhance, monitor, and, when necessary, remediate, professional functioning (Bernard & Goodyear, 2014). Supervision is a distinct professional practice with knowledge, skills, and attitudes components. For some professions (e.g. Marriage and Family Therapy), supervisors require specific training to be recognised as an “approved supervisor” while other professions promote experienced clinicians into the role of “supervisor” after some time and clinical experience (Falender, Burnes, & Ellis; 2013; Falender, Ellis, & Burnes, 2013; Bernard & Goodyear, 2014; Reiser & Milne, 2012). The chief function of supervision is to minimize non-purposeful activity and maximize intentionality with the goal of directly optimizing clinician competencies, ensuring quality control, and enhancing confidence for the end goal of improving patient outcomes (Milne, 2009). Supervision is provided in a variety of formats including one-on-one supervision, small group supervision, peer-based consultation, and facilitated team-based consultation. Supervision can include presentations via case discussion, video review or live presentation/demonstrations (Todd & Storm, 2002).
There are a number of definitions of supervision put forth by the various mental health professions. For example, the American Psychological Association (2014) defines supervision as:
… a distinct professional practice employing a collaborative relationship that has both facilitative and evaluative components, that extends over time, which has the goals of enhancing the professional competence and science-informed practice of the supervisee, monitoring the quality of services provided, protecting the public, and providing a gate keeping function for entry into the profession. (p. 5)
The Association for Counsellor Education and Supervision (2011) provide a list of guidelines in twelve areas for addressing the ethical and legal protection of the rights of supervisors, supervisees, and clients; and meeting the professional development needs of supervisees while protecting client welfare. While the Council for Accreditation of Counseling and Related Educational Programs (2009) offer a definition of supervision in an educational setting as:
A tutorial and mentoring form of instruction in which a supervisor monitors the student's activities in practicum and internship, and facilitates the associated learning and skill development experiences. The supervisor monitors and evaluates the clinical work of the student while monitoring the quality of services offered to clients. (p. 62)
Further definitions are offered by the American Association for Marriage and Family Therapy (AAMFT), the National Association of Social Work (NASW), the British Association of Counselling and Psychotherapy (BACP), and the Royal Australian and New Zealand College of Psychiatrists (RANZCP).
AAMFT (2014) describe the process of supervising marriage and family therapists (MFT) as:
…evaluating, training, and providing oversight to trainees using relational or systemic approaches for the purpose of helping them attain systemic clinical skills. Supervision is provided to an MFT or MFT trainee through live observation, face-to-face contact, or visual/audio technology-assisted means. (p. 5)
Another definition of supervision is offered from the NASW (2013). This organisation defines professional supervision as:
the relationship between supervisor and supervisee in which the responsibility and accountability for the development of competence, demeanor, and ethical practice take place. The supervisor is responsible for providing direction to the supervisee, who applies social work theory, standardized knowledge, skills, competency, and applicable ethical content in the practice setting. The supervisor and the supervisee both share responsibility for carrying out their role in this collaborative process. (p. 6)
The BACP (2016) defines supervision as a “formal arrangement for therapists to discuss their work regularly with someone external in order to maintain adequate standards of therapy”. The BACP recommends that supervisors not know the identity of clients and be members of a recognized professional body and preferably accredited. A final definition of supervision is offered by the RANZCP (2012) that outlines an apprenticeship process that includes a minimum of four hours per week for 40 weeks of supervision by a College-accredited supervisor. The RANZCP define supervisors as clinically competent in the area they are supervising in and familiar with the use of a competency-based assessment tool. Each of the definitions developed by the different professional organisations share developmental, ethical, and supportive roles while each offer different emphases based on the epistemological roots of their professions and understandings of supervisory relationships.
There are also different approaches to
One challenge is the varied understanding of supervision across the mental health professions. Psychology is the only mental health profession that explicitly adheres to a science-practitioner model (Frank, 1984) and places an emphasis on what is measurable by the senses. While each profession engages in evidence-based practices there is a richer dialogue of what constitutes evidence among the different professions as well as a critique of the narrow band of empiricism that drives the science-practitioner approach (e.g. Coulter, 2011; Elliott, 1998; Gambrill, 2010; Holmes, Murray, Perron, & Rail, 2006; Slife, Wiggins, & Graham, 2005; Staller, 2006; Wendt, Jr., 2006). To be systematic, it is important that we review research from across the mental health professions.
For the purpose of this systematic review we will include research that defines supervision as involving a supportive learning process for clinicians when carrying out psychotherapy, counselling and/or the provision of psychosocial mental health interventions of all levels to develop, enhance, monitor, and when necessary, remediate, professional functioning. We will include research where the supervision sessions may have: case-presentations, presentation of video or audio-tapes from a therapy session, exploration of self-of-the-therapist issues, or process recordings (interpersonal process recall). And the supervision can take place: one-to-one, triadic (1 supervisor and dyad), in a group format, live (with call-in and/or with bug in ear), consultation teams, reflecting teams, or online. One of the objectives of this review is to assess the research about supervision of multicultural competencies. A further clarification is required to address the understanding of multicultural competencies.
Developing multicultural competence is integral to the formation of clinical competence (Falender, Shafranske, & Falicov, 2014). Multicultural competencies in the mental health professions include a range of attitudes, beliefs, knowledge, skills, and actions (Ratts, Singh, Nassar-McMillan, Butler, & McCullough, 2015) that provide a framework to optimize client engagement, participation and benefit from psychotherapeutic intervention and research. Developmental domains of multicultural competencies include mental health professional self-awareness, the client's worldview, the therapeutic relationship, and mental health and advocacy interventions (Ratts et al., 2015). There are a number of obstacles to integrating cultural perspectives in supervision including the need to clarify the role of understanding what cultural heritage and sociopolitical context have to do with human suffering and critically examining the epistemological foundations of the psychotherapies that are used (Falicov, 2014). Multicultural competence is considered an ethical and practice imperative and there is a need to clarify the best research-based approaches to the supervision of mental health professionals in this area (Falender, Shafranske, & Falicov, 2014). The aim of this part of the systematic review is to identify the research pertaining to the supervision of multicultural competencies in the mental health therapies. This will not include training or workshops about multicultural competencies.
How the intervention might work
The logic model for this intervention outlines a series of themes as opposed to a singular intervention, as clinical supervision takes many forms and can be informed by a myriad of models ranging from behavioral to psychodynamic approaches (often mirroring mental health intervention approaches). There is, however, a shared objective for supervision across professions to minimize non-purposeful activity and maximize intentionality with the goal of directly optimizing clinician competencies, ensuring quality control, and enhancing confidence for the end goal of optimizing care (Milne, 2009). Supervisors accomplish one or more of these objectives by targeting supervisee knowledge, skills, attitudes and/or behaviours.
Educational settings responsible for supervising mental health trainees/students share the duties as outlined by Council for Accreditation of Counseling and Related Educational Programs (2009). These supervision interventions include atutorial and mentoring form of instruction in which a supervisor monitors the student's activities and facilitates the associated learning and skill development experiences. The measurement of this form of supervision happens with direct verbal feedback, grading, and feedback forms. Also important to consider when discussing supervision interventions is the clinical approach such as family or systemic therapy.
For systemically oriented supervisors and mental health professionals, the self is unavoidably a part of the therapeutic system (Cheon & Murphy, 2007; Lee & Everett, 2004). Integrating the self into the therapeutic system and supervision generates self-of-the-therapist issues that are to be explored in supervision (Aponte et al., 2009). One of the purposes of this self-exploration is to to learn how to use these emotional materials to enhance the effectiveness of the professional use-of-self (Timm & Blow, 1999). Aponte et al. (2009) report that the degree to which mental health therapists commit to exploring the challenges in their lives and engage in personal growth and development is proportionate to the ability to relate to clients’ efforts to deal with their challenges. Progress in supervision is measured by supervisees’ reflective journaling, review of video of therapy sessions, and exploration of challenges in group and individual supervision. Assessment of the supervision is through supervisee self-reports and review of clinical work by supervisors.
The intervention of supervision includes knowledge, skill, attitude, and/or behavioural components and the priority placed on each will vary across supervision approaches for reasons previously noted. Moreover, supervision of mental health professionals can vary according to the: supervisor, therapeutic approach, clinical setting, mental health worker, resources available to support supervision, access to supervisors, profession, as well as other factors. The means by which supervision is evaluated varies from experimental designs to qualitative approaches. Experimental and quasi-experimental designs typically employ quantitative measures, compared before and after supervision and with comparison groups in order to determine whether supervision is effecting improvements on the supervisee's skills and/or knowledge, as well as on clinical outcomes with the supervisee's clientele. Qualitative methods are often harnessed to help illuminate the experiences and perspectives supervisees and/or supervisors have with the supervision process as well as help identify the key facets of the supervision mechanism.
Logic model:
Why it is important to do the review
There are no systematic reviews on this topic registered with the Campbell Collaboration or the Cochrane Collaboration to date. However, a number of previous reviews have addressed clinical supervision, but have been somewhat restricted and have not looked at it from an overall mental health professional perspective. For example, Bogo and McKnight (2006) published a review of the research and non-research literature pertaining to clinical supervision within the field of social work and social work trainee field education. Due to the broad scope of practice of social workers Bogo and McKnight did not focus on the practice of mental health therapy by social workers or the supervision of this practice. A systematic review of social work supervision practices specific to the field of child welfare was conducted by Carpenter, Webb and Bostock (2013), however their scope did not include mental health therapies.
Brunero and Stein-Parbury (2008), Francke and Graaff (2012), Butterworth, Bell, Jackson, and Pajnkihar (2007), as well as Cummins (2008) also conducted reviews of the literature examining the effectiveness of clinical supervision for nurses. The focus of the reviews included a broad range of nursing interventions and included few studies pertaining to nursing supervision specific to the practice of mental health therapy. Dawson, Phillips and Leggat (2013) broadened the population of their review to include all allied health professions, however, similarly they did not focus on mental health therapy.
Faman and colleagues (2012) completed a systematic review of outcomes of supervision on patient care and medical residents’ competencies. Although this review included a targeted population for psychiatry, this study did not search for, or identify studies specific to supervision for mental health therapies. Other reviews of studies within the field of medicine, such as those by Kilminster and Jolly (2000) identified key mechanisms of supervision leading to positive outcomes including the nature of the supervisory relationship. Other reviews looking at allied health professionals (Barak, Travis, Pyun, & Xie, 2009) focused on positive outcomes for supervision arising through the supervision process including enhanced job satisfaction and reduced burnout. Follow-up reviews are needed as these reviews did not specifically examine supervision pertaining to mental health therapy. A review conducted by Milne and James (2000) did focus on supervision for mental health therapy, however, mental health therapy was narrowly defined as cognitive behavioural therapy (CBT).
This review is important for a number of reasons. First, little is known about what evidence exists for supporting the use of supervision or what supervision practices may be help/harmful for supervisees, which ultimately impacts the level of care received by patients accessing mental health services (Wheeler & Richards, 2007). Second, little is known about the evidence for multicultural and/or culturally sensitive supervision practices and how these may impact supervisee competence and patient care (Falendar et al., 2014). Third, there is indication in the research literature that competent supervision results in improved patient care outcomes and that it acts as a quality assurance mechanism (Tracey, Wampold, Lichtenberg, & Goodyear, 2014; Watkins, 2012). Without supervision, the quality control of mental health interventions depends on the ability of mental health therapists to self-evaluate their competencies. Self-evaluations prove to be difficult with early career and lower skilled therapists who are found to typically over-rate their competencies, which can have negative implications for patient outcomes and safety. Finally, supervision is considered an essential component of modern effective health care systems and health care training programs in general (Kilminster & Jolly, 2000). This systematic review will contribute to enhancing our knowledge of effective supervision practices, including the impact of multicultural supervision, which will lead to improved care, better training, and better management of care.
OBJECTIVES
The primary objectives of this review are to summarize and synthesizes the available evidence on the effects of supervision for enhancing the clinical and multicultural competencies of mental health professionals. The specific research questions guiding our review include: What effect does clinical supervision (versus peer consultation or no supervision) have on mental health professionals’ clinical competencies (i.e., knowledge, skill, attitude & clinical behaviour) in the practice of mental health therapy? What effect does clinical supervision (versus peer consultation or no supervision) have on mental health professionals’ multicultural competencies (i.e., knowledge, skill, attitudes and clinical behaviour) in the practice of mental health therapy? What effect does clinical supervision (versus peer consultation or no supervision) have on the health outcomes of patients/clients participating in supervised mental health therapy?
The secondary objectives of this review include:
Identifying the key experiences of supervisees and/or supervisors with the supervision process that help identify any reasons why clinical supervision may succeed or fail in improving clinical and/or multicultural competencies in mental health professionals and/or succeed or fail in improving health outcomes for patient/clients participating in mental health therapy.
METHODS
Criteria for including and excluding studies
Types of study designs
This review will include experimental (i.e., Randomized controlled trials: RCTs) and parallel cohort quasi-experimental designs (QED) if the treatment and control groups were assessed at the same points in time and matching or statistical controls have been employed to ensure no significant baseline differences between these groups in relation to key constructs (i.e., supervisee competence in mental health therapy/interventions, multicultural competency, nature of the supervisory relationship). Studies will be included that have a control group that include either peer-based consultation or no supervision condition. We will include QEDs due to the likelihood that most evaluations of supervision for mental health professionals will be conducted in real world, naturalistic settings that reduce the feasibility of RCTs. All studies formally included in the review must evaluate the effectiveness of clinical supervision on at least one of the outcomes listed in the subsequent section.
In relation to the secondary objective of this review we will also review non-experimental studies. In our preliminary review of the state of the research landscape pertaining to clinical supervision, qualitative studies reveal that many studies relevant to our focus employ diverse methodologies (i.e., single-subject designs, cohort/longitudinal studies, case control and qualitative studies). Therefore, in order to gain insight into reasons why supervision may succeed or fail to enhance clinical and multicultural competencies of mental health professionals and/or health outcomes of patients, we will report findings from these studies in a tabular format and narrative summary that will provide a fuller context to our findings regarding our analysis of the RCTs and QEDs formally included in our review. Similarly, findings from relevant previously conducted review articles and/or meta-analyses will not be formally included in our review or analyses, but will be reported in the form of a narrative summary within our review. Review articles and meta-analyses identified in our literature search will also serve as a means of identifying any additional studies not previously identified within our search.
Types of participants
The participants to be included in this review must be in a professional, structured supervisory relationship either as a supervisee or supervisor, or as a student in practicum, internship, or residency, whereby the supervisor is identified as having a formal role and responsibility for the supervision process. This supervision must also be in relation to the provision of mental health therapy. Peer supervision (i.e., supervision provided without a formal mandate/scope of practice between two or more individuals practicing in a same or similar role, typically that of psychotherapist) will only be included in this review as a control condition. Types of mental health professionals included in this review are psychotherapists, psychologists, counsellors, couple/marital and family therapists, social workers, psychiatrists, nurses, pastoral counsellors, and trainees in each of these professions.
Types of interventions
This review will include studies examining clearly specified clinical supervision models intended to minimize non-purposeful activity and maximize intentionality with the goal of directly optimizing mental health therapists’ competencies (i.e., knowledge, skill, attitude &clinical behaviour) and client outcomes, ensuring quality control, and/or enhancing confidence for the end goal of optimizing care. The intervention may be delivered to a range of mental health therapists in a variety of formats including: one-to-one, triadic (1 supervisor and supervisee dyad), in a group format, live (with call-in and/or with bug in ear), consultation teams, reflecting teams, or online.
Supervision will be demarcated from theoretical-based training in models of mental health therapy. Research will be included that examines supervision in relation to a supervisee's application of mental health therapy to a specific patient/client(s) situation and/or the supervisory discourse will pertain to a unique patient/client situation. Research will be excluded that is not directly focused on investigating the supervision process as it pertains to the needs of unique patients/clients, but instead focus only on the training of individuals related to the acquisition of a theoretical understanding of a mental health model and/or the acquisition of general knowledge and/or general skills related to mental health therapy without the application of this model to specific patient(s)/client(s) situations. Research studies examining competency development using simulated-patients/clients will be included in the present review, but will be treated as a distinct sub-group for analytic purposes.
Types of outcome measures
The primary outcome(s) for this study will include the measurement of effectiveness of competencies (i.e., knowledge, skills, attitudes and clinical behaviours) in the provision of mental health therapy, including multicultural competencies. We are also interested in outcomes pertaining to the effectiveness of the supervisory relationship/process and client outcomes. Examples of these measures are included below. Additional relevant and validated measures identified through the review of study will also be considered.
Mental health professional's competencies (i.e., knowledge, skill, attitude, and clinical behaviours) in mental health therapy outcomes: Licensing exam scores Objective Structured Clinical Examination (OSCE) (Harden, Stevenson, Downie, & Wilson, 1975) Measures of fidelity to mental health therapy interventions, such as the Therapist Behavior Rating Scale (Hogue et al., 2008) or the Cognitive Therapy for Psychosis Adherence Scale (Startup, Jackson, & Pearce, 2002). Supervisor reports and evaluations
Mental health professional's multicultural competencies (i.e., knowledge, skill, attitude, and clinical behaviours) in mental health therapy outcomes: Cultural Awareness Tool (Multicultural Mental Health Australia, 2002) Cultural Competence Self-Assessment Questionnaire (CCSAQ; Mason, 1995) Multicultural Awareness Knowledge/Skill Survey (MAKSS; D'Andrea, Daniels, & Heck, 1990). Multicultural Counseling Inventory (MCI; Sodowsky, Taffe, Gutkin, & Wise, 1994). Multicultural Counseling Knowledge and Awareness Scale (MCKAS; Ponterotto, Gretchen, Utsey, Rieger, & Austin, 2002). Client, Supervisor, Psychotherapists Reports Sexual Orientation Counselor Competency Scale (SOCCS; Bidell, 2005)
Supervision outcomes/outcomes pertaining to supervisory relationship: Leeds Alliance in Supervision Scale (Wainwright, 2010) The Manchester Clinical Supervision Scale (MCSS; Winstanley, 2000) Short form Supervision Satisfaction Questionnaire (SFSSQ; Carlton, 2002).
We will also examine the effectiveness of supervision for mental health therapy related to patient/clinical care outcomes. Possible measures to be included are listed below. Additional relevant measures identified through the review of study will also be considered.
Individual Client Functioning Outcomes: Global Assessment of Functioning (GAF; American Psychiatric Association, 2000), Beck Depression Inventory (BDI; Beck, Steer, & Brown, 1996), Beck Anxiety Inventory (BAI; Beck & Steer, 1990), Child Behavior Checklist (ASEBA; Achenbach, 2001 & 2001), Behavior Assessment System for Children (BASC; Reynolds &Kamphaus, 1992), Parent Stress Index (PSI-4; Abidin, 2012), State-Trait Anxiety Inventory (STAI; Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, 1983), Outcome Rating Scale (ORS; Miller &Duncan, 2000), Patient Health Questionnaire, Clinically Useful Depression Outcome Scale (CUDOS; Zimmerman, Chelminski, McGlinchey, & Posternak, 2008), Centre for Epidemiological Studies Depression Scale (CESD; Radloff, 1977) and Stages of Change Readiness and Treatment Eagerness Scale (SOCRATES; Miller &Tonigan, 1996).
Couple/Marital Functioning Outcomes: Dyadic Adjustment Scale (DAS; Spanier, 1989), Relationships Assessment Scale (RAS; Hendrick, 1988), Experiences of Close Relationship (Revised) (ECR-R; Fraley, Waller, & Brennan, 2000), Locke-Wallace Marital Adjustment Test (MAT; Locke & Wallace, 1959), The Quality of Marriage Index (QMI; Norton, 1983), Golombok-Rust Inventory of Marital State (GRIMS; Rust, Bennun, Crowe, & Golombok, 1986), Relationship Dynamics Scale, The Couples Satisfaction Index (CSI; Funk & Rogge, 2007).
Family Functioning Outcomes: Family Quality of Life Scale (FQOL; Beach Center on Disabilities, 2006), Family Assessment Device (FAD; Epstein, Baldwin, & Bishop, 1983), Family Beliefs Inventory (FBI; Roehling & Robin, 1986), Family Adaptability and Cohesion Evaluation Scale, Kansas Family Life Satisfaction Scale, Self-Report Family Instrument.
Adverse events: Decrease in competency of the supervisee including reports of ethical and/or competency violations to the various mental health regulator/licensing bodies Deterioration of the supervisor-supervisee relationship including reluctance to seek-out/engage in supervision. Decrease in client/patient outcomes and/or deterioration in client/patient health outcomes.
Duration of follow-up
The duration of follow up may vary due to the nature of the different types of supervision provided, and the limitations of the studies. A portion of the studies are likely to include supervision that is time limited, such as graduate program internship, practicum and/or residency supervision. Others may include clinical and professional supervision that are likely an on-going experience throughout a helping professional's career. The follow up duration for this study will be inclusive of all of these supervisory experiences. There will be no restriction placed on the inclusion of articles based on the duration of the intervention and/or the duration of the follow-up measurement pertaining to study outcomes.
Types of settings
This proposed review will include studies of supervision conducted in settings that employ or train mental health professionals engaged in the practice of mental health therapy. Settings will include universities/colleges, child welfare services, psychiatry institutions, training programs, web-based programs, hospitals, counselling agencies, and physicians’ offices. The review will include the following experiences: residencies/internships/practicums, student supervision, and clinical supervision across mental health professions. We will exclude studies of educational curriculum of supervision and workshops.
Search methods for identification of studies
We will perform electronic searches of bibliographic databases, as well as on open web-sites and in the grey literature. We will also search for on-going studies. We will have no publication date, geographic, or language restrictions. Media and software reviews will not be included. The following sources will be searched:
Ovid PsycINFO
Ovid MEDLINE
CINAHL
Social Work abstracts
EMBASE (Embase.com)
Web of Science
Social Services abstracts
Educational Resources Information Centre (ERIC)
PubMed
EBSCO ACADEMIC
JSTOR
SAGE
Science Direct
Springer
Wiley
Applied Social Sciences Index and Abstracts (ASSIA)
Cochrane Library
Campbell Library
Google Scholar
Theses and Dissertations (Proquest)
Description of methods used in primary research
The search strategy is comprised of 3 main concept categories: supervision, profession, methodology, and for the objective of addressing multicultural competencies we will add a 4th category, culture. Within each category there are textword (“free text”) terms and, where appropriate, subject headings in indexed databases. We will approach the search from broad-to-narrow in order to capture general studies on clinical supervision, while narrowing the broad search to focus on studies related to multicultural competencies.
The following is one example of the search strategy in OVID PsycINFO:
A hand search will also be conducted on key journals within the last year to ensure early on-line publications not yet indexed are identified related to supervision and psychotherapy including: The Clinical Supervisor, Counselor Education and Supervision, Social Work Supervision, and Journal of Therapy, Consultation and Training. A search of the online table of contents of each of these journals over the last two-year period will be conducted in the event that relevant articles have not been indexed into searched databases. Hopewell et al. (2002) noted that utilising a hand search can identify research that can be missed or not indexed ensuring the broadest inclusion possible.
The review authors will check reference lists from key articles, literature review and systematic reviews in order to identify additional references. Furthermore, leading authors in the field will be contacted requesting that they share their reference lists, and RefWorks or EndNote bibliographic management files relevant to the study's objectives. These reference materials will be examined to identify any possible studies not previously identified through our employed search strategies.
Google Scholar is a large and valuable database but searches here are difficult to track. Different from our approach with more structured databases, we therefore plan to run and record individual terms and their combinations. Since Google Scholar ranks the records by relevancy and by citation counts (Google, n.d.), we will limit inspection of citations to the top 10 pages of retrieval which may vary depending on discovery of relevant citations. For example, the search strategy, professional supervision AND family therapy AND multicultural, results in a total of over 33,000 hits out of which the first 10 pages includes about 100 of the most relevant records. Beyond these pages, the relevancy of the records diminishes significantly.
Finally, records retrieved from each resource will be organized using the Endnote bibliographic management application.
Data collection and analysis
Selection of studies
Two members of the review team (RA and AM) will independently review titles and abstracts to exclude studies that are clearly irrelevant to the current reviews objectives. We will retrieve the full text reports for those that are not obviously ineligible and for those in which reviewers disagreed. Should disagreements persist a third member of the review team (LH) will be consulted to resolve the disagreement.
In the second phase the two team members (RA and AM) will review the full text versions of the articles and determine eligibility based on inclusion criteria. Any disagreements regarding exclusion/inclusion will again be resolved via consultation with the third team member (LH). The rationale for exclusion will be documented for each study that is retrieved in full text. All studies will be uploaded to Endnote, which will be employed as our citation management tool. The overall search and screening process will be presented in a PRISMA diagram (Moher et al., 2009).
Data extraction and management
Two members of the review team (RA and AM) will independently code and extract pertinent information from each included study. A piloted extraction tool will be employed to guide the extraction process (see Appendix A for the codebook). Using this codebook in relation to RCTs, QEDs and qualitative research data will be extracted on the characteristics of the participants (i.e., age, gender, professional discipline), experience level of the supervisee, experience level of the supervisor), format/nature of the supervision (e.g., one-on-one, group, video review, live supervision), model/approach of mental health therapy (e.g., CBT, systemic therapy, psychodynamic therapy), characteristics of the study (i.e., publication date, geographic location of the study, sample size, intervention setting,) and characteristics of the patients/clients regarding objectives of the study pertaining to patient/client health outcomes (i.e., age, gender, culture, presenting concern, outcome of treatment). Additional information will be extracted for RCTs and QEDs related to intervention and control conditions, measures and results. For qualitative studies additional materials will be included on the code sheet, including the specific methodology employed in the study (i.e., phenomenology, grounded theory, ethnography, case study), along with a verbatim samples that illustrate the key themes from the findings sections that illuminate factors that contribute to the success or failure of supervision. Extracted information will be stored electronically within an Excel spreadsheet. Coding disagreements will be settled via discussion and inclusion of an independent third party (LH). Management and analysis of the extracted data will be conducted in RevMan5, NVivo11, and SPSS 21.
To achieve the primary objective of this review we will include studies that utilize experimental or quasi-experimental research designs. One example of a randomized controlled trial (RCT) conducted by Weck and colleagues (2015) who studied the effectiveness of supervision methods for master's level mental health professionals (n=46) training in CBT. Specifically, this study compared live supervision using real-time computer-messaging between supervisor (i.e., treatment condition) and supervisee versus post-intervention video-based review (i.e., control). Employing third party reviewers blinded to the conditions, Weck and colleagues concluded that the therapeutic alliance and trainee competence were significant greater for the computer enhanced live supervision. However, patient outcomes following the mental health intervention as measured using the Beck Depression Inventory-II (Beck & Steer, 1996) and the Brief Symptom Inventory (Derogatis&Melisaratos, 1983) were shown to have no significant difference between the intervention and control conditions. Another RCT likely eligible for inclusion in the proposed review was conducted by Kass and colleagues (1984). This study examined the impact of supervision methods on psychiatry residents’ (n=83) competence for mental health interventions and the impact on patient outcomes from these interventions. In this three arm RCT, supervision with a senior psychiatrist was compared to peer-based consultation among residents and with a no-supervision/consultation condition. Kass and colleagues found that psychiatry residents’ self-reports indicated both supervision with a senior psychiatry resident and peer-supervision to be a positive educational experience. However, patient outcomes measured using resident-completed Likert-based surveys implemented post-intervention revealed that residents who received supervision from a senior psychiatrist rated patient outcomes more positively than those participating in peer-based consultation, which in turn was rated higher than no supervision/consultation processes.
In order to achieve the secondary objectives of this review we will include qualitative studies in order to gain insight into reasons why supervision may succeed or fail to enhance clinical and multicultural competencies of mental health professionals and/or health outcomes of patients.
Multiple time points for measurement
We anticipate that most eligible studies will include both pre-test and posttest measurement. Pretest will likely occur prior to supervision commencing while we anticipate that some posttest measurement to occur following the completion of block of supervision associated with a training milestone, such as the completion of an educational rotation/practicum or following a training regimen in a particular model of therapy (e.g., CBT). Due to the idiosyncratic nature of ongoing supervision unrelated to new trainees and/or a specific training regimen, we anticipate that studies will likely conduct posttest measurements and measurement at various follow-up time points. Therefore, they will be analysed in the following groups: post-test (within 2 weeks following the completion of supervision); medium term follow-up (3-6months following the completion of supervision) and long term follow-up (6+ months following the completion of supervision). Posttest measures will be considered distinct from follow-up measures, as their purpose is to determine the immediate effects of supervision versus follow-up conditions that are intended to measure the lasting impact of supervision. These groups (i.e., pretest, posttest and follow-up) will be considered distinct and will not be pooled for analysis.
Assessment of risk of bias in included studies
In relation to the first objective we will assess for potential risk of bias present and/or the quality of the RCTs and QEDs in our review using the Cochrane Risk of Bias Assessment Tool. Two members of the review team (RA and AM) will independently assess for risk of bias (i.e., high, low, unclear) by exploring each RCT and QED in relation to the following conditions/factors as outlined by Higgins and Green (2011) and Mathews and colleagues (2015):
For RCTs Random sequence generation: Is the method employed to generate the random sequence described with sufficient detail to allow an assessment of the adequacy of the randomization of the sequence process.
For QEDs Sequence generation: Is the method employed to generate the allocation sequence described with sufficient detail to allow an assessment of the likelihood that a comparable baseline could be produced following a comparable process.
For RCTs and QEDs Allocation Concealment: Is the concealment method for allocation (i.e., treatment vs: control) described with adequate detail to allow reviewers to determine whether group allocation could have been predicted/determined by participants involved in the individual studies. Blinding of study participants & researchers: Are participants/researchers blinded to participant's group allocation/membership. Blinding of outcome assessment: Are outcome assessors (i.e., third party evaluators of competency using the OSCE) aware of the participants’ group status. Incomplete outcome data: A rationale is provided for incomplete data (e.g., attrition, exclusions and withdraws, number-needed-to-treat analysis) and, where applicable imputation methods were articulated. Selective Reporting: Outcomes were reported a priori with sufficient details to allow reviewers to adjudicate their completeness. Our assessment for potential reporting bias will be further commented on in a subsection section of the current protocol. Other Biases: Are other possible biases present? For example, is there a conflict of interest bias, where the researcher also has a possible fiduciary connection to the training/supervision model or psychosocial intervention?
In relation to the second objective of the review we will employ the Critical Appraisal Skills Programme (CASP) checklist pertaining to qualitative research to guide our inclusion of qualitative research studies. For the purpose of this study, qualitative studies will be rated as low when the study fails to receive an affirmative (i.e., Yes) response to all of the following a) the appropriateness of the research design to answers the study's question(s), b) the appropriateness of sampling strategy to achieve the research aims and c) were the analyses suitably rigorous (see Appendix A). Those studies rated as low on the CASP Qualitative research checklist will not be included into the final tables or narrative summary.
Measures of treatment effect
Continuous data/Outcomes
Continuous data are likely to arise from studies employing measures such as the Child Behavior Checklist, Behavior Assessment System for Children, Parent Stress Index and Outcome Rating Scale. Due to a high likelihood that included studies will employ a diverse array of outcome measures, we will report effects size differences as standardised mean differences (SMD). For studies of relatively small sample sizes, we will use a Hedge's ĝ corrected SMD. Principal investigators will be contacted by the reviewers to obtain means and standard deviations when this information is not provided. Calculation of 95% confidence intervals for the SMD will be calculated from available data (i.e., F-ratios and t-values) in order to facilitate a comparison of the effects of the different treatments (see Lipsey & Wilson, 2001).
Dichotomous data/Outcomes
For dichotomous data we will calculate risk ratio. For primary measures, such as ratings of the OSCE, outcomes can be pass/fail in relation to competency and therefore we would report on the risk of failing. In relation to the study's objectives pertaining to client/patient health outcomes, several client/patient outcome measures will produce scores separating normative levels from clinically significant levels. This will allow for a risk ratio to be expressed articulating the risk of patients/clients demonstrating clinical significant levels concern versus normal/subclinical levels of concern.
Studies with multiple interventions
Where multiple intervention groups occur within a study with a single comparison condition, such as the case where one method of supervision is compared head-to head with a second method (e.g., one-on-one supervision versus group supervision), and then to a control condition, such as no supervision, we will include for analysis only that intervention condition deemed most relevant to the primary objective (i.e., mental health competence or culture competency). The determination of which intervention condition is most relevant will be made by the team during the data-extraction coding phase. Should both intervention conditions be deemed relevant to the review's objectives, the intervention coded for inclusion in pooling will be that which is deemed to have the least potential for bias (i.e., low risk of bias).
Unit of analysis issues
We will be aware of and take steps to control for factors that arise from differing units of analysis. Challenges associated with units of analysis may arise as a result of the different levels of analyses associated with the different levels of supervision practice. For example, one-to-one supervision will produce individual level data, while group supervision may produce outcomes that speak to group-level processes (i.e., team cohesion). Furthermore, in relation to the possible impact of supervised mental health practice/psychotherapy on client/patient health outcomes, it is likely that some studies will move beyond the health comes of individuals to examine the well-being of parent-child dyads and/or family systems. These various levels of analyses will not be pooled, with a separate meta-analytic procedure being conducted for each level, should there be sufficient studies.
Dealing with missing data
We will assess for missing data and attrition in our review. In the event of missing or incomplete data, the reviewers will endeavour to obtain missing information from the principle investigators (PIs) of studies containing missing information. Should authors provide any missing data, this will be imputed such that all studies undergoing imputation will be documented and a sensitivity analysis will be conducted to determine any effects in these studies compared to studies without imputation.
Assessment of heterogeneity
The reviewers will examine variables related to population (e.g., professional discipline of trainee), and intervention, including medium of supervision (e.g., individual or group-based supervision) and method of supervision (e.g., case consultation, video review or live supervision) in order to determine if variation in study distributions exist. We will assess for study heterogeneity using the the Q-statistic and its related p value to determine diversity of intervention effects; Tau2 with confidence interval to determine the magnitude of variation between studies; and the I2 statistic with confidence intervals to estimate what proportion of variability in effect estimates may be due to chance (Higgins & Green, 2011).
Assessment of reporting biases
We will assess for reporting bias by employing funnel plots if a sufficient number of studies are identified, suggested by Higgins and Green (2011) to be at least 10, in order to examine whether symmetry in the relationship between effect estimates and standard error exists. In the event that visual inspection reveals an asymmetrical funnel plot (i.e., effect size clustering on one side of the funnel), suggesting possible publication bias, we will proceed to utilize a trim and fill method suggested by Sutton and colleagues (2000).
Data synthesis
In relation to our first objective we will employ RevMan5 to support our data synthesis and meta-analytic procedures. ES will be entered into RevMan5 to calculate the standardized effects size, as well as standard error. For continuous outcomes we will use Hedge's ĝ as the measure of effect sizes (ES). For binary outcomes we will convert log odds to Cohen's d then into Hedge's ĝ. In order to calculate any meta-analysis we will use the Hedge's ĝ.. f any additional conversions as required.
We anticipate that the studies included in our review will use a diverse array of outcome measures, supervision methods, mediums and models, as well as include a wide range of professional backgrounds and experience levels for the trainees. Therefore, due to these likely sources of heterogeneity across studies we will employ a random effects model for pooling results. We will examine whether heterogeneity exists across the RCTs and QEDs studies and combine only those studies that are homogeneous. Separate analyses will also be conducted for general competencies in the provision of mental health therapy, distinct from cultural competencies. Conceptually different studies, such as those examining only increases in knowledge, distinct from a clinical skills will be pooled separately with distinct analyses being conducted. Where feasible and appropriate we will also pool the results pertaining to patient/client outcomes for those involved in supervised mental health interventions. As previously discussed, separate meta-analyses will also be conducted for each distinct unit of analysis (i.e., individual versus group/family level outcomes). The results of each outcome will be reported using Forest plots with 95% confidence intervals.
Summary of Findings Tables
In relation to our primary objectives our work will be presented in a summary table format that describes the key findings from the included studies and adjudicates the quality of the evidence according to the GRADE classification system (see Guyatt et al., 2011; Guyatt et al., 2008). The GRADE systems will guide our classification of included studies according to their relative rigour into high, moderate, lower and very low quality. Specifically, GRADE criteria will be applied by examining each study's methodology/research design, risk of bias, imprecision, inconsistency, indirectness and magnitude of effect (see Guyatt et al., 2011).
In relation to the review's second objective we will present a separate table summarizing the findings from the meta-synthesis of qualitative research. This table will present a taxonomy of themes that emerged across like-qualitative methods, including a presentation of key differences and similarities in sampling, analysis and findings from the various qualitative methods (i.e., phenomenology, ethnography, and grounded theory). In relation to the findings from the studies, differences and similarities of themes, including meta-themes (themes that run through studies of similar method) will be presenting including, where possible key quotes from the various studies will be used to illuminate the key theme being described.
Subgroup analysis and investigation of heterogeneity
There are likely meaningful subgroups within the population under investigation in the current review, making it important to estimate the effects for these groups. Although there are number of potential moderators, two in particular may be most salient including the medium of supervision (i.e., one-one-one, triadic and small group) and the supervisee training level, which may best be divided into those who have achieved their professional licensure/registration and those who are still students or residents and are not yet expected to conduct independent (i.e., unsupervised) practice as mental health professionals. Important subgroups may include studies specifically pertaining to cultural competencies and those pertaining to general clinical competencies. Findings from these moderators and subgroup analyses will be presented in tabular format including the pooled estimates of the percentage point impacts of different supervision mediums (i.e., individual, triadic and small group) by supervisee training level. Should there be sufficient numbers of studies, suggested by (Higgins & Green, 2006) to be at least 10 studies per grouping variable, there is impetus to conduct a moderator analysis on these two salient subgroups and on the influence of supervision method (e.g., case consultation, tape review, live supervision) as well as the duration of the supervision. This analysis will be conducted using a meta-regression as a direct test of the differences between subgroups and moderator influences on the mean effect (Littell, Corcoran & Pillai, 2008).
Sensitivity analysis
A sensitivity analysis will be conducted in order to test the soundness of decisions made throughout the current review. Specifically we will compare our models through re-analyses to determine whether studies with a) missing data and/or use of imputation, b) lack an intent-to-treat analysis, c) lack rigorous controls for assignment at baseline (i.e., methodologically weaker QEDs) and d) studies with high-risk of bias may impact the overall results of the analysis. These studies will systematically be dropped and not included in the re-analysis, allowing the authors to gain insight into whether potential methodological issues and/or bias is possibly influencing the results (Littell, Corcoran, & Pillai, 2008). In such cases these differential findings will be reported.
Treatment of qualitative research
Qualitative research will be included in this review in relation to the second objective and will be informed by Sani and Shlonsky's (2012), interpretative approach to meta-synthesis for qualitative research. Following our screening process, included qualitative studies will be imported into Nvivo, with links to full-text articles. For analytic purposes, included studies will be grouped separately by specific methodology (e.g., phenomenological studies, ethnographies, grounded theory, case study) to allow us to complete a separate synthesis for each specific qualitative methodology. Our goal in using interpretative meta-synthesis will be to integrate the materials from conceptually congruent methodologies in order to identify taxonomies or central themes throughout the studies in each methodology, with particular emphasis on synthesize themes, to best illuminate reasons that clinical supervision may succeed or fail in improving clinical and/or multi-cultural competencies and/or enhanced patient outcomes. We will engage in an iterative process of developing and applying a coding framework through the analysis of the full-text articles, moving systematically through the coding of studies of like-methodology. Harnessing a constant comparative approach we will apply the coding framework to each subsequent article, allowing new codes to emerge from the qualitative findings while also ensuring that we revisit previously reviewed articles to examine the presence of any newly identified themes. After all studies from one method have been successfully coded, coding will commence on the next methods until complete. Coding will be done my two team members with expertise in qualitative research and the application of Nvivo for thematic coding. Results of themes will be presented in a summary table with narratives from each method, as well as a grand narrative related to divergent and convergent themes across the various methodologies. We will utilize this meta-synthesis as an enhancement model (Saini &Shlonsky, 2012) that will help us to interpret the findings from the quantitative meta-analytic findings conducted on RCT and QEDs. Qualitative methods will be valuable to understand the key mechanisms underpinning supervisory interventions that contribute to supervision being effective or that enhance psychotherapists’ competency and improve patient outcomes.
Footnotes
REVIEW AUTHORS
ROLES AND RESPONSIBLIITIES
Please give brief description of content and methodological expertise within the review team. The recommended optimal review team composition includes at least one person on the review team who has content expertise, at least one person who has methodological expertise and at least one person who has statistical expertise. It is also recommended to have one person with information retrieval expertise.
Who is responsible for the below areas? Please list their names: Content: Robert Allan, Alan McLuckie Systematic review methods: Lillian Hoffecker working with all authors Statistical analysis: Alan McLuckie Information retrieval: Lillian Hoffecker working with all authors
Sources of Support
University support for research assistants, no external funding.
DECLARATIONS OF INTEREST
None known.
PRELIMINARY TIMEFRAME
| Activities | Date of completion |
|---|---|
| Searches for studies | October 31, 2017 |
| Pilot testing of inclusion criteria | November 30, 2017 |
| Pilot testing of study codes and data collection | January 31, 2018 |
| Extraction of data from research reports | March 31, 2018 |
| Statistical Analysis | May 31, 2018 |
| Preparation of report | October 31, 2018 |
PLANS FOR UPDATING THE REVIEW
Robert Allan, Alan McLuckie, and Lillian Hoffecker will be responsible for updating this review. We anticipate that this review will be updated as we accumulate new evidence and/or at least every three years.