PROTOCOL: The Therapeutic Alliance and Psychotherapy Outcomes for Young Adults Aged 18 to 34: Protocol for a Systematic Review

3.1.1 Types of studies

This review will include observational, cohort studies of the therapeutic alliance and psychotherapy outcomes for young adults in face-to-face psychotherapy with mental health professionals. Eligible studies must include at least one measure of the therapeutic alliance that preceded at least one outcome measure. This is necessary to establish time order (the alliance precedes outcome), one of several conditions that need to be met to support causal inferences. Cross-sectional studies in which all alliance and outcome measures were obtained at the same point in time will be excluded. Studies that attempt to manipulate or influence the alliance itself (studies in which the alliance is an outcome) will be included. Since the quality of the relationship between therapist and client is not something that can be assigned at random, randomized controlled trials, controlled before-and-after designs, and interrupted time series studies are irrelevant. Qualitative studies will be excluded.

3.1.2 Types of participants

Study participants must be young adults (ages 18–34). Studies that include wider age groups will be included only if they report alliance-outcome associations specifically for young adults as defined here. There will be no exclusions based on diagnosis, treatment modality, or severity of illness as we know of no evidence that indicates that any of these factors necessarily negate the formation of an alliance. This means we will include a wide range of diagnoses such as depression, anxiety, personality disorders, and substance abuse disorders.

3.1.3 Types of interventions

We will include all bona fide psychotherapies delivered to individuals in face-to-face interactions with mental health professionals or trainees. Interventions must be performed by mental health professionals or those in psychotherapy training programs supervised by Master's level or doctoral level mental health professionals. Activities outside of formal mental health services (such as guidance counselling, mentoring, peer counselling) are excluded. Interventions delivered by paraprofessionals are also excluded, because mental health professions require licensure or supervision of students in training programs leading to licensure. Therapy must be provided to individuals in face-to-face sessions. We will exclude studies in which therapy was performed using online technologies, because the role of the therapist is often unclear in these interventions. We will exclude couples, family, and group therapies because alliances are much more complex and difficult to measure in these contexts than in individual treatment. All types of bona fide psychotherapies are included, since the alliance may be central to all. Examples of bona fide therapies include Psychodynamic, Emotion-Focused Therapy, Gestalt, Cognitive Behavioural Therapy, and Behavioural Therapy. Any duration of psychotherapy is acceptable since some studies show that psychotherapy can be effective in as little as one session (Borsari & Carey, 2000; Raue, Castonguay, & Goldfried, 1993).

3.1.4 Types of outcomes

The primary outcomes in this review are ratings of mood disorders such as depressive symptoms (measured, for example, by the Beck Depression Inventory; Beck et al., 1961 or the Hamilton Rating Scale for Depression; Hamilton, 1960), psychiatric symptoms (measured, for example, by the Symptom Checklist-90-R; Derogatis, 1996), global functioning (measured, for example, by the Global Assessment of Functioning (GAS; Endicott et al., 1976) or assessments of change), specific outcomes (e.g., target complaints or estimations of drug use), and treatment participation or termination status (e.g., whether the client attended the agreed upon number of sessions or terminated early) (Martin et al., 2000). Studies may include multiple measures to evaluate outcomes in more than one category of symptoms or functioning. We will only include outcome measures for which there is published evidence of reliability and validity, and we will determine whether this information was obtained in a sample comparable to the one in the included study. This requirement will result in the exclusion of purely clinical assessments (or opinions) with no known reliability or validity, as these could introduce additional bias or error into our analysis.

This review has no secondary outcomes.

3.1.5 Types of data

Alliance measures are primarily reported in the format of continuous scales. Psychotherapy outcome measures can be continuous (e.g., degree of symptom reduction, days of hospitalization) or dichotomous (e.g., any hospitalization, relapse). We expect that most of the data in this review will be in the form of correlations between alliance and outcome measures. These associations are usually expressed in terms of the Pearson product-moment correlation coefficient (r) and similar metrics. If we find studies that controlled for other variables in analyses of associations between alliance and outcome measures, we will explore options for using partial correlation coefficients or regression coefficients in our analysis.

Data may have been obtained from direct observation, interviews, self-reports, and/or clinical records. As indicated above, over 30 alliance measures have been used to measure the alliance; however, we will only include studies that used alliance and outcome measures that have some published empirical evidence of reliability and validity in the sample studied or within a comparable sample. Elvins and Green (2007), for example, produced a comprehensive description of 32 leading alliance measures and related theoretical frameworks. For each measure, the authors listed any published evidence of reliability or validity (e.g., concurrent or criterion validity, internal consistency assessed with Chronbach's alpha, inter-rater reliability, etc.), the sources of this information, and comments about the relative strengths and weaknesses of each measure. Multiple compendiums, such as the Mental Measurements Yearbook (and others), list similar information for published outcome measures.

3.2.1 Electronic searches

The following databases will be searched:

AMED (Allied and Complementary Medicine)

Campbell Library

CINAHL

The Cochrane Library

Dissertation Abstracts International

EMBASE

ERIC

LILACS

MEDLINE

OpenGrey

ProQuest

PsycINFO

Sociological Abstracts

Social Services Abstracts

SveMed+

Web of Science®

3.2.2 Search terms

3.2.3 Searching other resources

As detailed below, we will conduct hand-searches, grey literature searches, website searches, and reference harvesting and branching (searching references of known studies to find additional material). We will contact authors of included studies and experts in the field to request copies of potentially eligible studies that are unpublished.

3.3.1 Description of methods used in component studies

The studies in the therapeutic alliance literature are observational, but vary according to research design, treatments, perspective of the person rating the therapeutic alliance (client, therapist, and/or observer), alliance measure, and outcome measures used. One study that meets our inclusion criteria is Baldwin, Wampold and Imel (2007) based on the sample's age group (18–34), types of interventions (performed by licensed mental health therapists), alliance and outcome measures with reported reliability and validity, timing of measures (at least one alliance measure before outcome measure), and type of reported data (Pearson's correlation). This study evaluated the relationship between the therapeutic alliance and psychotherapy outcome using patient outcomes from the database of the Research Consortium of Counseling and Psychological Services in Higher Education (USA). The authors analyzed data from 331 clients seen by 80 therapists. This sample only included patients who had completed the Working Alliance Indicator prior to session four and had attended at least four therapy sessions. Outcomes were analyzed using a completed initial and final Outcome Questionaire-45. In addition to a Pearson's correlation for alliance to outcome, this study also attempted to explore the contribution of patient and therapist variability to outcomes. The Pearson's correlation produced for this study was r = .24 indicating that better alliances were associated with better outcomes for study participants.

In the most recent meta-analysis of alliance-outcome studies, Horvath et al. (2011) employed a number of statistical procedures to calculate r for primary studies that had reported data in various other formats. We cannot replicate these calculations based solely on published information; hence, we will need additional information from authors in order to calculate r accurately and transparently for some primary studies.

3.3.2 Criteria for determination of independent findings

We expect to encounter dependent effect sizes in most studies. These are likely to include repeated measures (over time) and multiple alliance measures (scales and subscales). There is also the likelihood that multiple clients have seen the same therapist. It is also possible that the same researchers conducted multiple studies. Because we expect to find several types of dependencies in this literature, we will estimate robust standard errors (Hedges, Tipton, & Johnson 2010) in all analyses (as outlined in detail below). In addition, we may identify multiple reports published from individual studies and may need to contact authors if we cannot clarify whether findings are independent.

3.3.3 Details of study coding categories / data extraction

Prior to screening, results of multiple search strategies will be merged into one list of titles and abstracts, deleting any duplicates. If the search yields more than 10,000 citations, we will explore the use of semi-automated screening (Wallace et al., 2010) and other strategies to expedite the screening process. When necessary, two reviewers will independently read and evaluate titles and abstracts. We will obtain the full text for any report that may meet the eligibility criteria. We will link together any reports that originated from the same study to eliminate the possibility of counting single studies multiple times. Two review authors will independently read full text reports and will then compare notes, creating a consensus list of eligible studies. If missing data prevent us from being able to decide upon a study's eligibility we will correspond with the researchers to obtain the data to facilitate a decision. The review authors will not be blinded to identifying information on journals, authors, affiliations or outcomes. If the review authors disagree about a study's eligibility, they will work together to resolve the issue with possible input from a third evaluator who has methodological proficiency. The final outcome will be a list of both included and excluded studies with a brief rationale for each exclusion decision. In addition, we will produce a PRISMA flow chart, documenting the status of studies and decision points (Moher et al., 2009).

Two review authors will separately evaluate, extract and document the pertinent study information using data extraction forms shown in Appendix 1. Initially, the extraction forms will be tested on two studies and any needed adjustments will be made before beginning the project. Two review authors will perform the study data extraction and then meet and compare forms. Disagreements will be resolved by adjusting the coding or forms, if necessary. Any changes in data extraction forms will be applied retroactively to all studies that have already been coded. Should any issue be unresolved in the data extraction process, the review authors may contact the primary study authors for more information. If this fails to enable resolution, we will consult a third review author. To evaluate reliability we will compute Kappa coefficients on all items involved in eligibility decisions and risk of bias assessment on coding forms 2 and 5.

Five forms have been designed to record data extracted from the reports evaluated in this study. Form 1 is used for initial screening to determine if the study meets basic inclusion criteria such as subject matter, outcomes, and study design. Form 2 is used to evaluate eligibility criteria details such as alliance and outcome measures, and population. Form 3 records study level data extraction criteria such as sampling, recruitment, and setting elements. We will extract data on the location of each study, including city/region, state/province, and country. Form 4 captures reported outcome data and data collection methods. Form 5 reports study quality standards including evaluation of multiple forms of bias and potential conflicts of interest. The forms are located in Appendix 1.

3.3.4 Dealing with missing data

If we identify missing data in a report, we will contact the authors of the study to attempt to obtain data. Information on therapy dropout and attrition from research will be recorded for each study. We will assess attrition (missing cases) and the potential for outcome reporting bias (missing data) as part of our evaluation of study quality (risk of bias). If we have sufficient studies and we observe high levels of attrition in some studies, we will conduct sensitivity analyses to assess the impact of excluding studies with high levels of attrition. Similarly, we will explore for possible associations between risk of selective outcome reporting and results.

3.3.5 Assessment of risk of bias in included studies

Study reports will be evaluated for potential sources of bias associated with: selection, allocation procedures (allocation to therapist), performance, attrition, detection, unstandardized observation periods, unstandardized alliance measures, unstandardized outcome measures, conflicts of interest, and selective outcome reporting (see Appendix 1, Level 5). Further analysis will determine whether all outcomes are reported as well as whether all individuals in the sample are accounted for in the analysis.

The GRADE approach will be used to analyze the quality of evidence related to each of the key outcomes (GRADE Working Group, 2004; Guyatt, Oxman, Vist, Kunz, Falck-Ytter, Alonso-Coello, & Schünemann, 2008). The potential ratings for each outcome are High, Moderate, Low, or Very Low.

3.3.6 Data synthesis

We expect effect sizes to be correlated within studies because each study will likely provide multiple estimates of associations between alliance and outcome measures. The covariance structure of these effect sizes is not likely to be reported in any of the studies (Hedges & Olkin, 1985), so we will use a newly developed technique to handle statistically dependent effect sizes through the estimation of robust standard errors (RSE) (Hedges, Tipton, & Johnson 2010). The RSE approach estimates standard errors based on empirical

assumptions of the variance, reducing the need for assumptions about the distribution of the effect size estimates. The few existing assumptions of the RSE approach are usually met; simulation studies demonstrate that confidence intervals and p-values in samples as small as 10 studies (to estimate average effect size) or between 20–40 studies (to estimate slope) approach the correct size (Hedges et al., 2010). This method therefore allows multiple effect size estimates from the same study to be included in a meta-analysis. The RSE method is an improvement upon other methods such as selecting one effect size per study, or creating a synthetic average effect size for each study, which throw away potentially valuable information about and their variability within a study. If we find sufficient studies, we plan to use RSE to model all of the types of outcomes within the same analysis (mood, symptoms, treatment completion, etc.) rather than using separate models.

Due to the expected variety of alliance and outcome measures, we plan to use a random effects model for all analyses that account for both within study sampling variance and between study variability (τ²). Estimating τ² within the RSE approach requires an estimate of the correlation between all the pairs of effect sizes (ρ). Because ρ is rarely reported in primary studies, as recommended by Hedges et al. (2010) we will use a sensitivity approach to estimating τ² by estimating values of τ² at different values of ρ ranging from 0 to 1.0. Presuming there is little variation in the of τ² estimates at different values of ρ, we will use a fairly conservative value of ρ = .80 for the final τ² estimate to be used in all models.

As recommended by Hedges, Tipton, and Johnson (2010), we will use a conservative approach when estimating the random effects weights, and assume ρ = 1.00 for these calculations. In this approach, each effect size within a study is assigned a random effects weight equal to the inverse of the number of estimates in that study multiplied by that study's variance (i.e., the sum of the mean within study variance plus the between studies variance component). This conservative approach means that any given study does not receive additional weight simply because it provides multiple effect sizes. The software we will use in this review will include Microsoft Word, Microsoft Excel, and Stata. The graphical displays that will be used are Forest plots and Contour Enhanced Funnel plots (Palmer et al., 2008).

Since there is a risk of a non-normal distribution using Pearson's r, we will use Fisher's z transformation, which converts Pearson's r to the normally distributed variable z.

3.3.7 Assessment of heterogeneity

Heterogeneity of results across studies will be assessed informally by checking for overlapping confidence intervals. Formal tests of heterogeneity will be based on the Chi-square (Q) test and the I² statistic (Higgins & Thompson, 2002). The interpretation of the Chi-squared test will be conditional due to its low statistical power. We expect some unexplained heterogeneity and will therefore use a random-effects meta-analysis model.

3.3.8 Moderator analysis

If possible, we will use meta-regression analysis with RSE (Hedges, Tipton, & Johnson 2010) to examine factors associated with the strength of alliance-outcome associations. The potential effect size moderators of interest are the age (e.g., younger 18–24, older 25–34), alliance measure (e.g., Helping Alliance Scales (HAS), Treatment Alliance Scales (TAS), Working Alliance Indicator (WAI), alliance rater (therapist, client, observer), time of alliance assessment (e.g., early, middle, late), outcome and outcome measure (e.g., Beck Depression Inventory, SCL-90-R (Derogatis, 1996), GAS), type of treatment (e.g., Psychodynamic, Rogerian, Cognitive Behavioral), pre-therapy severity of impairment (e.g., mild, moderate, severe), number of sessions, and experience of therapist (e.g., less than three years, more than three years).

One advantage of the RSE approach is that the inclusion of multiple effect sizes per study permits examination of within-study effects of moderators that vary within studies as well as between studies. Four of the moderators of interest may vary within and between studies: rater, timing, type of alliance measure, and type of outcome measure. If any of these moderators do vary within and between studies, we will create two variables for each moderator: (1) the mean value of the moderator within each study used to estimate the between-study effect of the moderator, and (2) a study-mean centered value of that moderator to estimate the within-study effect of the moderator). The remaining four moderators (type of treatment, pre-therapy severity of impairment, number of sessions and experience of therapist) are expected to only vary between studies, and will be included as standard moderators in the meta-regression model. We will also examine and present the bivariate correlations between all moderators to assess any potential confounding among them.

3.3.9 Sensitivity Analysis

We will conduct several types of sensitivity analyses. First, we will conduct sensitivity tests with different values of ρ in the RSE models, to examine whether this assumed correlation value had any impact on the estimate of τ²and the resulting parameter estimates and standard errors. Second, we will examine the potential impact of missing data on the review's findings. We will also remove studies with high risk of bias of specific types (e.g., high risk of selective reporting) to see whether an analysis of the studies with low risk of bias gives a different result. Finally, we will use the Egger test (Egger et al., 1997), trim and fill analysis, and Contour Enhanced Funnel plots (Palmer et al., 2008) to check for the possibility of publication bias and small sample bias.

3.3.10 Roles and responsibilities

Stacy J. Green will lead the writing of the protocol, including the background sections. She will compile the search terms and perform some of the searches, review documents for inclusion and exclusion criteria, extract and compile data, conduct the analysis and synthesis of data, and lead the writing of the review.

Julia H. Littell will review and edit the protocol, the pilot testing, and the review as it progresses, providing substantive and methodological guidance and direction as needed.

Karianne Hammerstrøm will design and carry out the search strategy, performing many of the database searches. She will also write and/or edit the sections in the protocol that refer to the search and will also write the search sections of the review.

Emily Tanner-Smith will write the protocol sections on data synthesis. She will provide consultation on the synthesis and will review and edit the data synthesis sections of the review including discussion of unit of analysis issues.

Jessica Schaffner Wilen will review and edit the protocol as it progresses; review documents for inclusion and exclusion criteria; and extract, compile and review data.

3.3.11 Preliminary timeframe for completion of review

After the protocol is approved, we estimate that the search will take approximately six months. Following the initial search, coding and data extraction will take two to three months and analysis will take three months. Therefore, the timeframe for the entire review is one year in total.

3.3.12 Plans for updating the review

Stacy J. Green will be responsible for updating this review every three years in accordance with Campbell Collaboration guidelines and as funding becomes available.