Multisystemic Therapy® for social,emotional,and behavioural problems in youth age 10 to 17: An updated systematic review and meta-analysis

What is this review about?

MST® is an intensive, home-based intervention for families of youth with social, emotional, and behavioural problems. MST therapists engage family members in identifying and changing individual, family, and environmental factors thought to contribute to problem behaviour. Intervention may include efforts to improve communication, parenting skills, peer relations, school performance, and social networks. MST is widely considered to be a well-established, evidence-based programme.

We synthesised data from all eligible trials to test the claim that MST is effective across clinical problems and populations.

What is the aim of this review?

This Campbell updated systematic review and meta-analysis synthesised data from all eligible trials to test the claim that Multisystemic Therapy® is effective across clinical problems and populations.

What studies are included?

Included studies examined outcomes of MST for juvenile offenders, sex offenders, offenders with substance abuse problems, youth with conduct or behaviour problems, those with serious mental health problems, autism spectrum disorder, and cases of child maltreatment.

This review summarises findings from 23 randomised controlled trials of the effects of MST. These trials were conducted in the USA, UK, Canada, the Netherlands, Norway, and Sweden.

Most trials compared MST to TAU. In the USA, TAU consisted of relatively little contact and few services for youth and families, compared with more robust public health and social services available to youth in other high-income countries. One USA study provided “enhanced TAU” to families in the control group, and two USA studies compared MST to individual therapy for youth.

What are the main findings of this review?

Available evidence shows that MST reduced rates of out-of-home placement and arrest or conviction in the USA, but not in other countries. Moderate to low quality evidence shows that MST had positive effects on self-reported delinquency and parent and family functioning, but we found no evidence of overall impacts on youth symptoms, substance abuse, peer relations, or school outcomes. Prediction intervals indicate that future studies are likely to find positive or negative effects of MST on all outcomes.

What is the quality of the evidence?

The quality of evidence for MST is mixed. There was only one prospectively registered trial with complete reporting on all planned outcomes and endpoints. Nineteen trials (83%) had missing data on subgroups, outcomes, or endpoints.

We identified high risks of bias due to: inadequate randomisation procedures, lack of comparability between groups at baseline; systematic omission of cases; attrition; confounding factors, such as between-group differences in race, gender, and attention; selective reporting of outcomes; and conflicts of interest.

Most MST trials (96%) had high risks of bias on at least one indicator. GRADE ratings of the quality of evidence for seven primary outcomes are low to moderate, with high quality evidence on out-of-home placements from non-USA studies. USA studies led by MST developers had higher risks of bias, and USA control groups received fewer services and had worse outcomes (more out-of-home placements and arrests) than those in independent trials conducted in other high-income countries. Although these moderators are confounded, the USA/non-USA contrast appears to be more closely related to variations in effects across studies than investigator independence or risks of bias.

What are the implications for research and policy?

Our results stand in stark contrast to many previous reports and reviews on MST. Although most MST trials produced a mixture of positive, negative, and null findings, many reports focused selectively on positive, statistically significant results instead of all results.

Careful appraisal of study methods and risks of bias was lacking in many published reports and reviews. Some investigators and many reviewers failed to consider alternative plausible explanations for results that appeared to favour MST (e.g., lack of comparability of groups at baseline; differential attrition; confounding influences of race, gender, and additional attention paid to MST cases; and selective reporting of results).

How up-to-date is this review?

The review authors searched for studies that were reported through March 2020.

Research base

In 2020, funding for research on MST exceeded $75 million USD (MST Services 2020b). According to MST Services LLC, 79 MST outcome studies have been published, involving 58,000 families across studies (because reviews of existing studies are included in this list, many families are counted more than once). Of these studies, 28 were randomised controlled trials (RCTs) conducted to assess impacts of MST for youth with a wide range of presenting problems (including studies of youth with medical problems, which were not included in our review).

Most MST trials were conducted in the USA by the developers of MST, many of whom were based at the Family Services Research Center (FSRC) at the Medical University of South Carolina (MUSC). Independent trials have been conducted in six countries.

Studies have assessed effects of MST on a wide array of outcomes in diverse samples of youth and families. Outcomes were measured after treatment and at several follow-up points. Follow-ups range from several months to 22 years after referral. Thus, there is ample evidence to assess the effectiveness of MST across problems, populations, outcomes, and endpoints.

Our previous review included eight trials conducted in the USA, Canada, and Norway. Since then, more than a dozen new trials have been completed in the Netherlands, Sweden, USA, and the UK, and additional follow-up data are available on three of the eight studies included in our earlier review.

Other reviews

We identified 417 published reviews of research on the effectiveness of MST (note that the number of reviews is five times greater than the number of published studies and 15 times greater than the number of RCTs identified by MST Services). However, most of these reviews are nonsystematic narratives that do not meet scientific standards for evidence synthesis (e.g., PRISMA, Moher 2009).

Results of MST outcome studies are summarised in nonsystematic reviews of effects of family preservation services (Fraser 1997b), interventions for child physical and sexual abuse (Swenson 2003), treatment for substance abuse (NIDA 1999), treatment for delinquency and disruptive behaviour in youth (Smith 1997), children's mental health services (Burns 2004, Burns 2000, Kazdin 1998, Kazdin 2015), and programmes to reduce crime (Aos 2001, US DHHS 2001) and prevent violence (Mihalic 2004). Several reviews suggested that MST is one of the most promising empirically based treatments for children and youth (Hoagwood 2001, Kazdin 1998). One nonsystematic review concluded that MST has positive effects that been replicated “across problems, therapists, and settings. This shows that the treatment and methods of decision making can be extended and that treatment effects are reliable” (Kazdin 1998, pp. 27–28). These conclusions were often repeated. At least 20 published reviews relied primarily on other reviews of MST or did not cite any sources (Littell 2008).

MST trials are included in meta-analytic reviews of effects of a wider array of interventions with juvenile offenders (Lipsey 1998), family treatment of youth delinquency (Latimer 2001), and family and parenting interventions for conduct disorder and delinquency (Woolfenden 2002, Woolfenden 2004). These reviews do not speak to the effectiveness of MST per se.

There are seven previous systematic reviews or meta-analyses of research on effects of MST (not including the earlier version of our review). These reviews are described below and we provide a brief assessment of qualities of these reviews, using an adapted version of the AMSTAR tool (Shea 2007), in Table 1.

Farrington and Welsh reviewed results of six MST trials (five conducted in the USA and one in Canada) (Farrington 2003). Their search strategy and meta-analytic methods were not fully explained, there was no study quality assessment, and no heterogeneity tests were reported. Based on comparisons of mean effect sizes (ES) and confidence intervals (CIs), authors concluded that MST is the most effective family-based crime prevention programme (Farrington 2003, p. 143).

Curtis and colleagues (Curtis 2004) reported results of a meta-analysis of seven published studies of effects of MST programmes conducted by MST program developers in the USA. Unpublished studies and those conducted by independent researchers were not included. This review included studies of abusing or neglectful parents, juvenile sexual offenders, violent and chronic juvenile offenders, substance abusing juvenile offenders, and psychiatrically disturbed adolescents. Effect sizes (d indexes) and their variance were estimated incorrectly, and some nonsignificant and negative effects were ignored (see Littell 2008). Corrections for small sample bias were applied to only one study. Curtis and colleagues reported an overall, unweighted effect size of d = 0.55 based on 11 summary effect sizes from seven studies. The effect sizes in this estimate are not independent (as they should be), because some samples are represented twice. Reviewers did not use inverse variance methods or other methods to adjust for differences in the precision of the estimates. Results appear to be affected by publication bias (cf. Rothstein 2005), allegiance effects (cf. Luborsky 1999), and estimation errors (Littell 2008).

Lofholm and colleagues reviewed results of 13 RCTs of MST to explore differences in TAU conditions across studies (Lofholm 2013). They found greater variability in recidivism rates between the TAU groups in these studies than between MST groups. Authors noted that these differences made it difficult to compare outcomes and treatment effects across studies.

Van der Stouwe and colleagues conducted a multilevel meta-analysis of 22 studies of effects of MST for youth with antisocial behaviour, delinquency, and/or conduct disorders (van der Stouwe 2014). They included both randomised controlled trials and nonrandomised comparison group studies, and unpublished as well as published studies. Study quality was assessed with a uni-dimensional scale (a practice abandoned by Cochrane and other reviewers, who view study quality as a multidimensional construct; Jüni 1999, Jüni 2001). Van der Stouwe and colleagues found small, but statistically significant effects of MST on delinquency, out-of-home placement, substance use, and peer relations; but these effects were not significant after adjustments were made for publication bias. Small but statistically significant effects on psychopathology and family factors were evident, even after adjustments for publication biases. There were no significant effects on skills or cognitions, and no evidence of publication bias in reports on those outcomes.

Lux conducted a meta-analysis of 127 effect sizes from 35 unique MST studies (using 44 published and unpublished reports; Lux 2016). This review did not provide: a full description of the search strategy, methods for study selection, a list of excluded studies, study quality assessment, or discussion of methods used for moderator analysis. Both RCTs and quasi-experimental designs were included. Continuous and dichotomous study ES were converted to correlation coefficients for meta-analysis. Both fixed and random effects meta-analyses were performed (best practice is to select one of these models based on a priori assumptions about which model best fits the distribution of effect sizes; Borenstein 2010).

Markham provided a systematic review of 11 RCTs of MST conducted within and outside of the USA (Markham 2016, Markham 2018). Picking up where our previous review left off, Markham included studies published from 2006 to 2014. She noted that comparisons between these studies are challenging, due to inconsistencies in reporting on usual services and cultural differences in the cross-national transportation of MST. She used narrative review methods, not meta-analysis, and concluded that outcomes for MST “continue to be mixed across studies” (Markham 2018, p. 67).

Tan and Fajardo (Tan 2017) reported a systematic review of 12 RCTs on the efficacy of MST. This review was limited to published studies, hence it is vulnerable to publication bias. Authors assessed study quality on a uni-dimensional scale (limitations of this approach are noted above). Tan and Fajardo presented results in narrative and tabular forms. Instead of conducting meta-analysis, they used simple vote-counting to summarise results across studies (e.g., “2 out of 3 studies showed positive outcomes of MST in reduction of antisocial behaviour”, Tan 2017, p. 97).

Problems with vote counting have long been recognised (Hedges 1980, Gurevitch 2018). The Cochrane Handbook states that, when based on statistical significance or subjective rules, vote counting is an “unacceptable synthesis method” (Higgins 2020). This approach often leads to the wrong conclusions.

As shown in Table 1, none of these reviews had protocols that were available a priori, none provided a list of excluded studies, none completed thorough study quality or risk of bias (ROB) assessments, none had adequate methods for taking study quality into account, and none provided conflict of interest statements. Our previous review, published in 2005, had most of these features; the present version has all of them.

In 2005, we thought it was premature to draw conclusions about the effectiveness of MST based on inconsistent results from eight trials that varied in quality and context (Littell 2005a, Littell 2005b). Others have cited more limited evidence with weaker review methods and more surety. Even after the publication of more than 400 reviews, questions about the benefits of MST remain: are effects of MST consistent across populations, problems, outcomes, and over time? Can variations in effects be explained by study qualities, sample characteristics, intervention characteristics, comparison conditions, or contexts? Methodological weaknesses in many previous reviews limit confidence in the answers they provide.

By updating our systematic review—with evidence from new trials, additional follow-up data on old trials, and newer meta-analytic methods—we address unresolved issues and provide more robust estimates of effects of MST on outcomes for youth and families.

Types of studies

This review was limited to experimental studies in which participants were randomly assigned to treatment and comparison groups. Outcome evaluation studies using other group designs were identified, but not included. There were no publication or language restrictions.

Types of participants

Participants included children and youth (age 10 to 17) with social, emotional, and behavioural problems, and their family members. These youth may have been at risk of out-of-home placement. Participants included:

Abused, neglected, and dependent children and youth at risk of foster care or other out-of-home placements in child welfare settings;

Given these eligibility criteria, programmes for emerging adults (age 17 to 26) were excluded, as were programmes for youth whose presenting problems were medical in nature (e.g., diabetes, HIV, obesity, asthma).

Types of interventions

MST (as defined above) was compared with any counterfactual condition, including (a) TAU, (b) an alternative treatment condition (e.g., individual therapy, group therapy), or (c) no treatment. To be included in this review, focal programmes had to be licensed MST programmes; other “multisystemic” treatments were not included.

In recent years, MST developers created specialised programmes to address needs of various clinical populations (MST Services 2019). In addition to the original version of MST, specialised MST programmes included in our review focused on

Child abuse and neglect (MST-CAN),

Consistent with our original eligibility criteria, studies were not included in our review if focal interventions: (a) served youth and families whose problems are primarily medical in nature, (b) targeted youth younger than 10 or older than 17 years of age, or (c) combined MST with other treatments. For example; MST has been combined with Contingency Management (CM) for substance abuse; CM is a distinct intervention with its own evidence base (Blonigen 2015). Thus, we excluded studies of the following programmes:

MST plus contingency management (MST-CM) for substance-abusing youth;

Types of outcome measures

We examined measures of behavioural, psychosocial, and family outcomes.

Youth behavioural outcomes included antisocial behaviour (evidenced by arrest, conviction, or sentencing for criminal offences), drug use, and school attendance.

These outcomes were assessed in a variety of ways, including data extracted from official agency records, self-reports on standardised instruments, observational measures, and biologic tests. Data on events such as arrest or conviction, out-of-home placement, and school attendance were often obtained from official agency records (law enforcement, hospital, school, and child welfare agency administrative records), although some studies relied on interviews with caregivers to ascertain children's living arrangements or grades in school. Psychosocial outcomes were often assessed on standardised instruments that were self-administered or embedded in structured interviews. Observational measures were sometimes used to assess certain aspects of family functioning or relationships. A few studies used biologic measures of substance use; others used self-reports. Many studies employed multiple data collection procedures, which had different potential risks of bias. We conducted separate risk-of-bias assessments for the following types of data:

Data extracted from administrative records, and

Outcome measures were obtained at varying points in time; some were anchored to the time that had elapsed since random assignment, others were anchored to the end of treatment. Some studies collected data during or immediately after treatment (4 to 8 months after random assignment). Because some cases were still receiving treatment at 8 months, we assessed outcomes in the following categories:

1 year follow-up (9–18 months),

Before beginning our update of this review, we identified the following primary and secondary outcomes.

Electronic searches

We searched for new studies in September 2010 and again in March to April 2020. In advance of these searches, we revised our original search strategies to reflect changes in databases and interfaces, and to increase the sensitivity of the research design terms. We used the Cochrane Highly Sensitive Search Strategy for identifying randomised trials for MEDLINE. Original date restrictions were lifted in order to find any relevant studies which the original search may have missed. Specific search strategies for each database are shown in Appendix A No language restrictions were applied. We searched the following databases:

ASSIA (ProQuest): 1987 to September 2010 (searched September 17, 2010), 2010–2020 (searched March 31, 2020)

Two databases were searched for the original review, but not included in this update: the C2 Spectr database is no longer maintained and InfoTrac is not available to us.

Searching other resources

We searched the following websites on April 13, 2020, using search strings shown in Appendix A.

MST Services (www.mstservices.com)

In September 2010 we conducted a Google Scholar search and examined the top 200 hits. On March 31, 2020, we updated this search, limiting the date range to 2010–2020 and using the following search string: (multisystemic OR multi-systemic OR “multi systemic”) AND (therapy OR treatment). We examined the top 100 hits. (These searches were more specific than the Google searches we ran in January 2003.)

Selection of studies

Two reviewers independently screened titles and abstracts identified in the search, using the Level 1 coding scheme shown in Appendix B to indicate which reports were clearly ineligible (and why) and which documents should be retrieved. If an abstract was not available, we attempted to retrieve the full text. We made inclusive screening decisions at this first stage; that is, if either reviewer thought the document might be eligible for our review or if there was not enough information in the title and abstract to make this decision with confidence, we retrieved the full text.

Before formally applying our eligibility criteria to each study, we grouped all documents that belonged to that study. It was important to focus on the study as the main unit of analysis, instead of focusing on study reports. We define a study as a set of research procedures that involves a unique sample of participants, a sample which does not overlap with samples used in other investigations. This is to avoid confusion in the narrative, allow more in-depth analysis of study characteristics and methods, and avoid double-counting of participants in meta-analysis. Studies often generated multiple reports related to different research questions, subgroups, types of analyses, or end points; these are not treated as separate studies, if they are based on the same sample or overlapping subsamples.

Working independently, two reviewers read all of the documents that belonged to each study and applied the eligibility criteria to that study, following the algorithm shown in Appendix B, Level 2. We recorded one reason for exclusion for each excluded study, using a “first strike” rule: eligibility questions were answered in a predetermined order and, if a study failed to meet one criterion, that reason for exclusion was documented and the screening process was stopped. It is possible that studies failed to meet additional criteria that are not documented. Selection decisions were reviewed and disagreements were resolved by the review team.

We summarised results of searches, screening, and eligibility decisions using a PRISMA flowchart. A complete list of excluded studies is provided (see Section 5).

Data extraction and management

Information on study design and implementation, sample characteristics, intervention characteristics, and outcomes was extracted from included studies and coded using a structured data extraction form (see Appendix B, Levels 3–5). Two reviewers independently read all reports associated with an included study and coded information on that study. Differences between raters were discussed in attempt to resolve any discrepancies. When needed, a third rater was consulted.

When we encountered conflicting reports on the number of cases that had been randomly assigned to treatments within a study, we selected the largest credible count. We used this number as the denominator when calculating rates of attrition over time and in subsequent reports.

When we encountered conflicting reports on the presence or absence of between-group differences on baseline characteristics, we relied on accounts that provided descriptive data on those characteristics at the group level. We used the What Works Clearinghouse criteria for group equivalence on baseline characteristics (between-group differences d < 0.25, WWC baseline). We used David Wilson's ES calculator to compute the d statistic (using the probit method) to quantify the magnitude of differences between groups.

We extracted information on all outcome measures mentioned in the study protocol (if there was one) and in all subsequent reports, regardless of whether outcome data were ever reported. This provided us with the information we needed to assess selective reporting of outcomes and missing data.

To the extent possible, we extracted data on all primary and secondary outcomes at all endpoints, recording data on the timing of the measurement of each outcome. We extracted data on total scores and subscale scores, when both were provided. We recorded data on composite events (e.g., all arrests) and their subtypes (e.g., arrests for violent crimes, arrests for nonviolent crimes) when these data were provided.

Some studies provided reports on the same outcome at the same endpoint in multiple documents (e.g., preliminary and final reports). To avoid duplication and include the most complete report, we selected the outcome data with the largest valid n.

A few studies provided results for observed data along with analyses that used multiple imputation of missing data. When both types of results were available, we extracted results for observed data, because this approach was more common across studies. Multiple imputation made little difference in results of one of the largest studies (Fonagy 2018, 2018a, p. 17). When observed results were not available, we extracted data from analyses that used imputation, but only if the valid n was >50% of the total sample size.

When data on the full sample was available, we did not extract data on subgroups (e.g., we did not extract data from analyses limited to program completers or recidivists). We did not extract data on outcome measures collected during treatment (<4 months after referral).

We did not extract data on time-to-events or hazard rates, given our plans to use SMD and OR effect size metrics.

We found conflicting reports on some outcomes, despite the fact that the data came from identical samples (same valid n), measures, reporters, and endpoints. When this occurred, we selected reports that provided data needed to calculate effect sizes (e.g., valid ns, means, SDs for treatment and control groups) and/or more complete accounts of the details of measurement and analysis. Given their greater length and some evidence that dissertations exhibit stronger methodologies than published reports in this field (McLeod 2004), we sometimes used data from dissertations instead of published reports.

As indicated above, some studies anchored follow-ups to the beginning of treatment and others anchored follow-ups to the end of treatment. For follow-up periods anchored to end of treatment, we added six months to the reported endpoint to make these observations comparable to those anchored to random assignment. For example, a one year follow-up period that begins at the end of treatment is estimated to be 18 months after random assignment.

For outcomes related to events (e.g., out-of-home placement, arrest, or conviction), we included reports on events that occurred between random assignment and the follow-up endpoint (whenever possible) in pairwise meta-analysis. Some studies did not provide data on events that occurred during treatment, so that the observation period began when treatment ended. Others provided data on events that occurred within discrete intervals (e.g., 0 to 6, 6 to 12, and 12 to 18 months). We could not collapse dichotomous data on events into longer intervals (e.g., 0 to 12 or 0 to 18 months), because we did not know how many people experienced events within multiple time periods.

When studies provided two estimates of outcomes within the same interval used in our pairwise meta-analysis (e.g., observations at 18 and 30 months post random-assignment both fit our criteria for the 2.5 year observation period), we selected the estimate with the largest valid n or (if valid ns were identical) the longest observation.

For continuous data on events that occurred within specific time intervals, we were able to aggregate data across intervals when the valid ns for those intervals were identical. The cumulative mean is computed by adding group means for all relevant intervals (e.g., mean number of offences in 0 to 6 months + mean number of offences in > 6 to 12 months = mean for 0 to 12 months). The corresponding standard deviation is calculated by adding the variances for each interval and taking the square root of the sum of the variances.

Authors of studies with missing data were contacted and some additional data were obtained as a result.

Assessment of risks of bias (ROB) in included studies

We updated our approach to the assessment of ROB, to incorporate more explicit methods that had been developed since the publication of the protocol for this review (Littell 2004). We adapted the first version of the Cochrane ROB tool (Higgins 2011) and What Works Clearinghouse standards for baseline equivalence and attrition (WWC attrition; WWC baseline) and applied these criteria to all studies.

Measures of treatment effect

Continuous data were analysed if means and standard deviations were available or there was some other way to calculate effect size (e.g., from t tests, F tests, or exact p values). When reports contained insufficient data, we sought additional information from the authors. Studies used diverse scales to measure the same clinical outcomes (e.g., psychiatric symptoms), so we used standardised mean differences (SMD) to facilitate comparisons across studies. The RevMan formula for SMD is Hedge's g, which is like Cohen's d but includes an adjustment for small sample bias.

Binary outcomes were analysed by calculating odds ratios (OR) with 95% CIs. Attempts were made to preserve information about base rates (in control groups) and between-group differences in proportions, since this provided important contextual information.

After computing effect sizes (ORs and SMDs), we examined outliers and checked to make sure that our data accurately reflected study reports. We used log odds ratios (LORs) in meta-analysis, and converted results back to ORs for presentation.

Unit of analysis issues

MST trials randomly assigned youth and their families to treatments. In addition to a focal young person, some studies conducted analyses of outcomes for siblings or sibling groups. We did not include data on outcomes for siblings, because families were the main units of analysis in most studies, the focal youth and parent(s) were the main focus of intervention, and some focal youth did not have siblings.

When we encountered multi-armed studies, we limited our comparisons to the two arms that best represented typical implementation of MST and a non-MST control group. For example, if a three-armed study compared MST plus another intervention to MST-only and a usual services control group, we ignored the first group (which did not meet our inclusion criteria) and compared results for the last two groups. Similarly, for studies that used factorial designs to test MST, another intervention, and the interaction of these two treatments, we compared the arms that best represented MST and a similarly situated control group.

Dealing with missing data

When we identified missing data (on studies, cases, outcomes, or effect sizes), we contacted investigators with requests for more information.

We are concerned here with possible reasons for missing data. Data can be missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). MCAR and MAR data are not likely to affect results of meta-analysis, but MNAR data will (Pigott 2019). When studies, cases, outcomes, or effect sizes are not fully reported for reasons that are related to their results, meta-analysis of available data will be biased. For example, nonpublication or nonreporting of negative or null results will lead to inflated effect sizes in meta-analysis, as will the systematic loss or omission of subgroups of participants or sites with more negative outcomes.

To assess issues related to missing data, we recorded data on attrition and differential attrition for each outcome and each endpoint. As discussed below, we tracked the reporting of outcomes and assessed evidence of reporting bias and publication bias.

When published analyses systematically excluded data on subsamples (sites or cases) with poor outcomes, we conducted best case/worse case (BC/WC) scenario analysis to calculate the range within which a reported effect size must lie. For dichotomous outcomes, this involves calculating a lower bound, which assumes that all missing MST cases had negative outcomes and all missing all control cases had positive outcomes (worst case), and an upper bound, in which all missing MST cases had positive outcomes and all missing control cases had negative outcomes (best case).

When important details of analyses (e.g., valid ns, SDs) were not available from authors, we estimated missing data using methods described in Appendix C. We used Cochrane's Finding_SDs.xls to calculate missing standard deviations (training.cochrane.org/resource/revman-calculator).

Assessment of heterogeneity

Heterogeneity was evaluated with I ², the χ ² test of heterogeneity, and visual examination of overlap between CIs in forest plots.

Assessment of reporting biases

We extracted data from all available study reports (including protocols, when available), and tracked the reporting and nonreporting of data on specific outcomes and endpoints. We identified full reporting, partial reporting, and missing data on specific outcomes in an outcome matrix (following Dwan 2010) and we documented missing data on endpoints in a separate table. When the number of studies (k) in an analysis was > 10, we examined funnel plots for evidence of publication bias and small sample bias.

Data synthesis

We used pairwise meta-analysis to synthesise data from multiple studies on comparable outcome measures at similar points in time. We used correlated effects (CE) meta-analysis models to synthesise data on all available outcomes within nine conceptually distinct outcome domains: out-of-home placements, arrest or conviction, self-reported delinquency, substance use, peer relations, youth behaviour and symptoms, parent behaviour and symptoms, family functioning, and school outcomes.

Given substantial differences between studies in participant characteristics, treatment implementation, comparison conditions, and research methods, we did not expect all studies to produce estimates of the same population parameters. For this reason, we used random effects models whenever possible (i.e., in pairwise meta-analysis and in CE models with more than five studies).

Subgroup analysis and investigation of heterogeneity

We examined several characteristics of studies, samples, programmes, and outcomes as possible moderators of treatment effects. In forest plots, we grouped studies using two categorical moderators (described below), examined results within and across subgroups of studies, and used χ ² tests for differences between subgroups.

We assessed eleven potential moderators (discussed below) using CE models. In our dataset, meta-regression models with multiple moderator variables produced statistical tests with fewer than four degrees of freedom; these results were not reliable. Given this constraint, along with an imbalance in the number of effect sizes reported across studies, concerns about statistical power, and confounded moderators, we used single-variable CE models to explore heterogeneity. That is, our exploratory analyses of potential effects of moderators uses a separate CE model for each moderator and outcome.

Sensitivity analysis

As described above, we conducted best case/worse case scenario analyses to estimate the range of effects that might have been obtained in studies with data that were missing not at random (MNAR).

Also mentioned above, we assessed the sensitivity of CE models to various assumptions about the size of the correlations between effect sizes within studies.

Summary of findings and assessment of the certainty of the evidence

We used the GRADE guidelines (gdt.gradepro.org) to assess the certainty of evidence regarding seven primary outcomes.

Results of the search

Electronic database searches produced a total of 3784 records (411 records were identified in 2003, 1103 in 2010, and 2270 in 2020). Internet searches conducted in 2003 produced 4662 hits (including results of a Google search); more sensitive internet searches (using Google Scholar instead of Google) were conducted in 2010 and 2020 and these yielded 218 and 220 hits respectively. Reference harvesting yielded 350 citations. Personal contacts helped us identify 55 documents. Many bibliographic records appeared in multiple databases and also in other sources. After duplicates were eliminated, we had 1808 unique citations (see Figure 1). Six of these could not be screened because we did not have access to abstracts or full-text.

OPEN IN VIEWER

Figure 1

PRISMA flow diagram

We screened 1,802 titles and abstracts, ruling out: unrelated topics and programmes (n = 739 records); studies of MST for medical conditions (n = 37); research reviews (n = 417); descriptive, correlational, single group, and case studies (n = 143); theory or position papers, editorials, and book reviews (n = 203); and practice guidelines and treatment manuals (n = 25).

We retrieved 723 full-text reports, including 353 reviews. Forty-two of these retrievals required inter-library loans.

We identified 104 distinct MST outcome studies. Many studies had multiple reports, with an average of 2.77 reports per study.

We contacted 22 experts in attempt to obtain missing data on 16 studies. Some experts were contacted in relation to multiple studies. Thirteen experts (59%) responded. From these contacts we learned that two studies did not meet our eligibility criteria (Hurley 2004, Sheidow 2003), results of one study were not available (Schoenwald 2004), and we received additional information on three studies (Asscher 2013, Butler 2011, Glisson 2010).

Included studies

Twenty-three studies met the inclusion criteria for this review. These studies involved 3987 participant families who were randomly assigned to eligible treatment and comparison conditions.

Included studies had at least one and up to 20 reports per study, with an average of 7.17 reports per study (SD = 3.01). Of the 165 reports associated with these 23 studies, 104 reports were published (e.g., journal articles, book chapters) and 61 were unpublished (e.g., conference presentations, dissertations, theses, government or foundation reports). As shown in Table 2, most studies produced both published and unpublished reports. One report was written in Swedish and the rest were written in English (we used Google translation software to read the Swedish report).

Characteristics of included studies are described below.

Timing of outcome observations

As shown in Figure 2, the 23 studies in our review collected data at multiple endpoints. Most (21) studies collected data at baseline and immediately after intervention ended (4–8 months after random assignment). Sixteen studies collected data at approximately 1 year (9–17 months), 16 collected data at about 2.5 years (18–40 months), 6 collected data at approximately 4 years (41–60 months), two at 9–10 years, one at 14 years, and one at 22 years following random assignment.

OPEN IN VIEWER

Figure 2

Data collection time points: months post baseline

Several studies did not use standardised observation periods in their data analysis; that is, enrolment into the study occurred over a period of months or years, but outcome data (e.g., from administrative records) were obtained at a fixed point in time. Some studies used survival analysis to account for varying observation periods or used event history data to calculate outcomes within common intervals; but others did not, leaving unadjusted variations in observation periods (e.g., for arrest rates reported in Borduin 1990, the mean length of the observation period is 37 months, but the range is 21 to 49 months). We classified these observations using the average number of months that had elapsed since random assignment, and we captured information on the range (width) of the observation period in months.

Only five studies (22%) provided full reports on all outcomes on all endpoints (post-treatment and follow-up periods), and four studies (17%) provided no public data on main effects of treatment at one or more endpoints (see Figure 2). For example, Henggeler 2006 collected outcome data on 161 cases at six distinct follow-up endpoints from one to five years after random assignment, but only reported main results for the first follow-up at one year. In contrast, Fonagy 2018 collected data on 684 cases at seven points over a five-year period, and provided full reports on all outcomes at all endpoints.

Excluded studies

Of 104 studies identified, 80 were excluded because they did not meet eligibility criteria for this review. Specific reasons for exclusion are shown in Characteristics of excluded studies. Using the decision criteria shown in Appendix B, Level 2, we recorded one reason for exclusion per study, although there may have been multiple reasons. As shown in Figure 1, 10 studies were excluded because they were not focused on youth with social, emotional or behavioural problems (these studies involved families of youth with Type 1 diabetes, HIV/AIDs, obesity, and those with family members receiving methadone); 35 studies were excluded because they lacked comparison or control groups; 25 did not use random allocation to treatments; seven studies assessed “multisystemic” interventions that were not licensed MST programmes, or examined treatments that combined MST with other interventions; and three studies were excluded because they did not meet our age criterion (most participants were younger than 10 or older than 17 years of age).

One randomised controlled trial could not be assessed, because we did not have access to the study report (see Schoenwald 2004 in Characteristics of studies awaiting classification).

We had planned to include a randomised trial in Denmark, but this study was cancelled after all five MST teams declined to participate (Pontoppidan 2012).

Sequence generation

All studies indicated that a random component was used to allocate families to treatments. Some studies described this process in detail, others did not. Some approaches were more sophisticated and more fool-proof than others (e.g., computer generated assignments at a remote location are more secure and pose lower risks of bias than coin tosses conducted on-site).

It is not clear whether randomisation was applied to all cases in some studies. For example, in the Henggeler 1997 study, 146 cases were assigned to MST or usual services in 73 yoked pairs and nine cases were assigned to MST. The Ogden 2004 study assigned 62 families to MST and 38 to usual services, but later replaced four of the cases that were originally assigned to MST (T. Ogden, personal communication, 4 October 2003). It is not clear whether these additional cases were randomised.

Two studies have high risks of bias on this indicator, 13 have unclear risks, and 8 have low risks of bias (Figure 4).

Allocation concealment

Methods of allocation concealment were not always fool-proof. Coin tosses were used in some studies (Borduin 1995, Timmons-Mitchell 2006), while sealed envelopes were used in others (Henggeler 1999b, Ogden 2004). Some studies noted when and where randomisation occurred, but did not describe the method of randomisation. These details raised concerns, for example, when random assignment took place in the family home with a MST therapist present (Leschied 2002).

Two studies have high risks of bias on this indicator, 15 have unclear risks, and six have low risks of bias.

Baseline equivalence

As indicated above, we used the standardised mean difference (Cohen's d) to assess the magnitude of differences between groups at baseline, following WWC guidelines 2018. Baseline equivalence was rated as unclear for studies that reported insufficient data to compute Cohen's d. Note that our assessments of baseline equivalence differ from some assessments by investigators, who reported whether between-group differences were statistically significant. Nonsignificant differences with d > 0.25 occurred in small samples and statistically significant differences with d < 0.25 appeared in very large samples; we treated the former as evidence of “real” differences (high ROB) and the later as evidence of low ROB.

Available data on baseline equivalence often excluded cases that refused to participate after random assignment. Whether due to randomisation or early attrition, there was a lack of baseline equivalence in 15 studies (65%). In some studies, the MST group included substantially larger proportions of Whites and female youth (Borduin 1995), parents with higher levels of education, two-parent families, and families with greater wealth or higher socio-economic status (Asscher 2013) than their counterparts in the comparison group.

Unpublished reports on 176 families that remained in the Borduin 1995 study showed significant differences between the MST and control groups on race, gender and previous arrests. Schaeffer 2000 (p. 120) found a higher proportion of Whites (74% versus 69%, p < .05) and fewer males (62% versus 77%, p < .05) in the MST group (n = 92) compared with the control group (n = 84). The MST group had an average of 1.65 nonviolent arrests (SD = 1.09), compared with 0.98 (SD = 1.96) in the control group (p < .01). Further racial and gender differences appeared when comparing youth who completed MST (82% were White and 59% were male) and those who dropped out of MST (73% were male; Schaeffer 2000, p. 123). In a follow-up study of caregivers of youth in this study, Johnides 2015 found that 84% of caregivers of MST youth were White, compared with 73% of caregivers in the control group (χ ² = 3.94, df = 1, p = .047). In contrast, published reports in this study stated that there were no between-group differences in demographic characteristics or criminal histories, but provided no data to support these statements (Henggeler et al., 1991, p. 45; Borduin et al., 1995, p. 570, 572; Klietz et al., 2010, p. 659; Sawyer & Borduin 2011, p. 644; Dopp et al., p. 314).

An early report on the Timmons-Mitchell 2006 study indicated that there was a significant gender imbalance between groups, with 23 females in the MST group and 14 in the control group; authors noted that this was the result of random assignment, which does not always produce equivalent groups (Timmons-Mitchell 2003b, p. 11). The final report on this study stated that there was no significant between-group difference on gender, but did not provide data to confirm this (Timmons-Mitchell 2006). Data on between-group differences are presented in Table 4.

There were substantial differences in race and referral source at baseline in the Fonagy 2017 trial, but these differences were fully reported: There were more Black youth and fewer Whites in the MST group. Cases in the MST group were more likely to be referred from social care services and less likely to be referred by youth offender services, compared with cases in the control group (Fonagy 2017, Table 5).

Performance bias (confounding)

Ten studies had high risks of performance bias related to large imbalances in the amount of attention paid to participants and/or clinicians. In these studies, MST cases received substantially more contact and/or longer treatment, not just different treatment. MST therapists often received special attention, training, and supervision (sometimes by program evaluators) that was not available for clinicians who provided services to the control group. It can be argued that this extra or special treatment is part of the MST intervention, but then we cannot know whether any differences in outcomes are due to content (MST) or amounts of contact.

Detection bias (blinding)

In MST trials, study participants and therapists could not have been blind to the type of treatment they received or provided. Our assessments focused on blinding of assessors.

As mentioned above, we examined risks of detection bias for two types of data: participant reports and administrative data. Participants' reports were gathered by researchers, who administered interviews or questionnaires, usually in families' homes. Administrative data were compiled by professionals in social service or government agencies. Hence, these types of data could have different risks related to blinding (and attrition) within studies.

Collection of archival data (e.g., from juvenile justice records) might be considered to be blind; however, law enforcement officials were not always blind to group assignment and their knowledge that a youth was receiving or had received MST could have affected key decisions about youth (e.g., arrests, convictions, and incarceration; Leschied 2002a).

In some studies, post-treatment and follow-up measures were collected by MST therapists or researchers who were not blind to participants' group allocation. As Letourneau and colleagues (Letourneau 2009, p. 91) noted, it was often difficult for research assistants to remain unaware of treatment conditions, as they may have participated in randomisation procedures or received clues during assessment interviews with family members regarding the nature of the services these families received. Asscher 2013 reported that their research assistants were blind to the study hypotheses, but may have known which treatments families received and might have guessed the study hypotheses. This is common problem in field trials of complex psychosocial interventions.

In most studies the risks of detection bias were unclear. Only one study blinded assessors for all outcomes (Fonagy 2018).

Attrition bias

Overall attrition ranged from 0% to 72% and differential attrition ranged from 0% to 30%.

Few studies conducted thorough analyses of differences between cases retained in the analysis and those lost to follow up, and few provided data on differential attrition. The Asscher 2013 study is an exception, as these authors conducted analyses to determine whether data were missing completely at random (MCAR).

In some studies, attrition was not random (MNAR). Some studies excluded certain participants after random assignment.

Eligibility decisions should have been made prior to random assignment, but some ineligible cases were discovered after the fact. When that occurred, “the intention to exclude such participants should be specified before the outcome data are seen” (Higgins 2020). This was not done in several studies (e.g., Ogden 2004).

Differential attrition changed the composition of groups in some studies, particularly in the Borduin 1995 study. Although reports vary, it appears that White families were more likely than nonwhite families to be retained in this study: 67% (Henggeler 1996a) or 70% (Henggeler 1991, Borduin 1995a) of the larger sample of 200 families were White, compared with 76.1% of the 176 families who remained in the study (Schaeffer & Borduin, 2005; Klietz et al., 2010; Sawyer & Borduin 2011). Shown in Table 6, if 70% of the larger sample was White, then the difference between families retained in the study and those who refused to participate (76.1% vs. 25% White) is statistically significant (χ ² with Yates correction = 23.92, df = 1, p < .00001). If 67% of the larger sample was White, then all 24 refusers were nonwhite. (We found no information on 10 cases that were identified in a 1990 report on this study (n = 210; Borduin 1990a) but not included in the 1991 report (n = 200, Henggeler 1991) or in subsequent reports.)

In the Henggeler 1992 study, dropouts were more likely to be White, male, and living with neither parent (Table 7).

Overall, there was less attrition in administrative data than in participant reports. Figure 4 shows that three studies have high risks of bias related to attrition of administrative data, while eight studies have high risks of bias related to attrition on outcomes based on participant reports.

Intent-to-treat (ITT) analysis

To minimise biases introduced by attrition and differential attrition, ITT analysis includes all participants in the group to which they were randomly assigned, regardless of whether participants received the assigned treatment or provided data. Some studies failed to obtain or include data on participants who refused services, did not complete treatment, or did not complete treatment “successfully” (Schaeffer 2000, p. 36). These exclusions violated the principle of ITT analysis. When exclusions were made after outcomes were known, we “cannot exclude the possibility that the reason for exclusion was biased by [knowledge of] the results” (Chalmers 2007).

There was some confusion in study reports about what ITT analysis means. When studies reported the number of cases randomly assigned and reported a smaller number of cases “retained for intent-to-treat analysis” (sic), we used the first number as the basis for our assessment of ITT analysis (and as the denominator in calculating attrition rates).

Studies that systematically excluded cases after random assignment, on the grounds that families did not accept or did not complete treatment, were considered to have a high ROB in relation to ITT analysis, no matter how many cases were excluded. (We did not think it wise to set an arbitrary threshold for “acceptable” violations of ITT analysis.)

Assessment of studies' ability to support ITT analysis was complicated by conflicting reports on the number of cases randomly assigned in some studies. As mentioned above, an early report on the Borduin 1995 study indicated that 210 families were randomly assigned to groups (Borduin 1990a, p. 76), another reports statesd that 200 cases were randomly assigned (Henggeler 1991), and subsequent reports put that number at 176 (Borduin 1995a). Similarly, an early report indicated that 96 cases were randomly assigned in the Henggeler 1992 study, but subsequent reports described this as a study of 84 cases, with no mention of 12 missing or excluded cases (Henggeler 1993, Henggeler 1996).

As indicated above, four studies used yoked pairs of MST and comparison cases (to link the timing of the second assessment for comparison cases to the post-intervention assessment for MST cases; Henggeler 1992, Henggeler 1997, Henggeler 1999a, Henggeler 1999b). However, if one of the cases dropped out of the study, its mate was retained in the analysis. Some methodological experts thought this undermined the yoked design and argued that the unyoked cases should have been dropped to retain the benefits of random assignment (Littell 2006). In any case, investigators could have used sensitivity analysis to determine whether inclusion of unyoked cases affected results; to our knowledge, this was not done.

The exclusion of MST drop-outs was problematic, because these cases tended to have more negative outcomes (e.g., higher rates of arrest or conviction) than those that completed MST (Borduin 1995, Leschied 2002).

Full ITT analysis was provided in only 10 studies, and only for some outcomes. For example, Leschied 2002 provided full ITT analysis on outcomes derived from archival data, but response rates on psychosocial measures were below 60%. Asscher 2013 provided full ITT analysis on post-treatment measures of psychosocial functioning, but not on longer-term outcomes derived from administrative data.

Selective reporting (reporting bias)

Only four studies had protocols in a national or international registry. All four registered studies were conducted by independent investigators in the UK or The Netherlands. Only one of these studies (the START trial, Fonagy 2018) was registered prospectively (before data collection began) and this is the only study that fully reported all planned outcomes at all planned endpoints.

Selective reporting was evident when authors limited analyses to some cases, sites, outcomes, or endpoints for reasons related to outcomes. This includes a focus on treatment completers or successful cases (Borduin 1995, Schaeffer 2000) and omission of the site with the “poorest” outcomes (Ogden 2004).

Selective reporting was evident when authors only reported effects sizes for the outcomes that had demonstrated statistically significant results that favoured MST (e.g., Swenson 2010, p. 504). Some studies limited follow-up assessments to outcomes that had previously favoured MST (e.g., Letourneau 2009), when a fuller assessment would have provide more completed data (including, perhaps, new findings in favour of MST). Some studies reported selected subscales and omitted reporting of other subscales in the same instrument (e.g, Wagner 2019 reported results for the MPRI bonding scale, but not the MPRI aggression or maturity scales).

As shown in Figure 2, outcome data were collected, but not reported at several endpoints in some studies. Butler 2011 did not report data collected at 30 and 42 months, but provided this data to us on request. Henggeler 1999a did not report main outcomes obtained at 18 months; Henggeler 1999b did not report outcomes at 30 months; Henggeler 2006 did not report outcomes at 18, 24, 36, 48 and 60 months; Letourneau 2009 did not report outcomes at 18 or 122 months (these authors did not respond to requests for missing data). There is reason to think that nonreporting of outcomes at these end points may be a biased decision, related to null results: For example, although the full report on 10-year follow-up on the Letourneau 2009 study is not available, the abstract stated that “Between-groups analyses indicated that MST-PSB was no more effective than TAU on most criminal and noncriminal outcomes” (Sheerin 2017).

To investigate the possibility of outcome reporting bias, we present an outcome matrix, following the work of Dwan 2010. Results show that partial reporting of outcomes was common in MST trials (Figure 5).

OPEN IN VIEWER

Figure 5

Reporting status by study and outcome

To investigate the possibility of publication bias, we produced a funnel plot, using data from the largest pairwise meta-analysis in our review. Results, shown in Figure 6, are not easy to interpret. Eleven studies provided 12 independent estimates of effects of MST on out-of-home placements of all types at one year after random assignment. Studies conducted by MST program developers in the USA tended to have effects favouring MST, while those conducted outside of the USA by independent teams had more negative effects. Given this pattern, it is difficult to tell whether publication bias was an issue.

OPEN IN VIEWER

Figure 6

Funnel plot: out-of-home placement, one year after random assignment (from Analysis 1.1, standard error by odd ratio)

Standardised follow-up periods

As mentioned above, some studies had unstandardised observation periods (i.e., cases observed for different lengths of time, with no adjustments for variations in the length of observations). In some studies, this range was quite substantial. For example, the length of the observation period in the Borduin 1990 was an average of 37 months, but the range was 21 to 49 months. The Henggeler 1992 study had a mean observation period of 59 weeks, with a range of 16 to 97 weeks. Authors reported the percentage of successes/failures on several measures, including all available observations, regardless of the length of observation. For example, the percentage of recidivists among sex offenders in the Borduin 1990 study included one case observed for 21 months and one observed for 49 months; we do not know whether the 21-month case recidivated within the next 28 months, hence its outcome is not comparable to cases observed over a longer periods of time.

In the Henggeler 1997 study, archival data were collected at a fixed point in time (1.7 years after the end of the project) and then annualized to account for variations in the follow-up observation period (e.g., by computing number of rearrests per year observed). Since recidivism rates tend to decline over time, cases with longer follow-up observation periods are likely to have lower annualized rates than those with shorter observation periods.

We requested fixed-interval data (1-year follow-ups) from authors, and received this for two studies (Butler 2011, Leschied 2002).

One study had somewhat different observation periods for MST and control cases (Timmons-Mitchell 2006).

Validated outcome measures

Most self-report measures were based on standardised instruments and measures used in previous studies. Questions can be raised about the suitability of some instruments in certain samples (e.g., the self-esteem scale used in the Henggeler 1999b study was developed for use with Mexican-American youth (Simpson 1992), but this study's sample was only 1% Hispanic).

Some standardised instruments were adapted for the purposes of a particular study; thus, there were cross-study variations in the content of some “standardised” measures. For example, in the Ogden 2004 study, back-translation methods were used for some measures (e.g., the CBCL) and not others; authors' reports on the internal consistency of these modified scales indicated that this was a reasonable approach. Letourneau and colleagues modified some standardised instruments so that all outcome measures referred to participants' experiences in past three months, instead of varying time frames (Letourneau 2009); the adequacy of this approach was bolstered by data on the internal consistency of the measures used in this study sample.

In earlier studies (those conducted before 2005), authors rarely reported information on the performance (e.g., internal consistency or inter-rater agreement) of standardised instruments in their study samples. This reporting was more complete in later studies.

Conflicts of interest (COI)

We rate the likelihood of bias related to two types of COI: situations in which (1) investigators could benefit if results favoured MST and (2) investigators could benefit if results favoured a control/comparison group. We found conflicts of the first type, but not the second.

More than half (13) of the studies in this review were conducted by MST developers, board members and shareholders, or former shareholders of MST Services, Inc. (or MST Services LLC), the for-profit consulting firm that promotes, disseminates, and licences MST services and/or MST Associates LLC, the organisation that provides training for MST-PSB. Some authors served as the clinical supervisor for cases in the MST arm of their study (e.g., Borduin 1990, Swenson 2010).

Ten studies were conducted by independent teams, but six of these teams did not provide COI statements. One independent study had a high risk of COI (Asscher 2013) and three studies (Butler 2011, Fonagy 2018, Sundell 2006) were assessed to have low risk of COI (details are provided in Characteristics of included studies).

ROB and other study characteristics

Studies conducted in the USA had more serious (high) risks of bias than those conducted outside of the USA, even after we excluded ratings of COI (Table 8).

Out-of-home placements

Eleven studies provided dichotomous data on out-of-home placements at approximately one year (9 to 18 months) after random assignment. Of these studies, seven were conducted in the USA with MST program developers and four were conducted outside of the USA by independent teams. One of the USA studies (Glisson 2010) provided data on two nonoverlapping samples. Results, shown in Analysis 1.1, were sorted by study date (earliest to latest) within two subgroups, and effect sizes (odds ratios) were pooled within subgroups and overall.

Results of USA/developer studies indicated that odds of out-of-home placement for MST cases were about half of the odds for control cases (pooled OR 0.52, 95% CI 0.32 to 0.84), a statistically significant effect (P = .007). Note that there were substantial variations between the studies within this subgroup in terms of their effect sizes (heterogeneity χ ² = 22.24, df = 7, P = 0.002; I ² = 69%).

The non-USA/independent studies in this analysis included two smaller trials with wide CIs and two larger studies with more precise estimates. Across these four studies, the odds of placement were slighter greater for MST cases than for controls at one year after referral (OR 1.14, CI 0.84 to 1.55), but this pooled effect iwas not significantly different from no effect (P = .40). Findings (no evidence of effects) were consistent within this subgroup of studies (χ ² = 1.49, df = 3, P = 0.69; I ² = 0%).

Overall effects of MST on out-of-home placements at one year were heterogeneous (χ ² = 36.18, df = 11, P = 0.0002; I ² = 70%), with significant differences in results between the two subgroups (USA/developer-involved versus non-USA/independent studies; χ ² = 7.42, df = 1, P = 0.006; I ² = 86.5%).

It is important to note that the “base rates” for placement differed in USA and non-USA studies: In the absence of MST, 40% (248/619) of all control cases in USA/developer studies experienced out of home placement within the first year, compared with 17% (102/599) of control cases in non-USA/independent studies (Analysis 1.1), a substantial difference (d(probit) = 0.70).

Analysis 1.2 shows results for placements at approximately 2.5 years (19–40 months) after random assignment. Analysis of data provided by two studies conducted in the USA by MST developers and four studies conducted outside of the USA by independent teams showed no evidence of effects of MST on placements within or across the two subgroups of studies (pooled OR 0.81, CI 0.54 to 1.41; P = .29). Overall results were homogeneous, and the test for differences between subgroups was not significant.

Only two studies provided data on placement rates at 4–5 years (41–60 months) after random assignment; both were non-USA/independent studies and both found no evidence of effects of MST on out-of-home placement rates at this point in time (Analysis 1.3, pooled OR 0.91, CI 0.51 to 1.62; P = .75).

Ogden 2004 reported data on out-of-home placements for only three of four sites, and some MST cases were replaced with others cases in this study. After outcome data were collected, authors declined to release results for the one site with the “poorest MST outcomes”, citing concerns about possible “misinterpretation” of results (Ogden 2004). To estimate the range of effects that could have been detected if data on all cases had been reported in this study, we conducted a best-case/worst-case analysis. Results, shown in Analysis 1.4, indicate that MST could have had a wide range of effects in this study, from significant reductions (OR 0.16) to substantial increases (OR 1.71) in out-of-home placements at two years after random assignment. These results—and concerns about high risks of bias in this study—suggest that results of this study should be viewed with caution.

Six USA/developer studies provided data on effects of MST on the duration of out-of-home placements at one year after random assignment, but effect sizes can be calculated for only three of these studies (others are missing SDs and/or CIs). Shown in Analysis 1.5, three USA/developer studies showed overall reductions in the length of placements at one year (pooled SMD −0.43, CI −0.66 to −0.21; P < .001), but these results were not replicated in two larger, non-USA, independent studies (pooled SMD −0.03, CI −0.16 to 0.11; P = .71). Analysis of “base rates” showed that control cases experienced longer placements in the USA (weighted mean of 71 days, k = 6) than in non-USA studies (weighted mean 26 days, k = 2).

At about 2.5 years, one USA/developer study found no significant effects on length of placements, but an independent study showed that MST reduced the duration of placements in Sweden (Analysis 1.6).

At 4–5 years, two non-USA/independent studies found no evidence of effects of MST on duration of placements (Analysis 1.7, pooled SMD 0.05, CI −0.10 to 0.19; P = .52).

Correlated effects (CE) models, shown in Table 9, included all available effect sizes for placement outcomes from all endpoints. Overall, MST reduced the odds that youth were placed outside of their home (OR 0.66, 95% CI 0.43 to 1.03; P = .06), but these results were greater in USA/developer studies (OR 0.50, CI 0.24 to 1.05; P = .06) and in studies with high risks of bias related to ITT analysis (OR 0.44, CI 0.22 to 0.87; P = .02). Similarly, MST reduced the duration of out-of-home placements (SMD 0.24, CI 0.41 to 0.08; P < .01), with larger effects in USA/developer studies (SMD −0.31, CI −0.53 to −0.09; P = .02) and in studies with high risks of bias due to deviations from ITT analysis (SMD −0.23, CI −0.50 to 0.04; P = .09). When USA/developer or ITT moderators were included in CE models, the main effects of MST on placements (intercepts) were not statistically significant.

Our other moderators (time, attrition, other ROB variables) were not related to placement outcomes, or these relationships could not be reliably estimated (df < 4; Table 9).

Overall, MST effects on placement have wide PIs, indicating that future studies can expect to find substantial decreases or increases in placement outcomes (PI for OR 0.24 to 1.84, PI for SMD −0.60 to −0.12; Table 9).

The same patterns emerged when we converted ORs to SMDs and included all effect sizes in CE models (Table 10). In the combined CE analysis (with 56 ES from 17 outcomes), the overall effect of MST was −0.27 (CI −0.43 to –0.12, P < .01; PI, −0.72 to –0.17). Moderator analysis showed that effects of MST on placement were significantly greater in USA/developer studies (ES −0.33, CI −0.57 to −0.10; P = .01), and in studies with high risks of bias related to deviations from ITT analysis (ES −0.33, CI −0.57 to −0.10; P < .01). Again, when these moderators were included in the analysis, overall effects of MST (intercepts) were not significant.