Mentoring Interventions to Affect Juvenile Delinquency and Associated Problems: A Systematic Review

Understanding Mentoring Effects

While a relatively large number of studies with some minimal evaluation design features have been found and utilized in prior evaluations of mentoring, there are characteristics of this field that have limited how informative reviews and meta-analyses have been. For example, the most often considered intervention feature is the extent of matching on similarity of demographic characteristics and interests of the mentor and recipient and that the quality of the personal relationship not be marked by dissatisfaction (referred to as mentor relationship quality; see Rhodes, 2005; DuBois et al., 2011). Yet, these are merely post-hoc identified variables differentiating effect sizes. They do not directly or indirectly indicate what are the processes through which mentoring has its effects or suggest what it is about mentoring that might make it different from many other helping relationships (MENTOR/National Mentoring Partnership, 2009). A major limitation of the field, and perhaps of progress in understanding more about how and why mentoring shows positive effects is the lack of specificity in describing the activities comprising a given mentoring intervention and perhaps more importantly tying activities and practices to theorized key processes through which positive impact is thought to occur. This limits ability to compare program features that may explain variations in effects as well as limiting ability to tie programs to theories about interpersonal processes that could explain how mentoring has impact (DuBois et al., 2011; Rhodes, Grossman, & Resch, 2000). Among the reports and reviews there is considerable variation in what activities are considered mentoring essentials and which are optional (DuBois & Karcher, 2005; (MENTOR/National Mentoring Partnership, 2009). At the same time there is not much attention to and little certainty about what constitutes a mentoring intervention and might distinguish mentoring from other helping relationships (Rhodes et al., 2000). In addition, more understanding of key processes of a given program could improve ability to compare across programs and cumulatively point toward the important or necessary components that define a program as mentoring (Roberts, Liabo, Lucas, DuBois, & Sheldon, 2004).

Limited intervention description may be because mentoring arose as a voluntary and “indigenous” approach to youth intervention. Thus, many mentoring efforts arose within a given setting without intention to formalize and standardize performance and activities. The practitioners who developed their particular approach may have had less training in and interest in formal evaluation features. As a field of intervention services and as a research focus, there appears to be mixed interest in facilitating more formal operations that will yield more informative and comparable results (MENTOR/National Mentoring Partnership, 2006). Also, because one common basis for mentoring is a view that the positive influence of an interested person providing a supportive relationship is what is helping, there may well be less interest in trying to specify what activities and processes constitute mentoring and what among these could explain any benefits derived. For all of these reasons formalized protocols and systematic training approaches may not have been a priority. Consequently the body of research is remarkable in the limited emphasis on systematic description of intervention content, description of intended processes through which effects are expected, and in important features of implementation and providers. There seems to remain limited valuing of and perhaps even some reluctance to aim for continuity across the field or specificity in applying and describing mentoring efforts that might facilitate scientific understanding of effects. Hence, there are few training, implementation, and dosage parameters that can be identified as having consensus. There are few indications of what is considered essential or critical for mentoring and helpful in distinguishing mentoring from other helping relationships and approaches. Similarly, the reports reviewed here continue an unfortunate tradition of having limited information by intervention science standards and are less informative than needed about what may account for benefits accrued. Overall, greater interest in relating to a common set of principles, theorized processes, or requisite structures and components would seem an advance that could serve the field well. Thus, one of the goals of this effort and one that is a different emphasis than prior meta-analyses was to code, to the extent possible, comprising activities, mentor selection/motivation, and training or implementation features as viewed from an intervention research lens. A second was to theorize four processes that are often mentioned or pointed to as how mentoring affects youth positively and that in total can distinguish mentoring from other forms of helping interventions. Of interest was that this combination do not as a group also from the critical processes of other helping interventions such as teaching or psychotherapy. Emphases on these two interests (intervention features and key processes) could potentially help advance understanding of mentoring, effects found, and potential for further study and use.

Important Intervention Features Affecting Mentoring Impact

One aspect of mentoring intervention characteristics given substantial attention is the implication that a strong personal relationship between the mentor and mentee is a key to any benefits derived (DuBois et al., 2002; Rhodes et al., 2006). Thus one advance in the field is to assess how positive and engaging the relationship is between the mentor and mentee (Rhodes et al., 2006). For example, DuBois et al. (2011) report larger effect sizes when matching of mentor and youth was based on shared interests; presumably this improves the likelihood of a good relationship. In most cases, a corollary is that the mentor is undertaking this activity, not as a professional in the helping or social service professions, but because of personal interest or sense of duty, often as a volunteer (Rhodes, 2002). When a person with professional background or duties to provide such services offers mentoring, the emphasis is more on the relationship and the personal interest in the mentee than on specific skills, activities, or formal protocols. Thus, it has been noted that one limitation of mentoring may be that providers may be less accountable as they are volunteers and/or may not be well prepared for challenges of developing and maintaining a relationship with sometimes challenging and less appreciative youth (Grossman & Tierney, 1998). In contrast, it may be that motivation that is not personal, that is for professional advancement or as a paid position might be expected to lessen the personal commitment and connection thought to spark effective mentoring. More understanding of how different reasons for undertaking mentoring influence effects would help with understanding effect variations and provide direction for improving impact. We test for differences by motivation of mentor for engaging in this work.

A second area of some importance in understanding how to direct mentoring efforts is the effect of structuring of the effort and expecting fidelity to an approach. While it is increasingly recognized that training in skills and expectations are important for mentoring, there is much less clarity about what is important to expect. Mentoring has been characterized as growing out of a mentor's commitment to youth (Rhodes, 2002) with the accompanying implication that structuring the activities and processes to be ensured would detract from the individualistic authentic engagement that carries the benefits. In contrast, research on other forms of intervention have not supported such a view, pointing to more clear expectations and fidelity prescriptions as promoting larger effects (Tolan & Gorman-Smith, 2003). Thus, the extent to which there is emphasis on following these procedures and principles thought to be helpful should relate to effect levels. Therefore in this review we examine if assessment of fidelity relates to effect size.

A third question of importance about mentoring is the relative value of mentoring as a high-risk selective and/or indicated approach rather than as a universal intervention (Tolan & Guerra, 1994). Mentoring studies have been applied to high-risk, identified, and general populations of youth. There is some indication the effects might be greater for higher risk youth, although the results are not fully consistent (DuBois et al., 2011). Some have argued that mentoring represents an alternative view about youth risk, a focus on promoting healthy or positive development through strengthening abilities rather than minimizing exposure to risk or remediation of undesirable behavior and characteristics (Jekielek, et al, 2002). Also, it may be that mentoring that is developed for and applied to high-risk youth has impact for that population that programs for non-delinquent or general population youth do not. There is evidence that preventive effects for high-risk youth may be quite different from those accrued for the general population (Tolan & Gorman-Smith, 2003). For example, it may be that mentoring is not valuable in affecting delinquency or related outcomes of high risk youth because it is not structured enough and focused on multiple risk factors thought to drive that behavior (Lipsey & Wilson, 1998). Thus, there is a policy interest in whether targeting high-risk youth (selective inclusion) is useful. Therefore, the review undertaken here was focused on youth defined as high-risk for or already engaged in delinquency (Tolan & Gorman-Smith, 2003).

Similarly, as others have noted it is common for mentoring to occur as part of a multi-component program, whether as one of several components or as a central focus augmented by additional supporting activities (Aos et al., 2004). This leaves open an important question of the extent to which effects attributed to mentoring might actually be coincidental inclusion with other effective components. It also leaves undifferentiated to what extent it matters if the delivery with other components is simply as one of a set of substantial program features or if the program is primarily mentoring with some augmentation to help support and enhance the mentoring impact. These questions of interest suggest coding of these features, where discernible, might improve understanding of the value of mentoring.

Identifying Potential Key Processes Defining and Differentiating Mentoring

In addition to these features of intervention organization that have been of interest in characterizing mentoring as a field of intervention and in furthering the evaluation knowledge about mentoring, there is an important but almost unattended to issue of what processes are typical of and constitute mentoring as an intervention. Are there activities or underlying purposes of activities that are common to mentoring or that might vary across mentoring programs and in doing so help account for differences in effects? As noted above, theoretical summaries of the field and attempts to relate mentoring to prevention science, developmental psychopathology, and/or youth development literature in general have suggested some likely key features of mentoring (Lipsey & Wilson, 1998; McCord, 1992; Tolan & Guerra, 1994). These processes are differentiated from the attention to the mentor-mentee relationship that has dominated evaluation of mentoring (Rhodes, 2005, DuBois & Karcher, 2005). The latter represents an aspect of connection that while important as a basis for mentoring is a common basis for any influence relationship.

Through systematic review of theoretical organization of process models of mentoring (e.g., (MENTOR/National Mentoring Partnership, 2009; Rhodes, 2005), indices utilized by DuBois et al. 2002 in examining how effect size of mentoring related to score on best practices index, components described in programs with significant effects (e.g., Davidson & Redner, 1988), and qualitative analyses of mentoring relationships (e.g. Deutsch & Spencer, 2009) we organized a set of processes that seemed to occur across mentoring programs, whether explicitly described or implicit in the activities utilized. In addition, we compared mentoring to other helping interventions to identify distinguishing features. For example, mentoring is distinguished from psychotherapy by the non-professional relationship and the lack of emphasis on mental health problem alleviating. From these multiple bases we identified four processes as central to mentoring: 1) identification of the recipient with the mentor that helps with motivation, behavior, and bonding to conventions; 2) provision of information or teaching that might aid the recipient in managing social, educational, legal, family, and peer challenges; 3) advocacy for the recipient in various systems and settings; and 4) emotional support and friendliness to promote self-efficacy, confidence, and sense of mattering (DuBois et al., 2002; DuBois et al., 2011; Rhodes et al., 2002). These processes are frequently mentioned individually as potential bases for mentoring benefits. More recently some attention has been given to how advocacy within mentoring can affect impact. DuBois et al. (2011) report when advocacy was considered a mentor function effect sizes averaged .07 standard deviation units larger than when not. Also, several of the more fully described efforts point to one or more of these processes as intended elements of the mentoring. Understanding of whether emphasis on such processes relates to effects is one intended contribution of this review. Therefore, we coded studies for evidence of each key process driving or comprising the intervention to permit examining how their inclusion may have affected outcome.

Prior Evaluation of Features Affecting Mentoring Impact

DuBois et al. (2002, 2011) recognized many of the issues related to advancing and deepening understanding of mentoring effects and incorporated coding of several of these features into their meta-analysis. In DuBois et al. (2002), they denoted an index of what could be considered best practices in youth mentoring based on recommendations of prior reviews and recommendations for establishing effective mentoring programs, such as the National Mentoring Working Group (1991) and coded to the extent possible from source data, each intervention report (DuBois et al., 2002). They included 11 program features to mark how methodic inclusion in the program was, whether mentors and mentees are matched on demographic characteristics, how structured or prescribed activities were, and the frequency or extent of contact. These codes were then amalgamated into an overall index of extent of desired features. While this represents an informative advance about how the extent of features considered useful for good mentoring related to effect size, because it is a single score across many areas it cannot indicate the importance of specific features. Also, it may have obscured how many of the reports did not have adequate reporting to fully assess the 11 features.

We attempt to build on efforts of DuBois et al. (2002) to code theoretically and empirically linked valued characteristics, activities, and organization by focusing on the moderating effects of each of several key features related to 1) selectivity in inclusion (high risk versus universal or no selectivity within the population); 2) explicit attention to presence of four key processes such as modeling, emotional support, advocacy, and teaching; 3) whether or not mentoring is a stand-alone approach in that study or was undertaken along with some other components: 4) the motivation of the mentors in participating; and 5) the extent to which quality of work and fidelity were assessed or emphasized. This coding was considered useful for suggesting what might differentiate mentoring from other similarly intended youth interventions. Despite prior identification of specificity of such features as a major limitation of the mentoring literature (Tolan & Guerra, 1996), we did not find much improvement over time in the ability to determine details needed to code many of these features for this review. We had to limit our analyses to those features that could be coded for enough studies to enable some useful comparison.

3.1 Criteria for inclusion and exclusion of studies in the review

Only studies that satisfy all of the following inclusion criteria and none of the following exclusion criteria were included in this review:

Outcomes measured

We focused this systematic review on outcomes related to juvenile delinquency. We included studies with outcome measures of juvenile delinquency, reported by the individual or by others, or derived from archival sources such as arrest or juvenile court records. We also included studies focusing on precursors of delinquency such as aggression or high levels of externalizing problems and studies with two outcomes that are correlated with and frequently co-occur with criminal involvement or delinquency risk (drug abuse and academic achievement/ school failure). As noted above, the specific terms for each outcome are provided in Table 1.

Types of participants

Juvenile delinquency is typically defined as antisocial or criminal behavior by persons under age 18 (Tolan, 2002). In this systematic review of mentoring interventions, we included studies that involved youth who were included because they were currently showing behavior that would constitute juvenile delinquency or were identified and included because they were “at-risk” for juvenile delinquency. At-risk is defined as the presence of individual or ecological characteristics that increase the probability of delinquency in later adolescence or adulthood (Tolan, 2000). Ecological characteristics include family and parenting influences on behavior, residence in neighborhoods with high levels of poverty or crime, exposure to gangs, and other social setting factors (Tolan & Gorman-Smith, 2003). Individual characteristics include high scores on screening measures for aggression, evidence of oppositional defiant or conduct disorders, school failure, or attitudes and beliefs consistent with elevated use of aggression or antisocial behavior (Farrington, 2004). Demographic characteristics were not considered as designating at-risk for consideration for inclusion here. Thus, a study that targeted a demographic group even if doing so because they are considered at risk was not included unless selection met our criteria otherwise.

Intervention Type

We included interventions focusing on prevention and treatment (referred to as selective and indicated population interventions). In the initial phase of study selection, we sought out any studies that described their interventions as mentoring, that mentioned mentoring as any part of their intervention strategy, or had interventions characterized by any of the four characteristics noted above, whether or not they specifically mentioned mentoring.

Regarding the defining characteristic of absence of formalized role inequality, previous reviews have differed on the inclusion of studies using professionals as mentors. DuBois et al. (2002) excluded interventions using professional providers, with the exception that some studies that employed mental health professionals as mentors were included under certain conditions (see DuBois et al., 2002; Rhodes, 2002 for those criteria). This appears to also have been the approach used in the updated meta-analysis by DuBois et al., 2011. We differed from these prior reviews by including studies with mental health providers as mentors if their involvement was unstructured or limited to a non-specific or support intervention (not psychotherapeutic). Functionally this means inclusion here of some critical studies for the current focus that were not included in the DuBois review, such as the McCord Cambridge-Somerville study (McCord, 1978, 1979).

We then excluded studies in which the intervention was explicitly psychotherapeutic, behavior modification, or cognitive behavioral training. Although we included studies in which mentoring was done as a part of another structured intervention, those studies that were conducted without providing results for the mentoring intervention separately were coded as including either an additional primary intervention (i.e., a major component in addition to mentoring) or an additional secondary intervention (i.e., a minor component in addition to mentoring).

In addition to requiring that studies investigate the effects of a mentoring intervention, as described above, we followed three additional criteria based on those used by Lipsey and Wilson (1998) in their meta-analysis of intervention effects on delinquency. We only included studies that measured at least one quantified outcome variable for the outcome of interest among the four considered here and that provided sufficient data to allow calculate an effect size and decipher its direction. When studies measured a delinquency-related outcome but did not report sufficient detail to allow calculation of an effect size, we attempted to contact the author to obtain additional information. Because of access to the Aos and Lipsey databases we had a relatively complete rendering of the studies from which such information could be extracted. There were, therefore, very few studies that we were uncertain about whether additional information was obtainable.

Research Design

The second criterion for inclusion in this review was that the study design involves a comparison that contrasted an intervention condition involving mentoring with a control condition. Control conditions could be “no treatment,” “waiting list,” “treatment as usual,” or “placebo treatment”. To ensure comparability across studies we made an a priori rule to not include comparisons to another experimental or actively applied intervention beyond treatment as usual. However, there were no such cases among the studies otherwise meeting criteria for inclusion.

We coded studies according to whether they were experimental or quasi-experimental designs. To qualify as experimental or quasi-experimental for the purposes of this review, we required each study to meet at least one of three criteria: 1) random assignment of subjects to treatment and control conditions or assignment by a procedure plausibly equivalent to randomization; 2) Individual subjects in the treatment and control conditions were prospectively matched on pretest variables and/or other relevant personal and demographic characteristics; 3) Use of a comparison group with demonstrated retrospective pretest equivalence on the outcome variables and demographic characteristics as described below:

Randomized controlled trials that met the above conditions were clearly eligible for inclusion in the review. At the other end of inclusion eligibility, single-group pretest-posttest designs (studies in which the effects of treatment are examined by comparing measures taken before treatment with measures taken after treatment on a single subject sample) were never eligible. A few nonequivalent comparison group designs (studies in which treatment and control groups were compared even though the research subjects were not randomly assigned to those groups) were included. Such studies were only included if they matched treatment and control groups prior to treatment on at least one recognized risk variable for delinquency, had pretest measures for outcomes on which the treatment and control groups were compared and found to be essentially equivalent. We required that non-randomized quasi-experimental studies employed pre-treatment measures of delinquent, criminal, or antisocial behavior, or significant risk factors for such behavior, that were reported in a form that permitted assessment of the initial equivalence of the treatment and control groups on those variables.

Time Period and English Language Criteria

We limited the review to those studies conducted within the United States or another predominately English-speaking country and reported in English. This was because we did not have resources for translation of studies not published in English and the vast majority of programs were conducted in the United States. Juvenile subjects did not need to speak English. A study conducted in the United States or Canada with resident Hispanic youth, for example, could have been included.

We limited the review to studies published since 1970. The time frame between 1970 and the present (time of completion of search to conduct coding, 2011) is consistent with start of the time interval used by the review of the literature on delinquency conducted by Lipsey and Wilson (1998) and others. This also is the time period for most almost all the available studies with the necessary information and design features to be included in this review.

Coding of Article Characteristics

We double-coded 20% of the new articles (N = 32), and calculated inter-coder reliability coefficients for study type (e.g., randomized trial), study quality, participant selection criteria (e.g., individual or behavioral risk), mentor motivations (e.g., survivor of abuse, professional development), and intervention components (e.g., modeling, teaching) using Cohen's kappa. We found high reliabilities for study type (κ = 1.0), study quality (κ = .93), and selection criteria (κ = .81). Coders easily determined some mentor motivations such as personal experience that connected to the youth needs (e.g. experienced abuse) (κ = .90), but were less certain with topics such as civic duty or professional development (κ = .68). Not all categories were coded in the random sample of studies that were double coded. For example, of the mentoring components (modeling/identification, teaching, and emotional support) only modeling was found in the studies randomly selected for double coding. Final kappa reliabilities all were above .6, a level Landis and Koch (1977) suggested represented full agreement. Coders sought consensus with their supervisors, particularly on difficult-to-code categories such as mentor motivations. If this could not resolve differences then author Schoeny made a decision about categorical coding.

Effect sizes for outcomes were also double-coded for 20% of the new articles. There were no substantial variations in these (r = .99) with only one disagreement. As with other coding decisions we first attempted to resolve bases for differences (e.g. technical inconsistency that if corrected removed difference). We had a protocol in place to then structure discussion of differences to attempt to reach consensus. If necessary a decision would then be made by either the first author (Tolan) or author Henry. Given the level of agreement we did not have to proceed past the technical comparison and a brief discussion to reach consensus.

We conducted a separate meta-analysis for each outcome (delinquency, aggression, drug use, academic achievement). Each grouping of studies was based on the outcome, such that some studies might be included in more than one meta-analysis due to measuring more than one outcome. Thirteen studies reported more than one outcome, four of which had three outcomes. A single outcome measure was used for each study for a given outcome category. No studies reported multiple measures of a single outcome (e.g., multiple measures of delinquency or aggression).

Statistical Procedures.

Effect Size Calculations: For this study we used inverse-variance meta-analysis with a random-effects model, performed and plotted through the metagen package in the R statistical language. The random effects model addresses the research question of whether the average effects of an intervention in the population are significantly different from zero (Bailey, 1987; Raudenbush, 1994).

The inverse variance method, as its name suggests, weights individual studies by the inverse of variance of their effect size. Thus, this method requires the calculation of standard errors of the effect sizes. For this purpose, we estimated variances for each effect size according to Hedges and Olkin's (1985, p. 86) Formula 14:

σ ^ d i 2 = ( n e + n c ) ( n e * n c ) + ( d i 2 ) ( n e + n c )

Where σ_di ² is the estimated variance of the effect size, n _e is the number of experimental subjects, n _c is the number of control subjects, and d _i ² is the square of the effect size of the study.

The standardized mean difference effect sizes of the interventions under evaluation were calculated in units of Hedges’ (1981) g. For studies reporting means, standard deviations, and Ns of numeric data, the effect size was calculated by dividing the treatment difference less the control difference over the pooled treatment and control standard deviation:

S M D = ( ( M E 2 - M E 1 ) - ( M C 2 - M C 1 ) ) ( ( N E 2 - 1 ) S E 1 2 + ( N C 2 - 1 ) S C 1 2 ) ( N E 1 + N C 1 - C ) ,

where: M = mean S = standard deviation E = treatment C = control 1 = pretest 2 = posttest.

For studies that reported dichotomous outcomes, we calculated odds ratios and converted them into an equivalent standardized mean difference effect size estimate (Lipsey & Wilson, 1998). Chinn (2000) noted that dividing the natural log of an odds ratio by π/√3 produces an excellent approximation of the standardized mean difference effect size.

We also applied a correction to all effect sizes that compensates for small sample bias:

g * ≈ ( 1 - 3 4 ( n 1 + n 2 ) - 9 ) g

We examined funnel plots from each meta analysis for visual evidence of asymmetry, and conducted Egger tests (Egger, Smith, Schneider, & Minder 1997) to obtain a statistical test for asymmetry. The Egger test fits a regression of the normalized effect estimate (estimate divided by its standard error) against precision (the reciprocal of the standard error of the estimate).

We conducted analyses to determine whether the effects of the mentoring interventions varied by five key aspects of the intervention approach and characteristics. Potential moderators that were tested were:

1)

selectivity in inclusion (high individual risk, high environmental risk, or no such selectivity)

Inspection of the coding across studies indicated that we had to simplify some moderation analyses due to sparse or no studies noting a particular characteristic of interest. For selection of participants, none of the interventions were coded as a universal, thus, under selection we could only test for moderation by the presence or absence of selection for individual risk and selection for environmental or ecological risk. We could not consider personal experience as a motivation as there were no studies in which this was measured or was able to be coded. Thus moderation tests of mentor motivations were conducted separately for presence or absence of civic duty and for professional development as motivation.

Only the tests of inclusion of other interventions with mentoring included all 46 studies. Other moderator analyses were limited by whether coders could determine whether the moderating factor was present or absent. The analysis of whether motivation by civic duty significantly moderated effect sizes included 36 studies, which was the smallest number of studies in any of the moderation analyses.

To conduct the moderated analyses we utilized all studies across the four outcomes to calculate an overall effect size by moderator condition (i.e., the mean of all effect sizes reported in each study). This was done because of the limited number of studies for testing moderators available even if examined for each outcome separately. We also reasoned that the interest was in testing moderation of mentoring for studies of delinquency and/or the related outcomes rather than for each specific outcome. That is, this meta-analysis is focused on youth at-risk for delinquency from the view that the four outcomes are related in sharing risk factors and likely impact of mentoring features. This approach has been used in other meta-analyses where multiple outcomes are of interest (see DuBois et al., 2011). In addition, given the power strain moderation analyses can impose on data sets limited in size like this one, as has been done by others we utilized a p level of .05 (one-tailed test). This standard was also used for these analyses because in each case we expected larger effects if the moderator was present, and the specific order of the levels of the moderator was not at issue. This is equivalent to a two-tailed p < .10 which has been justified given the power challenges for moderation effects (Wilson & Lipsey, 2007).

We tested for moderation with meta-regression analysis using the rma function in the metafor package in R (Viechtbauer, 2010). Each meta-regression analysis employed a random effects model that included terms for the moderator under consideration and a term representing whether the study was a randomized design or a quasi-experimental trial. The significance tests are one-tailed Z-tests.

We also conducted sensitivity analyses to assess the effects on conclusions of changes made to the inputs of an analysis (Morgan & Henrion, 1990). Accordingly, we conducted analyses to determine (1) the consistency of effect sizes obtained with different outcome variables, and (2) the consistency of outcomes within different levels of moderated analyses.

4.1 Main Effect Meta-Analyses Results

In the updated review's first phase of the literature search we identified additional studies to accumulate a total of 164 studies that were further evaluated for basic criteria for outcome and intervention type. Of these studies, 58 (34%) were determined to have none of the target outcomes. The remaining 107 were subjected to further scrutiny in order to determine their methodological suitability for the meta-analysis. Of these 53 (33%) had research designs that did not meet minimum quality standards for inclusion and 6 (4%) did not provide sufficient information for calculating effect sizes related to the outcomes in question This left 46 (28%) studies that were included in the quantitative review. The 118 excluded studies can be found in Table 7.

Table 3 provides details on the 46 studies selected for the meta-analysis, including citation, sample characteristics, design type, component and intervention information obtained for moderation analyses, and basic findings. Of the 46 studies included, 27 were randomized controlled trials and 19 were quasi-experimental studies involving non-random assignment, but with matched comparison groups as was described above. Twenty-five studies reported delinquency outcomes, 25 reported academic achievement outcomes, 6 reported drug use outcomes, and 7 reported aggression as an outcome.

Prior to calculating the mean effect size, we evaluated the heterogeneity of study effect sizes using multiple homogeneity measures, standard errors, and associated probability levels, including Cochrane's Q, and I ² (Higgins, Thompson, Deeks & Altman, 2003). Cochrane's Q is an indicator of heterogeneity that is distributed as a chi-square. Significant values of Q indicate heterogeneity. The degree of heterogeneity can be seen in the I² statistics. This indicates the approximate proportions of variance across compared studies that are due to heterogeneity of effects. ¹

We inspected forest plots of the effects and confidence intervals to explore for potential outlying studies. Our procedure was, after identifying possible outlying studies we repeated the meta-analyses, in order to determine whether removal of up to five outlying studies would reduce or eliminate the heterogeneity.

As can be seen in Table 4, heterogeneity of effects was substantial for delinquency and academic achievement. Also, examination of forest plots and re-analysis with removal of outlying studies did not reduce appreciably the heterogeneity of effects of mentoring for either delinquency or academic achievement. It seems evident there is substantial heterogeneity among studies in effects for delinquency and academic achievement.

In order to assist in understanding the heterogeneity in effect sizes, we conducted an analysis to determine whether the effect sizes differed substantially between randomized controlled trials (RCTs) and quasi-experimental designs. Using meta-regression with study design as the predictor, we found that although effect sizes were numerically larger in RCTs for all outcomes except drug use, none of these differences was statistically significant (Hedges & Pigott, 2004, formulas 11–12, p. 432).

For each outcome we calculated an average effect size and 95% confidence interval and a related Z statistic. To facilitate interpretation, we scaled all outcomes so that positive effect sizes represent effects in the desired direction, i.e., lower delinquency, aggression and drug use, higher academic achievement or lower school failure. Table 4 reports the results for the meta-analysis for each of the four studied outcomes.

4.2 Moderator Analyses

We conducted analyses to determine whether the effects of the mentoring interventions varied by four key aspects of intervention design and implementation and of four key processes theorized as avenues for mentoring effects:

1)

selectivity in inclusion (high individual risk, high environmental risk, or no such selectivity)

As noted earlier we combined across outcomes and for these analyses given the constrained sample sizes and used a test as all moderator analyses tested the null hypothesis that the effect of the moderator was zero, regardless of which level of the moderator was coded “1” and which was coded “0”. To check on the validity of combining across outcomes we tested for bias in effects due to this aggregation (e.g. effects are limited to one outcome or heavily dependent on specific outcome). To do so we conducted two sets of sensitivity analyses. For the first set of analyses, we employed Hedges and Pigott's (2004, formulas 11–12, p. 432) method for contrasting group mean effect sizes in meta-analysis to contrast effect sizes from studies reporting delinquency outcomes against those reporting each outcome against those reporting on the other three outcomes. These results produced no evidence that effect sizes differed substantially by any given outcome, which would mean moderation relations were not due to a true relation with only a single outcome, Z (delinquency-aggression) = -0.17, ns; Z (delinquency-drug use) = 1.61, ns; Z (delinquency-academic) = 1.77, ns; Z (aggression-drug use) = 0.74, ns; Z (aggression-academic) = 0.81, ns; and Z (academic-drug use) = -0.07, ns. We also coded outcomes of each study according to the outcome variables used (e.g., 1–4 = Delinquency, Aggression, Drug Use, Academic Achievement). We then cross-tabulated these codes with categorical scores for whether a given moderator could be coded. No significant results were obtained. Only one moderator, professional development as a motivation for mentoring, showed any such tendency, with a marginally higher than expected frequency by outcome (for academic achievement) χ2 (5, n = 36) = 11.05, p < .05. These results suggested to us sufficient confidence that moderation analyses collapsed across outcomes would be not biased or misrepresenting an overall relation for mentoring programs. In combination with the practical consideration of sample size limitations we judged this an appropriate way to serve the goals of the review with the available studies.

We tested for moderation using two methods. First, we calculated meta-analysis statistics separately by levels of the moderators (Hunter & Schmidt, 2004, p. 402). Table 6 reports the standardized mean difference effect sizes by levels of each moderator, the number of studies in each level of the moderator, and the lower and upper limits of the 95% confidence intervals for each random effect estimate. Table 6 also reports the moderator effect estimates, standard errors, and significance tests from the meta-regression analyses described above.

As can be seen in Table 6 there was significant moderation for Motivation for Mentoring but not for other program organization and implementation features. We provide plots for Mentor Motivation in Figure 5. As can be seen in Figure 5 effects were larger when mentor motivation was based in professional development.

In regard to key processes of mentoring interventions, there was evidence of significant moderation by the presence of two component processes in mentoring: Advocacy and Emotional Support (See Table 6). The results are illustrated in Figure 6. Stronger effects were observed when Emotional Support and Advocacy were components of mentoring than when these components were not present. Figure 6 suggests that stronger effects were observed when teaching was a component of mentoring, but the meta-regression that included a term for research design did not return significant evidence of moderation.