Mentoring Interventions to Affect Juvenile Delinquency and Associated Problems

Linked article:

Updated Systematic review.

Linked article:

PROTOCOL.

Synopsis

Mentoring is one of the most commonly-used interventions to prevent, divert, and remediate youth engaged in, or thought to be at risk for delinquent behavior, school failure, aggression, or other antisocial behavior. We conducted a meta-analytic review of selective and indicated mentoring interventions that have been evaluated for their effects on delinquency outcomes for youth (e.g., arrest or conviction as a delinquent, self-reported involvement) and key associated outcomes (aggression, drug use, academic functioning). Of 112 identified studies reported published between 1970 and 2005, 39 met criteria for inclusion. Mean effects sizes were significant and positive for each outcome category. Effects were largest (still moderate by Cohen's differentiation) for delinquency and aggression. However, these categories also showed the most heterogeneity across studies. The obtained patterns of effects suggest mentoring may be valuable for those at-risk or already involved in delinquency and for associated outcomes. Moderator analyses found stronger effects in RCTs compared to quasi-experimental studies, for studies where emotional support was a key process involved in mentoring, and where professional development was a motivation for mentors. However, the collected set of studies are less informative than expected with quite limited detail in studies about what comprised mentoring activity and key implementation characteristics. This limitation encourages caution particularly in interpreting the moderated effects. These findings add to the longstanding calls for more careful design and testing of mentoring efforts to provide the needed specificity to guide effective practice of this popular approach.

2.0 Background for the Review

Mentoring is one of the most commonly-used interventions to prevent, divert, and remediate youth engaged in, or thought to be at risk for, delinquent behavior, school failure, aggression, or other antisocial behavior (DuBois, Holloway, Valentine, & Cooper, 2002). It is the centerpiece of the work of the Boys and Girls Clubs of America. A recent account lists over 4500 organizations within the United States that use mentoring to promote youth wellbeing and reduce risk (Rhodes, 2002).

Definitions of mentoring vary, but there are common elements that can be identified across definitions (DuBois & Karcher, 2005). Most commonly the central feature is a one-on-one relationship between a provider (mentor) and a recipient (mentee) for the potential of benefit for the mentee. For the purpose of this review, mentoring will be defined by the following 4 characteristics: 1) interaction between two individuals over an extended period of time, 2) inequality of experience, knowledge, or power between the mentor and mentee (recipient), with the mentor possessing the greater share, 3) the mentee is in a position to imitate and benefit from the knowledge, skill, ability, or experience of the mentor, 4) absence of role inequality between provider and recipient that typifies most helping or intervention relationships whether based in professional training or certification of the provider or as occurs inherent in parent-child, teacher-student, or other professional-client relationships. Thus, mentoring differs from professional-client relationships such as counseling or therapy, and from parenting or formal educational relationships.

When applied to delinquency and other similar outcomes, mentoring usually involves older, usually adult, persons in the community who provide opportunities for imitation, gaining advice, pleasurable recreational activities that show care and interest in the mentee, and emotional support, information, and advocacy through a one-to-one relationship. Such opportunities are thought to foster healthy development and diversion from risk-elevating activities and attitudes (Jekielek, Moore, Hair, & Scarupa, 2002; Rhodes, Spencer, Keller, Lian, & Noam, 2006).

In recent years, mentoring has drawn substantial interest from policymakers, intervention theorists, and those interested in identifying promising and useful evidence-based approaches to interventions for criminal justice and child welfare outcomes (Grossman & Tierney, 1998; Jekielek et al., 2002). The popularity and extensive anecdotal praise for mentoring makes it important to have sound, evidence-based, understanding of its promise. In this study, we conduct a meta-analytic review of mentoring interventions that have been evaluated for their effects on delinquency (e.g., arrest or conviction as a delinquent, self-reported involvement) and three outcomes (aggression, drug use, academic achievement) that often co-occur with delinquency, share risk factors, are often also targets of delinquency interventions and show effects from such efforts (Tolan, 2002).

Unlike many types of intervention, there are a substantial number of studies that evaluate the effects of some form of youth mentoring (Rhodes, Bogat, Roffman, Edelmena, & Galasso, 2002). Critical reviews have focused on the potential benefits of mentoring and characteristics that might be associated with positive effects from mentoring (Hall, 2003; Rhodes, 2002). More recently, several meta-analyses have considered mentoring programs in relation to youth risk, including delinquency (Aos, Lieb, Mayfield, Miller, & Pennucci, 2004, DuBois et al., 2002; Lipsey & Wilson, 1998). Thus, unlike some areas of intervention for delinquency and related problems, the accumulated literature on mentoring is substantial and has had conceptual and statistical scrutiny. None of the meta-analyses to date correspond exactly with the focus of the present review, but they were very helpful in planning this review. They suggest standards against which to evaluate the completeness of study inclusion and choices about coding and methodological requirements.

Many of the conceptual reviews have been focused on the potential of mentoring as a general approach to youth development promotion and to reduce risk among high risk populations (Jekielek et al., 2002; Rhodes, 2002). The meta-analysis, by DuBois et al. (2002) focused on mentoring efforts related to youth development. Although there was differentiation of “problem-behavior” from other outcomes (e.g. educational attainment, vocational) there was not clear emphasis on delinquency indicators as a separate area. Lipsey and Wilson (1998) organized their review around an interest in serious juvenile offenders. Therefore, inclusion was not about delinquency risk in general, precursors such as aggression level, or related outcomes such as substance use or academic functioning. Also, the interventions were coded in such a way that interventions that included mentoring in an array of interventions could not be distinguished from those that focused primarily or exclusively on mentoring. Mentoring was denoted by its mention in the description of a study, but often was considered as one member of a class of interventions with similar features. Aos et al. (2004), undertook their meta-analysis to inform a state legislature about the potential impact, costs and benefits of many empirically tested interventions for delinquency and other youth problems such as early pregnancy. Thus, their emphasis was on specific programs rather than mentoring as a general approach. Moreover, that review only examined the relative effect sizes in relation to costs and potential cost savings rather than the usual focus on methodological issues and other moderators of effects. In addition, they were interested in programs with a high level of empirical support for effects, so that their inclusion criteria were more restrictive than was used here.

The aforementioned conceptual and statistical reviews provided excellent perspectives on mentoring evaluation and valuable benchmarks for guiding this review. In addition, they provided strong data bases from which to organize this review. Because of the generous sharing of information about content and methods by these reviewers (including access to their databases in some cases), this review was able to build efficiently from their prior efforts in determining coding. These prior reviews also helped reduce worry about file drawer and grey material that might be important to consider but not found without thorough searching. Of course, we conducted an independent search to verify the applicable literature, published or not.

The accumulated reviews and the variations in the studies they included also point to the value of this review. Each suggested mentoring programs can have important effects on delinquency and related outcomes. In the DuBois et al. (2002) review, the overall category of problem behavior, which includes delinquency, had the largest effect sizes of any outcome category. However, that review and others noted the variation in effects even among well-designed and completed studies; variation that undercut confidence in the mean effect findings. The field is marked by mixed results (significant positive and negative effects) among the methodological stronger studies (e.g. McCord, 1992; O'Donnell, Lydgate, & Fo, 1979). As that review excluded the McCord Cambridge-Somerville and another major mentoring study, the Diversion Project of Davidson and colleagues (Davidson & Redner, 1988), the implications for mentoring as a criminal justice intervention are not clear. Both are studies that carried likely substantial impact on overall effect estimates and for design impact or moderator interpretation. Also, a scan of the literature at the outset of the review showed several new pertinent studies since the prior reviews.

Understanding Mentoring Effects

While a relatively large number of studies with some minimal evaluation design features have been found and utilized in prior evaluations of mentoring, there are characteristics of this field that have limited how informative reviews and meta-analyses have been. A major limitation is the lack of specificity in describing the activities and key processes of a given mentoring program and a way to discern clearly what constitutes a mentoring intervention and what are the important or necessary components for mentoring to have occurred (Roberts, Liabo, Lucas, DuBois, & Sheldon, 2004). Among the reports and reviews there is considerable variation in what activities are considered mentoring essentials and which are optional (DuBois & Karcher, 2005). In addition, there has been relatively little work on what might differentiate mentoring from other helping relationships (Rhodes, 2002). Limited intervention description may be because mentoring arose as a voluntary and “indigenous” approach to youth intervention. Thus, many mentoring efforts arose within a given setting without intention to formalize and standardize performance and activities. The practitioners who developed their particular approach may have had less training in and interest in formal evaluation features. As a field of intervention services and as a research focus, there appears to be mixed interest in facilitating more formal operations that will yield more informative and comparable results. Also, because one common basis for mentoring is a view that the positive influence of an interested person providing a supportive relationship is what is helping, there is less interest in trying to specify what activities and processes constitutes mentoring and what among these could explain any benefits derived. For both reasons formalized protocols and systematic training approaches may not have been a priority. Consequently the body of research is remarkable in the limited emphasis on intervention content and interest in relating to a common set of principles, theorized processes, or requisite structures and components. There seems to remain, limited valuing and perhaps even some reluctance to aim for continuity across the field or specificity in applying and describing mentoring efforts that might facilitate scientific understanding of effects. Hence, there are few training, implementation, and dosage parameters that can be identified as having consensus. There are few indications of what is considered essential or critical for mentoring and helpful in distinguishing mentoring from other helping relationships and approaches. Similarly, the reports reviewed here continue an unfortunate tradition of having limited information by intervention science standards and are less informative than needed about what may account for benefits accrued. One of the goals of this effort was to code, to the extent possible, processes, training or implementation features, and activities or components within the mentoring programs to help advance understanding of mentoring, effects found, and potential for further study and use.

One important area to understand the implications of mentoring and its value for affecting juvenile delinquency is the implication that a strong personal relationship between the mentor and mentee is a key to any benefits derived (DuBois et al., 2002; Rhodes et al., 2006). In most cases, a corollary is that the mentor is undertaking this activity, not as a professional in the helping or social service professions, but because of personal interest or sense of duty, often as a volunteer (Rhodes, 2002). When a person with professional background or duties to provide such services offers mentoring, the emphasis is more on the relationship and the personal interest in the mentee than on specific skills, activities, or formal protocols. Thus, it has been noted that one limitation of mentoring may be that providers may be less accountable as they are volunteers and/or may not be well prepared for challenges of developing and maintaining a relationship with sometimes challenging and less appreciative youth (Grossman & Tierney, 1998). Non-specificity about features such as attrition of providers, motivation of mentors, the use of training, structuring of activities, and adherence of those trained may leave it uncertain why some interventions are helpful and others are not (DuBois et al., 2002). More understanding of what motivates mentors and how different reasons for undertaking mentoring affect mentoring would help in improving effects and understanding effect variations. Similarly, the extent to which there is emphasis on following these procedures and principles thought to be helpful should relate to effect levels.

A second question of importance about mentoring is the relative value of mentoring as a high-risk selective and/or indicated approach (Tolan & Guerra, 1994). Mentoring studies have been applied to high-risk, identified, and general populations of youth with variation within such populations in delinquency specifically or as part of a more general definition of high-risk behavior (DuBois et al., 2002). Some have argued that mentoring represents an alternative view about youth risk, a focus on promoting healthy or positive development through strengthening abilities rather than minimizing exposure to risk or remediation of undesirable behavior and characteristics (Jekielek, et al, 2002). There is evidence that preventive effects for high-risk youth may be quite different from those accrued for the general population (Tolan & Gorman-Smith, 2003). For example, it may be that mentoring is not valuable in affecting delinquency or related outcomes of high risk youth because it is not structured enough and focused on multiple risk factors thought to drive that behavior (Lipsey & Wilson, 1998). Thus, there is a policy interest in whether targeting high-risk youth (selective inclusion) is useful. Therefore, the review undertaken here was focused on youth defined as high-risk for or already engaged in delinquency (Tolan & Gorman-Smith, 2003).

As noted above, theoretical summaries of the field and attempts to relate mentoring to prevention science, developmental psychopathology, and/or youth development literature in general have suggested some likely key features of mentoring (Lipsey & Wilson, 1998; McCord, 1992). From the accumulated writing it can be extracted that mentoring processes of interest are 1) identification of the recipient with the mentor that helps with motivation, behavior, and bonding to conventions; 2) provision of information that might aid the recipient in managing social, educational, legal, family, and peer challenges; 3)advocacy for the recipient in various systems and settings; and 4)emotional support and friendliness to promote self-efficacy, confidence, and sense of mattering (DuBois et al., 2002; Rhodes et al., 2002). Given this abstracted set of potentially contributing processes/activities within mentoring, we coded studies for evidence of each to permit examining how their inclusion may have affected outcome.

Similarly, as others have noted it is common for mentoring to occur as part of a multi-component program, whether as one of several components or as a central focus augmented by additional supporting activities (Aos et al., 2004). This leaves open an important question of the extent to which effects attributed to mentoring might actually be coincidental inclusion with other effective components. It also leaves undifferentiated to what extent it matters if the delivery with other components is simply as one of a set of substantial program features or if the program is primarily mentoring with some augmentation to help support and enhance the mentoring impact. These questions of interest suggest coding of these features, where discernible might improve understanding of the value of mentoring.

DuBois et al. (2002) recognized these issues in large part and incorporated coding of many such features into their meta-analysis. They denoted an index of what could be considered best practices in youth mentoring based on recommendations of prior reviews and recommendations for establishing effective mentoring programs, such as the National Mentoring Working Group (1991) and coded to the extent possible from source data, each intervention report (DuBois et al., 2002). They included 11 program features to mark how methodic inclusion in the program was, whether mentors and mentees are matched on demographic characteristics, how structured or prescribed activities were, and the frequency or extent of contact. These codes were then amalgamated into an overall index of extent of desired features. While an informative advance about how the extent of features considered useful for good mentoring related to effect size, because it is a single score across many areas it can not indicate the importance of specific features. Also, it may have obscured how many of the reports did not have adequate reporting to fully assess the 11 features.

We attempt to build on efforts of DuBois et al. (2002) to code theoretically and empirically linked valued characteristics, activities, and organization by focusing on the moderating effects of each of several key features related to 1) selectivity in inclusion (high risk versus universal or no selectivity within the population); 2)explicit attention to presence of four key processes such as modeling, emotional support, advocacy, and teaching; 3) whether or not mentoring is a stand-alone approach in that study or was undertaken along with some other components: 4) the motivation of the mentors in participating; and 5) the extent to which quality of work and fidelity were assessed or emphasized. This coding was considered useful for suggesting what might differentiate mentoring from other similarly intended youth interventions. That despite prior identification of specificity of such features as a major limitation of the mentoring literature, we did not find much improvement over time in the ability to determine details needed to code many of these features for this review. We had to limit our analyses to those features that could be coded for enough studies to enable some useful comparison.

3.0 Objectives of the Review

This systematic review had the following objectives:

To statistically characterize the evidence to date on the effects of mentoring interventions (preventive and treatment) for delinquency (e.g. official records and self-reported), and the associated problems of aggression drug use, and school failure.

4.0 Methods

In order to provide a review that is as free of bias as possible, we adopted a systematic review strategy for the research on the effects of mentoring interventions. Prior to beginning this review, the title and associated protocol were reviewed and accepted as a Campbell Collaboration Systematic Review (http://www.campbellcollaboration.org/frontend2.asp?ID=102).

Search strategy for identifying relevant studies.

Three authors have conducted prior meta-analyses on mentoring or related topics: 1) DuBois et al. (2002) on mentoring in general, 2) Lipsey and Wilson (1998) on delinquency interventions in general, and 3) Aos et al. (2004) on interventions for delinquency and associated social problems. Prior to conducting this review, each of these authors allowed us access to some of the materials used in their analyses. Drs. Lipsey and Aos and their colleagues released the actual databases used for their analysis. We found that one or more of these authors had already located many of the studies to be included in this analysis. However, we conducted our own review to locate studies done since these earlier reviews were completed and to locate other studies, including those that were unpublished at the time of these previous analyses. We used dates, sample sizes, authorship, and information provided on studies to determine whether two effects on the same outcome came from the same study. We did not count effect sizes at different follow-up points as independent effects.

Search terms and databases.

We based our search terms on those used by prior meta-analyses. We used a combination of terms in searching electronic databases and research registers. Table 1 shows the search terms used, although slight deviations in key words (including derivative forms of the listed terms) required modification to achieve equivalent searches in some databases (e.g., choosing a broader search term when a narrower term was not supported in the database). We also provide details of combinations of the search terms and some examples of resulting search combinations (shown in the inner cells) in Table 2. We searched the databases using combinations of terms, each of which contained: 1) one of four outcomes (and derivative forms of these terms): delinquency, aggression, substance use, or academic achievement; 2) a cognate of mentoring; and 3) a cognate of intervention

Databases searched.

Databases were selected based on their potential relevance to the topic and to the outcomes of delinquency, academic achievement, aggression, and substance use more generally. The databases searched included PsychINFO, Criminal Justice Abstracts, Criminal Justice Periodicals Index, Social Sciences Citation Index (SSCI), Science Citation Index (SCI), Applied Social Sciences Indexes and Abstracts (ASSIA), MEDLINE, Science Direct, Sociological Abstracts, Dissertation Abstracts, Database of Abstracts of Reviews of Effectiveness, and ERIC (Education Resources Information Center). The following research registers were also searched: the Social, Psychological, Educational and Criminological Trials Register (SPECTR), the National Research Register (NRR, research in progress), and SIGLE (System for Information on Grey Literature in Europe). Finally, the reference lists of primary studies and reviews in studies identified from the search of electronic resources were scanned for any not yet identified that were relevant to the systematic review.

4.1 Criteria for inclusion and exclusion of studies in the review

Only studies that satisfy all of the following inclusion criteria and none of the following exclusion criteria were included in this review:

Outcomes measured

We focused this systematic review on outcomes related to juvenile delinquency. We included studies with outcome measures of juvenile delinquency, reported by the individual or by others, or derived from archival sources such as arrest or juvenile court records. We also included studies focusing on precursors of delinquency such as aggression or high levels of externalizing problems and studies with two outcomes that are correlated with and frequently co-occur with criminal involvement or delinquency risk (drug abuse and academic achievement/ school failure). As noted above, the specific terms for each outcome are provided in Table 1.

Types of participants

Juvenile delinquency is typically defined as antisocial or criminal behavior by persons under age 18 (Tolan, 2002). In this systematic review of mentoring interventions, we included studies that involved youth who were included because they were currently showing behavior that would constitute juvenile delinquency or were identified and included because they were “at-risk” for juvenile delinquency. At-risk is defined as the presence of individual or ecological characteristics that increase the probability of delinquency in later adolescence or adulthood (Tolan, 2000). Ecological characteristics include family and parenting influences on behavior, residence in neighborhoods with high levels of poverty or crime, exposure to gangs, and other social setting factors (Tolan & Gorman-Smith, 2003). Individual characteristics include high scores on screening measures for aggression, evidence of oppositional defiant or conduct disorders, school failure, or attitudes and beliefs consistent with elevated use of aggression or antisocial behavior (Farrington, 2004).

Intervention Type.

We included interventions focusing on prevention and treatment (referred to as selective and indicated population interventions). In the initial phase of study selection, we sought out any studies that described their interventions as mentoring, that mentioned mentoring as any part of their intervention strategy, or had interventions characterized by any of the four characteristics noted above, whether or not they specifically mentioned mentoring.

Regarding, the defining characteristic of absence of formalized role inequality, previous reviews have differed on the inclusion of studies using professionals as mentors. DuBois et al. (2002) excluded interventions using professional providers, with the exception that some studies that employed mental health professionals as mentors were included under certain conditions (see DuBois et al., 2002; Rhodes, 2002 for those criteria). We differed from this prior review by including studies with mental health providers as mentors if their involvement was unstructured or limited to a non-specific or support intervention (not psychotherapeutic). Functionally this means inclusion here of some critical studies for the current focus that were not included in the DuBois review, such as the McCord Cambridge-Somerville study(McCord, 1978, 1979).

We then excluded studies in which the intervention was explicitly psychotherapeutic, behavior modification, or cognitive behavioral training. Although we included studies in which mentoring was done as a part of another structured intervention, those studies that were conducted without providing results for the mentoring intervention separately were coded as including either an additional primary intervention (i.e., a major component in addition to mentoring) or an additional secondary intervention (i.e., a minor component in addition to mentoring).

In addition to requiring that studies investigate the effects of a mentoring intervention, as described above, we followed three additional criteria based on those used by Lipsey and Wilson (1998) in their meta-analysis of intervention effects on delinquency. We only included studies that measured at least one quantified outcome variable for the outcome of interest among the four considered here and that provided sufficient data to allow calculate an effect size and decipher its direction. When studies measured a delinquency-related outcome but did not report sufficient detail to allow calculation of an effect size, we attempted to contact the author to obtain additional information. Because of access to the Aos and Lipsey data bases we had a relatively complete rendering of the studies from which such information could be extracted. There were, therefore, very few studies that we were uncertain about whether additional information was obtainable

Research Design

The second criterion for inclusion in this review was that the study design involves a comparison that contrasted an intervention condition involving mentoring with a control condition. Control conditions could be “no treatment,” “waiting list,” “treatment as usual,” or “placebo treatment”. To ensure comparability across studies we made an a priori rule to not include comparisons to another experimental or actively applied intervention beyond treatment as usual. However, there were no such cases among the studies otherwise meeting criteria for inclusion.

We coded studies according to whether they were experimental or quasi-experimental designs. To qualify as experimental or quasi-experimental for the purposes of this review, we required each study to meet at least one of three criteria: 1) random assignment of subjects to treatment and control conditions or assignment by a procedure plausibly equivalent to randomization; 2) Individual subjects in the treatment and control conditions were prospectively matched on pretest variables and/or other relevant personal and demographic characteristics; 3) Use of a comparison group with demonstrated retrospective pretest equivalence on the outcome variables and demographic characteristics as described below

Randomized controlled trials that met the above conditions were clearly eligible for inclusion in the review. At the other end of inclusion eligibility, single-group pretest-posttest designs (studies in which the effects of treatment are examined by comparing measures taken before treatment with measures taken after treatment on a single subject sample) were never eligible. A few nonequivalent comparison group designs (studies in which treatment and control groups were compared even though the research subjects were not randomly assigned to those groups) were included. Such studies were only included if they matched treatment and control groups prior to treatment on at least one recognized risk variable for delinquency, had pretest measures for outcomes on which the treatment and control groups were compared and found to be essentially equivalent. We required that non-randomized quasi-experimental studies employed pre-treatment measures of delinquent, criminal, or antisocial behavior, or significant risk factors for such behavior, that were reported in a form that permitted assessment of the initial equivalence of the treatment and control groups on those variables.

Time Period and English Language Criteria

We limited the review to those studies conducted within the United States or another predominately English-speaking country and reported in English. Juvenile subjects did not need to speak English. A study conducted in the United States or Canada with resident Hispanic youth, for example, could have been included.

We limited the review to studies published since 1970. The 35+-year time frame between 1970 and the present (time of completion of search to conduct coding, 2005) is consistent with the time interval used by the review of the literature on delinquency conducted by Lipsey and Wilson (1998).

Coding of Article Characteristics

We double-coded 20% of the new articles (N=24), and calculated inter-coder reliability coefficients for study type (e.g., randomized trial), study quality, participant selection criteria (e.g., individual or behavioral risk), mentor motivations (e.g., survivor of abuse, professional development), and intervention components (e.g., modeling, teaching) using Cohen's kappa. We found high reliabilities for study type (κ = 1.0), study quality (κ = .93), and selection criteria (κ = .81). Coders easily determined some mentor motivations such as being a survivor of abuse (κ = .90), but tended to confuse civic duty and professional development (κ = .68). Not all categories were coded by either coder in the random sample of studies that were double coded. For example, of the mentoring components (modeling/identification, teaching, and emotional support) only modeling was found in the studies randomly selected for double coding. Final kappa reliabilities all were above .6, a level Landis and Koch (1977) suggested represented full agreement. Coders sought consensus with their supervisors, particularly on difficult-to-code categories such as mentor motivations.

We conducted a separate meta-analysis for each outcome. Each grouping of studies was based on the outcome, such that some studies might be included in more than one meta-analysis due to measuring more than one outcome. This affected 13 studies, two of which had three outcomes. We tested for influence of these studies on effects and found that the effect sizes in studies with single outcomes (SMD = 0.15, 95% CI = 0.06 - 0.24, Z = 3.35, p < .01) were slightly but not significantly lower than the effect sizes in studies with multiple outcomes (SMD = 0.29, 95% CI = 0.12 - 0.45, Z = 3.45, p < .01). Cross-tabulation of multiple outcomes by moderator variables revealed a single significant difference. Studies with a single outcome were more likely to have mentors that were coded as mentoring for professional development reasons than were studies that reported multiple outcomes, χ2 (1, N=33) = 4.01, p < .05.

Statistical Procedures.

For this study we used inverse-variance meta-analysis with a random-effects model, performed and plotted through the Metagen routine in the R statistical language. The random effects model addresses the research question of whether the average effects of an intervention in the population are significantly different from zero (Bailey, 1987; Raudenbush, 1994), whereas the fixed effects model addresses the question of the average effect of an intervention in the particular sample. Because our interest was in the population effects of mentoring, we chose to evaluate the random effects model for all variables.

The inverse variance method, as its name suggests, weights individual studies by the inverse of variance of their effect size. Thus, this method requires the calculation of standard errors of the effect sizes. For this purpose, we estimated variances for each effect size according to Hedges and Olkin's (1985, p. 86) Formula 14:

σ ^ d i 2 = ( n e + n c ) ( n e * n c ) + d i 2 ( n e + n c )

where σ_di ² is the estimated variance of the effect size, n _e is the number of experimental subjects, n _c is the number of control subjects, and d _i ² is the square of the effect size of the study. Finally, we ran each meta-analysis two ways: 1) as an un-weighted model, and 2) weighted by the inverse of the estimated variance.

The effect sizes of the interventions under evaluation were calculated in units of Cohen's (1988) d. For studies reporting means, standard deviations, and Ns of numeric data, the effect size was calculated by dividing the treatment difference less the control difference over the pooled treatment and control standard deviation:

d = ( ( ME 2 − ME 1 ) − ( MC 2 − MC 1 ) ) ( ( NE 2 − 1 ) SE 1 2 + ( NC 2 − 1 ) SC 1 2 ) ( NE 1 + NC 1 − 2 ) ,

where:

M = mean

E = treatment

1 = pretest

S = standard deviation

C = control

2 = posttest

For studies that reported dichotomous outcomes, we calculated odds ratios and converted them into an equivalent standardized mean difference effect size estimate (Lipsey & Wilson, 1998). Chinn (2000) noted that dividing the natural log of an odds ratio by π/√3 produces an excellent approximation of the standardized mean difference effect size.

5.0 Results

6.0 Conclusions

This review of the methodologically adequate studies on mentoring for high risk youth found positive effects for delinquency and for three other associated outcomes: aggression, drug use, and academic performance. The effects are significantly different from zero for all four outcomes. The most reliably evident and strongest effects were for delinquency and its close surrogate, aggression. In both cases the estimate of effect was about .3 to .4 SD units, with few incidents of studies reporting negative effects. More modest effect sizes were found for substance use and academic achievement.

These results suggest mentoring, at least as represented by the included studies, has positive effects for these important public health problems, albeit from small to modest in effect size. However, there were several limitations that preclude statements with much more specificity and certainty about what within mentoring is the basis for these effects. Perhaps most notably, the collected set of articles are limited in how much is described about the specific intervention and even more limited in how the intervention organization, components, and delivery are thought to relate to desired effects. As we noted in the introduction and as we attempted to code, there are key characteristics thought to distinguish it from other helping relationships and to be the basis for benefits and therefore should be common across studies and their quality relate to effect size.

However, we were unable to code many studies for many of these characteristics. There is a notable lack of adequate reporting of specific components, implementation procedures and adherence, and measurement of targeted processes thought to be affected by the intervention across the accumulated literature. This unfortunate characteristic seems distinct in comparison to other areas of intervention outcome studies and does not seem to be improving over time (Tolan & Guerra, 1994). Thus, we have limited ability to help explain what differentiated which mentoring approaches or programs might be most helpful or what directions might be most important to further study.

We were able to conduct some moderator analyses despite these limitations. The results were not very powerful but did suggest that effects were larger when the program included emphasis on emotional support for the recipient and if the mentor was motivated to undertake this role as part of professional advancement.

Although the review focused on selective and indicated populations (those with risk characteristics or already exhibiting delinquency as a basis for inclusion) we did not find moderation by whether inclusion depended on individual risk characteristics or environmental or other-than-individual characteristics. While duly cautious about interpreting these null effects, the finding may suggest that either approach may be viable for effective targeting.

We also did not find effect differences by whether or not other interventions were included with mentoring or mentoring was part of a multi-component intervention than when it was offered on its own. This leaves open whether or not the effects when other interventions are present is attributable to mentoring but does suggest that mentoring, at least as represented in these collected studies, has effects apart from those attributable to other interventions. Within the overall concern about the quality of information about mentoring programs there is much need to consider designs that might consider mentoring singularly and as part of a package or in comparison to other singular interventions. This could not only help clarify the relative importance of other components but also the relative value in comparison to other interventions that might be alternatives. As issues such as cost effectiveness, ease of training and implementation, and sustainability come into consideration, such information is increasingly important.

These findings are consistent with prior meta-analyses that overlap in focusing on mentoring. As reported by Lipsey & Wilson (1998) and DuBois et al. (2002) these analyses suggest general support for mentoring for intervention related to delinquency and closely associated outcomes. However, as with those analyses, the information obtainable about the “inside” of these interventions termed mentoring is limited. Thus, the conclusions to be drawn must remain very sketchy about what it is that makes mentoring effective. This persistent characteristic of the field undercuts ability to recommend it for use as it is not clear what should be recommended. Further, while the positive effects suggest promise, the lack of standard types of information and formal approaches to documentation that characterizes the best studies in most areas of behavioral intervention seriously impedes incremental progress in best practices. Thus, while consistent with prior findings, there seems to be little additional certainty of the nature of mentoring and information to guide further development, sound training and management of the programming, and adequate tracking of effects to activities, staffing, and other features. Unfortunately this seems to be qualitatively the same state of need as was identified in our consideration of mentoring in a review of violence prevention 14 years ago (Tolan & Guerra, 1994). This is not the case for most areas of delinquency intervention.

As noted in the introduction and as can be easily noted in examining youth development efforts, delinquency programming, and popular interest in prevention mentoring is one of the most common and most favored approaches. It is also one with considerable presence in the scientific literature. While of the 112 studies located only 39 met criteria for inclusion, this does not mean the other 73 were of no value for informing science. Yet, after reviewing these we do note there is consistency across these in the lack of attention to common features in descriptions of other interventions, theoretical formulations about other interventions, and in systematic organization of reports, irrespective of methodological quality by our standards.

Thus, we can only suggest some tentative and general statements about what might affect mentoring impact. Perhaps the more striking statement to be made is that despite its popularity and the apparent benefits it provides, there is little understanding of just what makes an intervention mentoring and what about such labeled interventions is related to benefits derived. It seems striking that given its prominence in attempts to address these critical public health and youth problems there is such a lack of systematic attempts to unpack mentoring and to understand it within a conventional framework for evaluating intervention. It is also striking that funding and promotion of these efforts proceeds without more stringent evaluation, including more careful identification of population of interest, inclusion criteria, skills and training of providers, content and theorized processes of component effects, fidelity tests, and implementation levels for intent to treat. Perhaps most fundamentally the co-occurring popularity and the general promise of these findings point to the critical need for concerted efforts for substantial and probably large scale evaluation that can efficiently provide more clear and directing information about what about mentoring is the reason positive effects are derived. In particular it may be that the promise suggested in the reasonable effect sizes yielded here is only a base estimation of potential benefit. If suggested organization was brought to the application and evaluation of mentoring to affect delinquency and related problem, it may be that the benefits might be greater than estimated from this meta-analysis.

7.0 Plans for updating the Review

The review will be updated every 3 years.

Contributors to the Review

Patrick Tolan (PT), David Henry (DH), and Michael Schoeny (MS) wrote the text of the protocol. Arin Bass (AB) and MS conducted the searches which PT directed. PT and DH wrote the text of the review, DH, MS, and PT responded to editorial comments and those from external referees.

We also want to acknowledge the generous aid in regard to databases and advice from Mark Lipsey, David DuBois, and Steven Aos.