Abstract
Linked article:
Linked article:
2.0 Background for the Review
Mentoring is one of the most commonly used interventions to prevent, divert, and remediate youth engaged in or thought to be at risk for delinquent behavior, school failure, aggression, or other antisocial behavior (DuBois, Holloway, Valentine, & Cooper, 2002). It is the centerpiece of the work of the Boys and Girls Clubs of America, and other similar organizations. A recent account lists more than 4,500 organizations in the United States that use mentoring to promote youth well-being and reduce risk (Rhodes, 2002).
Although definitions of mentoring vary, there are common elements. For the purpose of this review, mentoring will be defined by the following four characteristics: an interaction between two individuals over an extended period of time, inequality of experience, knowledge, or power between the mentor and mentee, with the mentor possessing the greater share, the mentee being in a position to imitate and benefit from the knowledge, skill, ability, or experience of the mentor, the usual absence of role inequality based in training, certification, parent-child or teacher-student relationships.
Thus, mentoring differs from professional-client relationships such as counseling or therapy, and from parenting. When used in the context of preventing delinquency and other similar outcomes, mentoring often involves older, usually adult, persons in the community who offer model behavior, emotional support, information, and advocacy through one-on-one relationships. Such opportunities foster healthy development and divert youth from risky activities and attitudes (Jekielek, Moore, Hair, & Scarupa, 2002). Although mentoring may involve modeling, emotional support, advocacy, or other types of social support, it is the one-on-one relationship, the potential benefit for the mentee, and the absence of necessary role inequality that defines mentoring.
Mentoring has drawn substantial interest from policymakers, intervention theorists, and those interested in identifying promising and useful evidence-based approaches to interventions for criminal justice and child welfare outcomes (Grossman & Tierney, 1998; Jekliek et al., 2002). Given its popularity and praise, it is important to develop a sound, evidence-based understanding of its promise. In the proposed study, we will conduct a meta-analytic review of mentoring interventions that have been evaluated for their effects on criminal justice outcomes for youth (e.g., arrest or conviction as a delinquent, self-reported involvement, and prevention of involvement). We also will examine key associated outcomes, including precursors such as aggression and the associated problems of academic functioning and substance use.
The Extent of Mentoring Studies and Reviews
Unlike for other types of intervention, several studies have evaluated the effects of youth mentoring (Rhodes, Bogat, Roffman, Edelmena, & Galasso, 2002). Critical reviews have focused on the potential benefits of mentoring and characteristics that might be associated with its positive effects (Hall, 2003; Rhodes, 2002). More recently, several meta-analyses have considered mentoring programs in relation to youth risk, including delinquency (Aos, Lieb, Mayfield, Miller, & Pennucci, 2004; DuBois et al., 2002; Lipsey & Wilson, 1998). Thus, unlike some areas of intervention for delinquency and related problems, the accumulated literature on mentoring is substantial and has undergone conceptual and statistical scrutiny. None of the meta-analyses to date corresponds exactly with the focus of the proposed review, but they have been very helpful in its planning. They suggest standards against which to evaluate the completeness of study inclusion and choices about coding and methodological requirements.
Many of the conceptual reviews have focused on the potential of mentoring as a general approach to promoting positive youth development and to reduce risk among high-risk groups (Jekielek et al., 2002; Rhodes, 2002). A meta-analysis by DuBois et al. (2002) focused on mentoring efforts related to youth development. Although these studies differentiated “problem-behavior” from other outcomes (e.g., educational attainment, vocational outcomes), delinquency indicators were not delineated separately, which is a strength of our proposed study. Although Lipsey and Wilson (1998) organized their review around serious juvenile offenders, the studies included in the meta-analyses were not about delinquency risk in general, nor were they focused on precursors such as aggression level or related outcomes such as substance use or academic functioning. Also, the interventions were coded in such a way that interventions that included mentoring among an array of interventions could not be distinguished from those that focused primarily or exclusively on mentoring. Mentoring was denoted by its mention in the description of a study, but often was considered as one member of a class of interventions with similar features.
Aos et al. (2004) undertook their meta-analysis to inform a state legislature about the potential impact, costs, and benefits of many empirically tested interventions for delinquency and other youth problems, such as early pregnancy. Thus, their emphasis was on specific programs rather than mentoring as a general approach. Moreover, that review only examined the relative effect sizes in relation to costs and potential cost savings rather than the usual focus on approaches and methodology and other moderators of effects. In addition, they were interested in programs with a very high level of empirical support for effects, so that their inclusion criteria were somewhat more restrictive than are planned here.
The aforementioned conceptual and statistical reviews provide excellent perspectives on mentoring evaluation and valuable benchmarks for guiding this review. In addition, they provide strong databases from which to organize this review. Because the authors of these studies generously agreed to share information about content and methods (including access to their databases in some cases), we can move forward with more efficiency than might typically occur. These prior efforts will also aid in coding and can help reduce worry about “file drawer” and “gray material” that might be important to consider. Of course, we are undertaking an independent search to verify the applicable literature, published or not, but do benefit from the careful searching and prior estimates of the validity of the findings from these excellent efforts.
The variations in the prior meta-analyses also point to the value of this current review. All past reviews suggest mentoring programs can have important effects on delinquency and related outcomes. In the DuBois et al. (2002) review, the largest effect sizes occurred in the overall category of problem behavior, which includes delinquency. However, that review and others have noted the variation in effects even among well-designed and completed studies. Among the methodologically stronger studies, the results are mixed (significant positive and negative effects) (e.g., McCord, 1992; O’Donnell, Lydgate, & Fo, 1979). In addition, given that DuBois et al. excluded the McCord Cambridge-Somerville and the Diversion Project (Davidson & Redner, 1988), both major mentoring studies, the implications for mentoring as a criminal justice intervention are not as clear as desired. Both of the latter two studies are likely to have a substantial effect on estimates overall and on design, program, and other issues in moderator analyses. Also, our scan of the literature suggests there are more recent studies that might enhance findings.
Need for More Extensive Evaluation of Mentoring Program Qualities and Processes of Effects
Although past assessments of mentoring have included relatively large number of studies with at least minimal evaluation design features, other characteristics of mentoring may have limited how informative reviews and meta-analyses are. A major limitation is the uncertainty and lack of specificity about what constitutes a mentoring intervention and what the important or necessary components of mentoring are (Roberts, Liabo, Lucas, DuBois, & Sheldon, 2004). Among the reports and reviews, there is considerable variation in what activities are considered mentoring. In addition, there has been relatively little work on what might differentiate mentoring from other helping relationships (Rhodes, 2002). Our review helps to identify studies using qualities thought to constitute mentoring and to also permit some inferences from meta-analysis results about what should be included in mentoring and what might be characteristics that ensuing studies should use to differentiate mentoring from other types of helpful interventions with similar goals.
Because mentoring arose as a voluntary and “indigenous” approach to youth intervention, many evaluations are limited in the structure of training, activities, and methods used. In addition, providers may shift their content when interest wanes in volunteering or when frustrations arise in developing and maintaining a relationship with a high-risk youth (Grossman & Tierney, 1998). In addition, there is no consensus about the importance of intervention structure and consistency of content, training of providers, implementation fidelity and guidance, or minimal dosage parameters, all of which can preclude consensus about what these should be. There is also an implicit contention that benefits derived stem from the quality of the relationship between the mentor and youth, rather than from training, activities, or dosage. These issues have received only limited research attention. One of the goals of our study is to code the processes and activities within the mentoring programs to determine which components are related to various effects.
Toward that end, DuBois et al. (2002) incorporated an overall index about quality of mentoring programs into their meta-analysis. They denoted 11 qualities thought to represent best practices on the basis of recommendations in prior reviews and recommendations from, for example, the National Mentoring Working Group (1991) for establishing effective mentoring programs. Among these qualities are how structured inclusion in the program is, whether mentors and child are matched on characteristics, how structured the mentoring activities are, and frequency of contact. DuBois et al. coded programs for the presence or absence of each quality and assigned an overall quality score. Although this index provides information on how influential certain program qualities are, it is limited in that it does not consider other, additional qualities. We will build on efforts of DuBois et al. to code these theoretically and empirically linked characteristics, activities, and organization, but extend that coding to key processes of mentoring: modeling, emotional support, advocacy, and teaching. This coding can help to differentiate mentoring from other similarly intended youth interventions.
We will also examine the role of the motivation of the mentor and relationship quality between mentor and mentee. Some evaluations indicate procedures, such as matching by gender, ethnicity, and interests, that capture relationship quality or mentor motivation. Others indicate they provide formal training for mentors, which can influence relationship and motivation. Still others have both formal training and supervision structures. DuBois et al. (2002) examine how these procedures might affect findings for mentoring programs. We will elaborate their coding in an attempt to capture, as available, information about recruiting mentors and indicators of how relationship quality was monitored. This elaboration is based on coding by Lipsey's group (Lipsey & Wilson, 1998) in delinquency interventions. We hope, as a next step, to relate these features to other non-mentoring programs to determine whether the effects are specific to mentoring or are more general effects for interventions.
A third area of importance is the relative value of mentoring as a preventive, remedial, or treatment approach (Tolan & Guerra, 1994). Some have argued that mentoring represents an alternative view of youth risk from that driving much treatment because it focuses on promoting healthy or positive development by strengthening abilities rather than minimizing risk or remediating undesirable behavior and characteristics (Jekielek et al., 2002). Although there is value in a theoretical distinction between programs that emphasize bolstering existing capabilities and creating opportunities that focus on lessening, tempering, or ending problematic characteristics, this distinction does not differentiate prevention and treatment necessarily (Tolan, 2002). There is variation in how mentoring programs have been applied as preventive, remedial, or as treatment interventions. Mentoring is often intended to be a universal prevention effort to help high-risk youth selectively, or to help those already exhibiting problems (indicated) (DuBois et al., 2002). Given that the extent of impact and the implications for future use vary by program type (universal, selective, or indicated prevention or treatment), determining variations in effects for these types of applications would be valuable (Tolan & Gorman-Smith, 2003). Therefore, we will examine effects by level of intervention (preventive and treatment).
3.0 Objectives of the Review
This systematic review has the following objectives: To statistically characterize the evidence on the effects of mentoring interventions (preventive and treatment) for delinquency (here to include arrest, reported delinquency, and aggression) and related problems (such as drug use and school failure); To statistically characterize the evidence of mentoring interventions on key processes thought to be affected by mentoring and that affect delinquency and related problems; To help define mentoring in a more systematic fashion and, in turn, to help clarify what constitutes mentoring and what might be key components for future research; To help clarify the variation in effects of mentoring related to program makeup and delivery, study methodology, and participant characteristics; To identify gaps in the research and make recommendations for further research; To inform policy about the value of mentoring and the key features for utility.
4.0 Methods
4.1 Criteria for including or excluding studies in the review
Only studies that satisfy all of the inclusion criteria and none of the exclusion criteria will be included in this review. Types of participants to be included
We include past studies that involve youth who were at risk for juvenile delinquency, or who were currently exhibiting signs of juvenile delinquency or its precursors (e.g., aggression, externalizing behavior problems). Delinquency is defined as antisocial or criminal behavior by persons under age 21. Risk is defined as the presence of individual or ecological characteristics that increase the probability of delinquency in later adolescence or adulthood. Ecological characteristics include residence in neighborhoods with high levels of poverty or crime. Individual characteristics include school failure, high scores on screening measures for aggression, or other evidence consistent with oppositional defiant or conduct disorders.
Types of interventions to be included
One limit of the field of mentoring, as indicated above, is that there is no accepted definition of what constitutes mentoring. For this review, we define mentoring as an intervention that includes any of the above-noted (p. 1) four emerging criteria of mentoring as the central characteristic or primary component of the program, or that mention mentoring in a description of the intervention or as a component of the intervention. In addition, we include studies that do not explicitly consider themselves mentoring but that have an intervention that possess one or more of the characteristics listed above.
DuBois et al. (2002) excluded those interventions that used professional providers, reasoning that this was not typical of mentoring. However, other reviews, such as the McCord Cambridge-Somerville study, include studies that employed mental health professionals under certain conditions (see DuBois et al., 2002; Rhodes, 2002 for those criteria). We will include studies that employ mental health professionals if the involvement is unstructured or limited to a nonspecific intervention. To help examine mentoring influence on outcomes by type of mentor relationship, we will code studies for the types of mentors involved (e.g., community members, paraprofessionals, professionals). If we find a mix of effects, we will use separate meta-analyses to determine whether the type of mentor is associated with the effects obtained.
We will include interventions that focus on prevention, treatment (mitigating the effects), or post-delinquency involvement (limiting delinquency's scope). We will distinguish programs by universal, selective, and indicated prevention. Selective and indicated interventions will be differentiated from universal interventions by the involvement in the latter of youth regardless of pre-existing risk or delinquent involvement. Universal, selective, and indicated interventions will be meta-analyzed separately initially.
We exclude studies in which the intervention is explicitly psychotherapeutic, behavior modification, or cognitive behavioral training. We also exclude studies that focus on another structured program provided by professional trained individuals.
Types of outcome measures to be included
Within this systematic review, we will focus on criminal justice outcomes, in particular juvenile delinquency or the committing of crimes by youth. We will include studies in which instances of juvenile delinquency are reported by the individual or as reported by others, or as indicated in archival sources such as arrest records or juvenile court records. In addition, we include studies that focus on precursors associated with delinquency (aggression and externalizing behavior problems) and outcomes that are correlated with and frequently accompany criminal involvement of youth (e.g., drug abuse and school failure).
We will also code any intervening processes and perceptions that can affect delinquency. For example, mentoring is often expected to improve a youth's self-efficacy, which can lessen the risk for delinquency. Similarly, mentoring may alter beliefs about aggression, norm violation, or relationships with parents, all strong predictors of delinquency. We want to determine whether mentoring programs affect such factors thought to mitigate risk for delinquency. Although not a primary focus of this report or a formal test of mediators, including such mediators in the review can indicate the extent to which mentoring affects these intervening factors.
We will include various outcome variables under composite categories. For each composite category, we conduct separate meta-analyses. Table 1 lists the specific variables we combine into composite categories.
Categories and Variables for Meta-Analysis
Studies of mentoring that do not include any of the outcomes detailed in Table 1 will be excluded from this review.
Types of studies to be included
In determining which studies to include, we follow criteria used by Lipsey and Wilson (1998) in their meta-analysis of intervention effects on delinquency. We have revised these criteria for the smaller number of studies likely to be found on mentoring. The inclusion criteria are as follows: The study must investigate the effects of a mentoring intervention, which is broadly defined as interaction between two individuals over an extended period of time. That interaction is characterized by 1) inequality of experience, knowledge, or power between the mentor and mentee, with the mentor possessing the greater share, 2) the mentee being in a position to imitate and benefit from the knowledge, skill, ability, or experience of the mentor, and, 3) an absence of role inequality based in training, certification, parent-child or teacher-student relationships. The intervention need not explicitly aim to reduce or prevent delinquency. The study must measure at least one quantitative delinquency outcome variable, such as those listed in Table 1 above. In addition, results must indicate or us to decipher the direction and magnitude of the effect. If a delinquency outcome is measured but the reported results fall short of our standard, the study will still be acceptable if the required results can be obtained from the author or other sources. The study design must involve a comparison that contrasts one or more conditions, including mentoring interventions, with one or more control conditions. Control conditions may be “no treatment,” “waiting list,” “treatment as usual,” or “placebo treatment” but they cannot represent an alternative intervention. Comparison conditions that include other treatments are expected to be rare. Studies will be coded according to whether they are experimental or quasi-experimental designs, the extent to which matching and pretest risk assessments are employed, and the nature of their control or comparison conditions. To qualify as experimental or quasi-experimental, the study design must meet at least one of the following three criteria: Subjects were randomly assigned to treatment and control conditions or assigned by a procedure plausibly equivalent to randomization, e.g., arbitrarily assigned wait-list, every other referral to the program. Individual subjects in the treatment and control conditions were prospectively matched on pretest variables and/or other relevant personal and demographic characteristics. Use of a comparison group with demonstrated retrospective pretest equivalence on the outcome variables as well as equivalence on demographic characteristics. In addition, studies must employ pre-treatment measures of delinquent, criminal, or antisocial behavior, or significant risk factors for such behavior, are reported in a form that permits assessment of the initial equivalence of the treatment and control groups on those variables. Random assignment designs that meet the above conditions are always eligible under this criterion. One-group pretest-posttest studies are never eligible (studies in which the effects of treatment are examined by comparing measures taken before treatment with measures taken after treatment on a single subject sample). Nonequivalent comparison group designs may be eligible (studies in which treatment and control groups are compared even though the research subjects were not randomly assigned to those groups). To be eligible, however, such comparisons must have either (a) matching of the treatment and control groups prior to treatment on at least one recognized risk variable for delinquency; (b) a pre-intervention measure (pretest) for at least one outcome variable on which the treatment and control groups can be compared; or (c) a pre-intervention measure on at least one recognized risk variable for delinquency (as above) on which the treatment and control groups can be compared. Note that the pre-intervention measures need not show that the treatment and control groups are actually similar; they must only be capable of showing their degree of similarity (or dissimilarity). The study must be set in the United States or a predominately English-speaking country and use juvenile residents of that country. Juveniles need not be English-speaking or “Anglo.” A study conducted in the United States or Canada with resident Hispanic juveniles, for example, would qualify. In addition, the study must be reported in English. The date of publication or reporting of the study must be 1970 or later even though the research itself might have been conducted prior to 1970. Both published and unpublished research reported since 1970 will be considered eligible for this review. This 30-year span will reflects a period when evaluation using randomized design became common. It is expected that few usable studies could be found with prior publication dates (DuBois et al., 2002).
The full report of any study that is judged to be potentially relevant to the review will be obtained. The full reports of all potentially relevant studies will be reassessed for relevance. If there are doubts about the inclusion or exclusion of a study, a second reviewer will be consulted. We will code and enter studies that have appropriate outcome measures but do not meet the experimental/quasi-experimental criterion noted above.
4.2 Search strategy for identifying relevant studies
Three authors have conducted meta-analyses on mentoring or related topics: 1) DuBois et al. on mentoring in general; 2) Lipsey et al. on delinquency interventions in general; 3) Aos et al. on interventions for delinquency and associated social problems. We have secured cooperation with each of these authors to release all included reports and their coding protocols. Lipsey et al. has released their actual database. Notably, each author was quite thorough. Therefore, we expect that most of the studies on this topic are already included in these prior meta-analyses. In addition, however, we will also search edited volumes and proceedings for further studies.
Search terms
Our search terms are derived from those used by prior authors, with augmentation to ensure we do not exclude relevant studies. We will use a combination of terms in searching electronic databases and research registers. Table 2 shows the search terms we will use, although in some cases we will use slight derivations in a given database.
We will adapt the search strategy for different databases according to the classification system of each database. When the format of a particular database does not permit use of this search strategy, we will use a broader search (e.g., by combining fewer categories of search terms). Further searches will be conducted using controlled language such as that found in subject headings, descriptors, and similar database fields.
We will search using combinations of terms, each of which contains: 1) one of four outcomes: delinquency, aggression, substance use, or academic achievement; 2) a cognate of mentoring; and 3) a cognate of intervention. Table 2 details combinations of search terms. Some exemplary search combinations are shown in the inner cells of Table 2
Combinations of Search Terms to be Used
Note: Combinations shown for delinquency and aggression outcomes only. Similar searches will be performed for substance use and academic achievement.
Years searched
Only studies reports since 1970 will be included.
Resources searched
Electronic Databases: PsychINFO, Criminal Justice Abstracts, Criminal Justice Periodicals Index, Social Sciences Citation Index (SSCI), Science Citation Index (SCI), Applied Social Sciences Indexes and Abstracts (ASSIA), MEDLINE, Science Direct, Sociological Abstracts, Dissertation Abstracts, Database of Abstracts of Reviews of Effectiveness, ERIC (Education Resources Information Center). Research Registers: Social, Psychological, Educational and Criminological Trials Register (SPECTR); National Research Register (NRR, research in progress), SIGLE (System for Information on Grey Literature in Europe). Reference Lists: The reference lists of primary studies and reviews identified via the electronic databases that satisfy the inclusion criteria were scanned for relevance to the systematic review.
On completion of the database and other searches, the references will be imported into the reference manager Endnote.
4.3 Description of methods used in the component studies
The studies included in the review are expected to employ an intervention versus control or comparison group research design and include, at least, pre-intervention and post-intervention measurements. For the purposes of this review, we will consider outcome measures administered to participants prior to the beginning of intervention to be pretest measures, and will regard the first measures administered within six months following the end of an intervention to be post-test measures. Waves of measurement beyond the first post-treatment assessment will be regarded as follow-up measures.
Control or comparison groups must be youth otherwise eligible to receive an intervention who have been assigned to observation only, treatment as usual, or waiting list control groups. Experimental studies using control groups are distinguished from quasi-experimental studies employing comparison groups by the presence or absence of random assignment to receive the intervention. Control or comparison groups must be equally eligible to receive intervention as those assigned to receive it. Studies employing nonequivalent comparison groups will be excluded.
The effect sizes of the interventions under evaluation will be calculated in units of Cohen's (1988) d, using the treatment difference less the control difference over the pooled treatment and control standard error based on pretest scores:
Where: M = mean
S = standard deviation E = treatment C = control
1 = pretest
2 = posttest
We will code studies for the methods used and, if evidence for heterogeneity of effects is found, we will investigate a possible moderating role for study design (Lipsey, 2003).
4.4 Criteria for determining independent findings
We will use dates, sample sizes, authorship, and information provided on studies to determine whether two effects come from the same study. In cases of doubt, we will contact the author of the report to determine independence of reported effects. We will not regard effect sizes at different follow-up points as independent effects.
4.5 Details of study coding categories
Data Extraction
Data extraction will follow the codebook developed by Lipsey and Wilson (1998) with some additions from DuBois et al. (2002) and our additions. As indicated above, there have been three major attempts to characterize mentoring interventions for juvenile delinquency. Two were focused on all types of interventions for delinquency. Lipsey and Wilson (1998) emphasized the broadest set of interventions from prevention through treatment, with a primary aim of characterizing effective approaches, how the methodology might affect findings, and the key features related to greater effect sizes. Aos et al. (2004) focused on prevention of juvenile delinquency and other social problems such as teenage pregnancy, with a key interest in relative cost-benefit of various levels of prevention and various approaches. There is much overlap in these meta-analyses in coding methods and programs included. All have remarkable similarity in the studies included, excepting variation due to when the review was completed. Lipsey and Wilson (1998) employ an extensive and well developed coding system that incorporates all the features of design, sample characteristics, effect size calculation, and other components and features of the study. DuBois et al. (2002) add more extensive coding on the mentors, the mentor-mentee matching, and the intent of the mentoring.
The coding scheme used by Lipsey et al. is extremely rigorous. Replicating that reliability and thoroughness from scratch is unlikely. Therefore, we will build on this exceptional approach. Similarly, the DuBois coding approach is well grounded in the mentoring literature and very applicable to our needs. The use of these existing systems not only will improve the efficiency but also will enhance the future efforts to relate and compare our findings to theirs.
Appendix I includes the Lipsey and Wilson (1998) codebook that will form the main part of our comparisons, including overall effect sizes and methodological considerations. Appendix I also includes the coding sheet developed by DuBois et al. (2002) and a set of additional or enhanced coding we will apply to all studies.
For those studies not already coded within any of the databases and for the new information, coding will be extracted by one of the two authors using the pre-specified data extraction/coding sheet (see Appendix I). When information is missing or unclear, attempts will be made to contact the authors directly.
We will double code 20% of the new articles, and calculate intercoder reliability coefficients using Cohen's kappa. In the event that reliabilities for any individual category are below .7, we will retrain coders (by double coding an additional 20% and comparing results on problematic categories). If, after retraining, intercoder reliability has not risen above .7, we will double-code the remaining 60% of the studies, discussing and reaching consensus on problematic items.
4.6 Statistical procedures and conventions
We will first evaluate the quality and characteristics of the data. In particular, we will document the extent of independent findings that can be considered for any of the major questions. In some cases, we expect that the number of studies will so small that metaanalysis calculations of effect sizes will be inadvisable or must be carefully qualified owing to potential instability of estimates. For each outcome, we will assess the extent to which there is heterogeneity in the effects of the mentoring interventions, and attempt, through moderated analyses, to determine whether such heterogeneity stems from differing program components, preventive or intervention levels, or participant selection criteria. These moderated analyses will determine the extent to which it is appropriate to estimate the average effect of mentoring on juvenile delinquency and associated behaviors (Deeks et al., 2001).
The meta-analyses will be conducted using macros developed by D. B. Wilson for SPSS (Lipsey & Wilson, 2001), as well as routines for meta-analysis in the R statistical language. As illustrated above, continuous data will be expressed as standardized mean difference effect sizes. If a study reports dichotomous data, these will be expressed as odds ratios and converted into an equivalent standardized mean difference effect (Lipsey & Wilson, 2001). Chinn (2000) notes that dividing the natural log of an odds ratio by π/√3 produces an excellent approximation of the standardized mean difference effect size.
The effect sizes will be transferred into an SPSS data sheet and will be weighted by the inverse variance to adjust for sample size bias. The weighted mean effect size for all the studies will be computed using a 95% confidence interval. The results of the meta-analysis will be presented using a forest plot, which will display the individual study effects, their confidence intervals, and the overall mean effect.
Prior to estimating effect sizes and confidence intervals, we will assess factors that indicate the validity of such estimates. We will use funnel plots of effect sizes by publication type and sample size, looking for evidence of bias (Light & Pillemer, 1984). We will also conduct an analysis of the homogeneity of effects. Because tests of homogeneity vary substantially in their power to detect true variance in effects, we will produce several homogeneity measures, standard errors, and associated probability levels, including Cochrane's Q, I2, and H (Higgins, Thompson, Deeks & Altman, 2003). If such tests do not rule out heterogeneity of effects, we will investigate potential moderators of the effects, such as the duration of the intervention, or true experimental versus quasi-experimental design. Because most potential moderators will be categories of studies, we will employ an analysis of variance approach to explore the relation between potential moderators and effect sizes. Using this method, we will produce a weighted mean effect size and 95% confidence interval for each subgroup, and tests of within-group and between-group homogeneity of effects.
For each meaningful subgroup, we will produce estimates of mean effect sizes and standard deviations from fixed and random-effects models. We will also produce weighted mean effect sizes and 95% confidence intervals for each homogeneous group on each variable. We will also produce weighted and unweighted Forest plots illustrating the distributions of effects and their associated confidence intervals. The weighted plots will include measures of the influence of individual studies on the effect size estimates obtained.
In addition to conducting both fixed and random effects meta-analyses to assess robustness, we will carry out sensitivity analyses by conducting the meta-analyses with and without studies that are 1) of questionable eligibility, 2) have outlying effect sizes, or 3) are of poorer quality. Among studies reporting such information, we will examine the impact of attrition and imputation/exclusion of missing data on the results (Deeks et al., 2001).
4.7 Treatment of qualitative research
At present, there are no plans to include qualitative research in this systematic review.
5.0 Timeframe
6.0 Plans for updating the Review
The review will be updated every three years.
7.0 Acknowledgements
Support for this work to date has been provided by the Jerry Lee Foundation, the University of Illinois, Department of Psychiatry, and a University of Illinois, Faculty Scholar Award to the first author. We also wish to acknowledge the generous aid in regard to databases and advice from Mark Lipsey, David DuBois, and Steven Aos.
8.0 Statement Concerning Conflict of Interest
None