Abstract
BACKGROUND
With the expansion of regional and national economies into a global marketplace, education has even greater importance as a primary factor in allowing young adults to enter the workforce and advance economically, as well as to share in the social, health, and other benefits associated with education and productive careers. Dropping out of school before completing the normal course of secondary education greatly undermines these opportunities and is associated with adverse personal and social consequences. Dropout rates in the United States vary by calculation method, state, ethnic background, and socioeconomic status (Cataldi, Laird, & KewelRamani, 2009). Across all states, the percentage of freshman who did not graduate from high school in four years ranges from 13.1% to 44.2% and averages 26.8%. The status dropout rate, which estimates the percentage of individuals in a certain age range who are not in high school and have not earned a diploma or credential, is slightly lower. In October 2007, the proportion of noninstitutionalized 18–24 year olds not in school without a diploma or certificate was 8.7%. Males are more likely to be dropouts than females (9.8% vs. 7.7%). Status dropout rates are much higher for racial/ethnic minorities (21.4% for Hispanics and 8.4% for Blacks vs. 5.3% for Whites). Event dropout rates illustrate single year dropout rates for high school students and show that students from low-income households drop out of high school more frequently than those from more advantaged backgrounds (8.8% for low-income vs. 3.5% for middle income and 0.9% for high income students). The National Dropout Prevention Center/Network reports that school dropouts in the United States earn an average of $9,245 a year less than those who complete high school, have unemployment rates almost 13 percentage points higher than high school graduates, are disproportionately represented in prison populations, are more likely to become teen parents, and more frequently live in poverty (2009). The consequences of school dropout are even worse for minority youth, further exacerbating the economic and structural disadvantage they often experience.
School dropout has implications not only for the lives and opportunities of those who experience it, but also has enormous economic and social implications for society at large. For instance, the National Dropout Prevention Center/Network (2009) reports that each annual cohort of dropouts costs the United States over $200 billion during their lifetime due to lost earnings and unrealized tax revenue; and even a 1% increase in high school graduation rates could save over $1 billion in incarceration costs. The Organisation for Economic Co-operation and Development (2009) has similarly documented the tremendous social and economic gains associated with secondary school completion in OECD member countries.
A relatively large number of intervention and prevention programs in the research literature give some attention to reducing dropout rates as a possible outcome. The National Dropout Prevention Center/Network, for instance, lists 192 “model programs.” Relatively few of those programs, however, bill themselves as dropout programs; many focus on academic performance, risk factors for dropout such as absences or truancy, or indirect outcomes like student engagement, but may also include dropout reduction as a program objective. The corresponding research domain includes evaluations of virtually any program provided to students for which dropout rates are measured as an outcome variable, regardless of whether they are billed as dropout programs. To represent the full scope of relevant research on this topic, all such programs should be considered in a review of dropout programs. However, because we are interested in summarizing the research on dropout programs that could be implemented by schools, we narrow our focus to programs that can be implemented in school settings or under school auspices.
There have been a handful of systematic reviews on the effects of prevention and intervention programs on school dropout and completion outcomes. However, the restrictive inclusion criteria and methodological weaknesses of these reviews preclude any confident conclusions about the effectiveness of the broad range of programs with dropout outcomes, or the potential variation of effectiveness for different program types or subject populations. For instance, the U.S. Department of Education's What Works Clearinghouse report on dropout prevention found only 15 qualifying studies that reported outcomes on direct measures of staying in school or completing school (http://ies.ed.gov/ncee/wwc/reports/dropout/topic/#top). This report, however, restricted discussion to interventions in the United States and did not include a meta-analysis of program effectiveness or examine potential moderators of program effectiveness. Another review on best practices in dropout prevention summarized the results of 58 studies of dropout programs (ICF, International, 2008). That report presented effect sizes primarily for individual program types and did not examine potential moderators or examine the influence of study method on effect size. The report also presented a narrative review of important variables associated with implementation quality, but implementation quality was not analyzed in a meta-analysis framework.
Two other systematic reviews have focused on the effectiveness of prevention and intervention programs to reduce school dropout or increase school completion (Klima, Miller, & Nunlist, 2009; Lehr et al., 2003). In their review, Lehr et al. (2003) identified 17 experimental or quasi-experimental studies with enrollment status outcomes. This review was completed seven years ago, and thus does not include the most recent studies. The authors did not perform a meta-analysis because they felt that the dependent variables differed too greatly across studies to create meaningful aggregates. This circumstance prevented the authors from examining the differential effectiveness of programs with different treatment or participant characteristics, something we plan to do in the proposed systematic review. In a more recent review, Klima et al. (2009) identified 22 experimental or quasi-experimental studies with dropout, achievement, and truancy outcomes. However, this review excluded programs for general “at-risk” populations of students (e.g., minority or low socioeconomic status samples), as well as programs with general character-building, social-emotional learning, or delinquency/behavioral improvement components. These exclusion criteria therefore limited the conclusions that could be drawn about the broader range of programs that aim to influence school dropout and completion outcomes. Further, this review only presented mean effect sizes for different types of interventions, and did not examine the potential variation of effects for different subject populations.
The findings of the Klima et al., (2009) and Lehr et al., (2003) reviews have some similarities. Both teams highlight the dearth of high-quality research on dropout programs, and mention especially the lack of key outcomes such as enrollment (or presence) at school and dropout. Both reviews demonstrate that some of the included programs had positive effects on the students involved. Lehr and her colleagues do not identify specific programs that were particularly effective or ineffective, but focus rather on implementation integrity as a key variable and emphasize the importance of strong methodologies for future research on dropout programs. Klima and colleagues conclude that the programs they reviewed had overall positive effects on dropout, achievement, and attendance/enrollment. They highlight alternative educational programs, such as schools-within-schools, as particularly effective. The Klima review also suggests that alternative school programs, that is, programs in separate school facilities, were ineffective. Overall, these two reviews identify several important potential moderators that will be included in the coding scheme for the proposed review. These include implementation quality, treatment modality, and whether programs are housed in typical school facilities or in alternative school locations.
OBJECTIVES
The objective of the proposed systematic review is to summarize the available evidence on the effects of prevention and intervention programs aimed at primary and secondary students for increasing school completion or reducing school dropout. Program effects on the closely related outcomes of school attendance (absences, truancy) will also be examined. Moreover, when accompanying dropout or attendance outcomes, effects on student engagement, academic performance, and school conduct will be considered.
The primary focus of the analysis will be the comparative effectiveness of different programs and program approaches in an effort to identify those that have the largest and most reliable effects on the respective school participation outcomes, especially with regard to differences associated with treatment modality, implementation quality, and program location or setting. In addition, evidence of differential effects for students with different characteristics will be explored, e.g., in relation to age or grade, gender, race/ethnicity, and risk factors. Because of large ethnic and socioeconomic differences in graduation rates, it will be particularly important to identify programs that may be more or less effective for disadvantaged students.
The ultimate objective of this systematic review is to provide school administrators and policymakers with an integrative summary of research evidence that is useful for guiding programmatic efforts to reduce school dropout and increase school completion.
METHODOLOGY
Criteria for inclusion and exclusion of studies in the review
Studies must meet the following eligibility criteria to be included in the systematic review.
Types of interventions
There must be a school-based or affiliated psychological, educational, or behavioral prevention or intervention program, broadly defined, that involves actions performed with the expectation that they will have beneficial effects on student recipients. School-based programs are those that are administered under the auspices of school authorities and delivered during school hours. School affiliated programs are those that are delivered with the collaboration of school authorities, possibly by other agents, e.g., community service providers, and which may take place before or after school hours and/or off the school grounds. Community-based programs that are explicitly presented as dropout prevention or intervention programs will be included whether or not a school affiliation is evident. Other community-based programs that may include dropout among their goals or intended outcomes, but for which dropout or related variables are not a main focus, and which have no evident school affiliation, will be excluded.
We expect that programs that might be excluded for being community-based with no school affiliation or dropout focus, but that happen to assess school dropout outcomes, would mainly be delinquency or drug prevention or treatment programs or. The rationale for this exclusion is that we believe these kinds of programs are likely to be outside the realm of strategies that school administrators might consider when selecting programs for dropout prevention or treatment.
Types of participants
The research must investigate outcomes for an intervention directed toward school-aged youth, defined as those expected to attend pre-k to 12th grade primary and secondary schools, or the equivalent in countries with a different grade structure, corresponding to approximately ages 4–18. The age or school participation of the sample must be presented in sufficient detail to allow reasonable inference that it meets this requirement. Recent dropouts who are between the ages of 18–21 will also be included if the program under study is explicitly oriented toward secondary school completion or the equivalent.
General population samples of school-age children will be included. Samples from populations broadly at risk because of economic disadvantage, individual risk variables, and closely related factors will also be included (e.g., inner city schools, students from low SES families, teen parents, students with poor attendance records, students who have low test scores or who are over-age for their grade).
Samples consisting exclusively of specialized populations, such as students with mental disabilities or other special needs, will not be included. The rationale for this decision is that dropout programs designed exclusively for students with mental or physical disabilities that generally prevent them for attending mainstream classes and typical schools are not likely to be applicable to mainstream students. However, inclusion of some such individuals in a broader sample in which they are a minority proportion does not make that broader sample ineligible. Students with learning disabilities, such as dyslexia, that generally don't require them to be in specialized schools or classrooms (i.e., they attend mainstream classes and typical schools) will be included.
Types of research designs
To be included a study must use an experimental or quasi-experimental design. Specifically, it must involve comparison of treatment and control conditions to which students are: (1) randomly assigned; (2) non-randomly assigned but matched on pretests, risk factors, and/or relevant demographic characteristics; or (3) non-randomly assigned but statistical controls (e.g., covariate-adjusted means) or sufficient information to permit calculation of pre-treatment effect sizes on key risk variables or student characteristics is provided. 1 Treatment-treatment studies that compare two or more treatments to each other without a control group will be included if one treatment group receives a “sham” or “straw-man” treatment that is equivalent to a control condition, or if one of the treatments is a practice as usual condition in which that practice is not a distinctive program delivered at a relatively high level. Posttest-only non-equivalent comparisons (not randomized, matched, or demonstrating equivalence) will not be included. Single-group pretest-posttest designs will not be included.
Types of outcomes
To be included, a study must assess intervention effects on at least one eligible outcome variable. Qualifying outcome variables are those that fall in or are substantially similar to the following categories: (a) School completion/dropout; (b) GED completion/high school graduation; (c) Absences or truancy. If a measure absences, truancy, or attendance is the only outcome provided, the majority of the students in the sample must be age 12 or older. The rationale for this exclusion is practical; there is a large literature on programs designed to influence attendance for elementary school age children that is beyond the scope of this review. Moreover, there is already an active Campbell Collaboration protocol on this topic (Maynard, Tyson-McCrea, Pigott, & Kelly, 2009).
Date and form of publications
Eligible studies should be relatively modern to be applicable to contemporary students. Therefore, the date of publication or reporting of the study must be 1985 or later even though the research itself may have been conducted prior to 1985. If, however, there is evidence in the report that the research was actually conducted prior to 1980 (more than five years before the 1985 cutoff date), then the study will not be included. Eligible studies can be published in any language and conducted in any country as long as they meet all other eligibility criteria. Campbell Collaboration affiliates outside the United States will be asked to assist with the location of studies published in other countries and languages other than English.
Study inclusion decision-making
Inclusion and exclusion decisions will be based on readings of the full reports for each study judged potentially relevant during the search procedure (described below). Any questions or doubts about the inclusion of a study will be discussed with the primary reviewer and/or a second reviewer.
Search strategy for identification of eligible studies
Resources searched
A comprehensive and diverse strategy will be used to search the international research literature for qualifying studies reported during the last 25 years (1985–2010). The wide range of resources searched is intended to reduce omission of any potentially relevant studies and to ensure adequate representation of both published and unpublished studies.
Electronic bibliographic databases to be searched include Dissertation Abstracts International, Education Abstracts, Education Resources Information Center (ERIC), ISI Web of Knowledge (Social Science Citation Index, SSCI), PsycINFO, and Sociological Abstracts.
Research registers to be searched include, the Cochrane Collaboration Library, the National Dropout Prevention Center/Network, the National Research Register (NRR), the National Technical Information Service (NTIS), and the System for Information on Grey Literature (OpenSIGLE). International research databases such as Australian Education Index, British Education Index, CBCA Education (Canada), Canadian Research Index will also be searched.
Reference lists in previous meta-analyses and reviews, and citations in research reports screened for eligibility will also be reviewed for potential relevance to the review.
Correspondence with researchers in the field will also be maintained throughout the review process.
Search terms and keywords
A comprehensive list of search terms and key words related to the population, intervention, research design, and outcomes will be used to search the electronic bibliographic databases. These include the following terms, with synonyms and wildcards applied as appropriate:
School dropouts, school attendance, truancy, school graduation, high school graduates, school completion, GED, general education development, high school diploma, dropout, alternative education, alternative high school, career academy, schools-within-schools, schools and absence, chronic and absence, school enrollment, high school equivalency, school failure, high school reform, educational attainment, grade promotion, grade retention, school nonattendance, school engagement, and graduation rate;
AND
intervention, program evaluation, random, prevent, pilot project, youth program, counseling, guidance program, summative evaluation, RCT, clinical trial, quasi-experiment, treatment outcome, program effectiveness, treatment effectiveness, evaluation, experiment, social program, effective.
The following search terms will be used to exclude irrelevant studies: higher education, post-secondary, undergraduate, doctoral, prison, and inmate.
Description of methods used in primary research
Studies to be included in the review will employ experimental or quasi-experimental research designs that compare outcomes for an intervention group to those for a control or comparison condition. The control or comparison conditions in these studies include youth receiving no treatment, observation only, treatment as usual, or wait-listed control groups.
Most potentially eligible studies include both pretest and posttest measurements that allow calculation of pretest group equivalence, posttest group differences, and pretest-posttest changes. Pretest measurements generally occur at or immediately prior to the beginning of the prevention or intervention program, with posttest measurements occurring at or after the end of the program. The posttest measurements comparing the intervention and comparison conditions are the key outcome measurements of interest for the proposed review. Many studies measure outcomes at multiple follow-up points; the first follow-up occurring at or after program completion will be considered the posttest measurement and subsequent waves will be considered follow-up measurements.
One study that exemplifies the methods likely to meet the eligibility criteria for the proposed review is a program evaluation of Ohio's Learning, Earning, and Parenting (LEAP) Program (Long et al., 1996). In 1989, almost 10,000 teenage parents throughout the state of Ohio were randomly assigned to the LEAP program or a no-treatment control group. The LEAP program used an incentive structure for teens to encourage regular attendance in a program designed to lead to a high school diploma or GED. Because it used random assignment, the LEAP study did not provide pretest group equivalence information for the intervention and control groups, as those differences were presumed negligible given the randomized study design (other studies using quasi-experimental designs, however, must provide such pretest information in order to be included in the proposed review). The key outcomes of interest from the LEAP study are the posttest measurements—in this case measured three years after random assignment. Outcomes measured at posttest included measures of the percent of students in the intervention and comparison conditions who completed 9th, 10th, and 11th grade, completed high school, completed a GED, or were currently enrolled in school or a GED program.
Criteria for determination of independent findings
Multiple reports from single studies, and multiple studies in single reports, will be identified through information on program details, sample sizes, authors, grant numbers and the like. If it is unclear whether reports and studies provide independent findings, the authors of the reports will be contacted.
All codable effect sizes will be extracted from study reports during the coding phase of the review (i.e., we plan to code multiple outcomes and multiple follow-ups measured within the same study). These will be separated according to the general constructs they represent (dropout, attendance, engagement, etc.) and each outcome construct category will be analyzed separately. We expect that some portion of the studies will provide more than one effect size for a particular outcome construct (e.g., report two measures of dropout). This circumstance creates statistical dependencies that violate the assumptions of standard meta-analysis methods. If there are relatively few instances of this for a given construct category, we will retain only one of these effect sizes in the analysis by selecting the construct that is most similar to those used by other studies in that category. 2 For any construct categories where this is relatively common, however, we will retain all the effect sizes in the analysis and use the technique recently developed by Hedges, Tipton, and Johnson (2010) to estimate robust standard errors that account for the statistical dependencies.
Details of study coding categories
Eligible studies will be coded on variables related to study methods, the nature of the intervention and its implementation, the characteristics of the subject samples, the outcome variables and statistical findings, and contextual features such as setting, year of publication, and the like. A detailed coding manual is included in Appendix I. All coding will be done by trained coders who will enter data directly into a FileMaker Pro database using computer screens tailored to the coding items and with help links to the relevant sections of the coding manual. Effect size calculation is built into the data entry screens for the most common statistical representations and specialized computational programs and expert consultation will be used for the less common representations. We will select a 10% random sample of studies for independent double coding. The results will be compared for discrepancies that will then be resolved by further review of the respective study reports. The coding team will be retrained on any coding items that show discrepancies during this process.
Statistical procedures and conventions
Analysis will be conducted using SPSS and the specialized meta-analysis macros available for that program (Lipsey & Wilson, 2001) as well as Stata and the meta-analysis routines available for it (Sterne, 2009).
Effect size metrics
We anticipate using odds ratios as the effect size metric for dropout and other binary outcomes, and standardized mean difference effect sizes as the effect size metric for outcomes measured on a continuous scale (e.g., group differences in average attendance rates). All effect sizes will be coded such that larger effect sizes represent positive outcomes (e.g., less school dropout, higher attendance, less truancy).
All computations with odds ratios will be carried out with the natural logarithm of the odds ratios, defined as follows:
where A and B are the respective counts of “successes” and “failures” in the treatment group, and C and D are the corresponding counts of “successes” and “failures” in the comparison group. The sampling variance of the logarithm of an odds ratio can be represented as:
Analytic results from the logged odds ratios effect sizes will be converted back to the original odds ratio metric for final substantive interpretation.
Standardized mean difference effect sizes (d) will be used as the effect size metric for outcomes measured on a continuous scale:
where the numerator is the difference in group means for the intervention and comparison groups, and the denominator is the pooled standard deviation for the intervention and comparison groups. All standardized mean difference effect sizes will be adjusted with the small-sample correction factor to provide unbiased estimates of the effect size (Hedges, 1981). This small-sample corrected effect size (g) and its sampling variance can be represented as:
where N is the total sample size for the intervention and comparison groups, d is the original standardized mean difference effect size, nG1 is the sample size for the intervention group, and nG2 is the sample size for the comparison group.
During the analytic phase of the project we will determine the number of coded effect sizes in the odds ratio and standardized mean difference metrics in each outcome construct category. If both occur in a given category, we will transform the effect size metric with the smaller proportion into the metric with the larger proportion using the Cox transform shown by Sánchez-Meca et al, (2003) to produce good results for this purpose. This will allow all the effect sizes for that outcome category to be analyzed together. If this involves a large proportion of the effect sizes in any category, sensitivity analyses will be conducted to ensure that the transformed effect sizes and those in the original metric produce comparable results.
Missing data
All reasonable attempts will be made to collect complete data on items listed in the coding manual (see Appendix I). Authors of the reports will be contacted if key variables of interest cannot be extracted from study reports. In the event that a small number of studies continue to have missing data on covariates or moderators of interest to be used in the final analysis, we plan to explore an option for imputing missing values using an expectation-maximization (EM) algorithm, which produces asymptotically unbiased estimates (Graham, Cumsille, & Elek-Fisk, 2003). A series of sensitivity analyses will be conducted to examine whether the inclusion of imputed data values substantively alters the results of the moderator analyses. If the EM algorithm fails to converge, or if other difficulties arise that make this technique not feasible, all resulting analyses will implement listwise deletion of missing data.
Outliers
The effect size distributions for each outcome construct category will be examined for outliers using Tukey's (1977) inner fence as the criterion and any outliers found will be recoded to the inner fence value to ensure that they do not exercise disproportionate influence on the analysis results. The distribution of sample sizes will also be examined and any outliers similarly recoded to ensure that the corresponding weights are not excessively large in any analysis. For odds ratio effect sizes, this examination of outliers will be performed by examining the distribution of weights, rather than sample sizes.
Analytic techniques
All analysis with effect sizes will be inverse variance weighted using random effects statistical models. Specifically, the weighting function will be:
Where wi is the weight for effect size i, Vari is the sampling variance for effect size i as defined above for the respective effect size metric, and τ2 is the random effects variance component estimated for each analysis with a method of moments or maximum likelihood estimator.
The unit of assignment to treatment and comparison groups will be coded for all studies, and appropriate adjustments will be made to effect sizes to correct for variation associated with cluster-level assignment (Hedges, 2007).
Summary and descriptive statistics of the study-level contextual characteristics, methodological quality characteristics, group and subject level characteristics, as well as outcome characteristics will be used to describe the eligible body of research studies. Main effects and moderator analysis will be conducted separately with the effect sizes in each outcome construct category with the latter done as multivariate (meta-regression) analysis when possible to minimize misleading results due to correlated independent variables. Random effects statistical models will be used throughout unless a compelling case arises for fixed effects analysis. Random effects weighted mean effect sizes will be calculated for all studies using 95% confidence intervals and displayed with forest plots. Estimates of Cochrane's Q, I2 , and τ2 will be used to assess variability in the effect sizes.
We plan to code pretest effect sizes when available. If available in sufficient numbers for certain outcomes, it may be possible to use the pretest effect sizes as covariates in our meta-regression models, to control for pre-treatment differences between treatment and comparison groups on the outcome variables. For the dropout outcomes, we do not expect to find many pretest effect sizes, because most programs are likely to involve students who are currently attending school (thus the pretest effect sizes would be zero). For attendance outcomes, we may have sufficient pretest effect sizes to use them in our analyses.
The main objective of the analyses, however, will be to describe the direction and magnitude of the effects of different interventions on the different outcome constructs in a manner that allows their comparative effectiveness to be assessed. Additionally, moderator analysis using meta-regression models will attempt to identify the characteristics of the interventions and student participants that are associated with larger and smaller effects for the various outcome constructs. Based on prior theory and research, the following moderators will be examined for their influence on effect sizes: Treatment modality Implementation quality Treatment duration Program location (typical school facilities, alternative schools, after school programs, non-school settings)
3
Ethnic and or socioeconomic mix of sample Grade level of sample
Examination of funnel plots, the use of Duval and Tweedie's trim and fill method (2000), and Egger's regression test (1997) will be used to assess the possibility of publication bias and its impact on the findings of the review. Sensitivity analyses will be conducted to examine whether any decisions made during analyses substantively influenced the review findings, e.g., transformation between effect size metrics, the way outlier effect sizes and sample sizes were handled, the inclusion of studies with poorer methodological quality within the range allowed by the inclusion criteria, and missing data imputations.
Treatment of qualitative research
Qualitative research will not be included in this systematic review.
SOURCES OF SUPPORT
Work on this review to date has been supported by a contract from the Campbell Collaboration.
DECLARATIONS OF INTEREST
The review authors have no conflicts of interest to report.
AUTHOR(S) REVIEW TEAM
Sandra Jo Wilson, Ph.D.
Peabody Research Institute, Vanderbilt University
230 Appleton Place, PMB 181
Nashville, TN 37203-5721 USA
Phone: (615) 343-7215
Fax: (615) 322-0293
email:
Mark W. Lipsey, Ph.D. Director, Peabody Research Institute, Vanderbilt University. Emily E. Tanner-Smith, Ph.D. Research Associate, Peabody Research Institute, Vanderbilt University. Chiungjung Huang, Ph.D. Professor, National Changhua University of Education, Taiwan. Katarzyna T. Steinka-Fry, M.P.A. Research Analyst, Peabody Research Institute, Vanderbilt University.
Sandra L. Christenson, Ph.D.
Birkmaier Professor of Educational Leadership
University of Minnesota
ROLES AND RESPONSIBLIITIES
PRELIMINARY TIMEFRAME
The research team will begin working on the systematic review upon approval by the Campbell Collaboration editorial staff. We plan to complete the review by February, 2011.