Abstract
Linked article:
II. Background for the Review
Nationwide, an estimated 8 million children between the ages of 5 and 14 are frequently unsupervised after school (NIOST, 2003). Recent data reveals that more than two-thirds of low- and moderate-income youth do not have parental supervision available after-school due to parental work requirements (Long & Clark, 1998; U.S. Bureau of Labor Statistics, 2000). These statistics are not surprising given the need for low-income families to meet pubic assistance work requirements, and the correlation between low to moderate income with single parent households.
Research has linked such unsupervised time with increased risk-taking behaviors, victimization, and poorer academic outcomes (Dwyer et al., 1990; Newman et al., 2000; Osofsky, 1999; Posner & Vandell, 1999; Richardson et al., 1989; U.S. DHHS, 1995; U.S. DOE & U.S. DOJ, 2000). Unstructured, unsupervised after-school time has increasingly been seen by policy makers and the public as holding “risk and opportunity” (Hofferth, 1995). And, after-school programs have been touted as a means to reduce negative behaviors and improve positive outcomes, especially for lower-income, urban students.
Within the last few years, after-school programming has seen tremendous growth. The federal government, states, localities and private foundations have invested substantial money and resources in programs. For example, appropriations for 21st Century Community Learning Centers have increased from $40 million in 1998 to the near $1 billion that is currently appropriated for the program.
In this short period of time, the number and strength of advocacy groups in this field has also experienced a great deal of growth. As evidence of their voices, tremendous fervor surrounded the recent release of the first year findings from the national evaluation of 21st Community Learning Centers (CCLCs) (U.S. DOE, 2003). Several criticisms were directed at this report 2 , but arguably the strong responses to the report's primarily null findings were likely more reactions to the use of a single, high profile experimental study to recommend a 40% reduction in 21st CCLC appropriations.
The resulting lobbying and grass roots efforts to maintain or increase the 21st CCLC appropriations served to highlight that continued support for a high investment in and expansion of after-school programming is not supported by a large or strong research base. Several quasi-experimental and non-experimental studies are frequently cited as evidence that after-school programming promotes positive developmental and emotional outcomes in low-income youth, may help to improve academic outcomes, and may decrease student participation in criminal or violent activities (Baker & Witt, 1996; Foley et al., 2000; Huang, et al., 2000; Jones & Offord, 1989; Le and Hamilton, 2001; McLaughlin & Irby, 1994; Posner & Vandell, 1994; Ross et al., 1992; Schinke et al., 2000; U.S. DOE & U.S. DOJ, 2000; Grossman et al., 2002; Welsh, et al., 2002). Some of these studies have compared participants’ outcomes to those of non-participating youth. However, their designs cannot completely control for differences (like motivation) between the youth who volunteered for the program and those who did not.
Does this matter? Recent research shows that it does. Strong bias affects the estimates from quasi-experimental studies, especially for studies of voluntary participation in programs (Guyat, et al., 2000; Agodini & Dynarski, 2001; Weisburd, Lum & Petrosino, 2001; Wilson & Lipsey, 2001; Glazerman, Levy & Myers, 2003). For this reason, a comparison of outcomes between program and non-program youth does not accurately measure what the program has added to participants’ development. Further work is being conducted to assess whether any quasi-experimental methods accurately replicate experimental results, and under what conditions (Glazerman, Levy & Myers, 2003). Until the field is more confident in how to control for the possible biases and/or which kinds of quasi-experimental studies, if any, can be reasonably substituted for experimental designs, it seems imprudent to combine reliable with unreliable impact estimates when answering questions on program effectiveness.
The field of after-school programming could benefit from a review of high-quality experimental, evaluations of the programs that have experienced the greatest amount of growth since funding for programming increased in order to better understand the current state of the evidence, and the impact these programs may be having on participants’ academic, behavioral, and social/emotional outcomes. The current increase in funding has allowed more districts and schools to offer programs that combine recreational, academic, and youth development programming. These programs are thus likely attended by a majority of youth in organized after-school programming and are therefore currently of great interest to policy makers. 3
Recent reviews of evidence
We identified seven recent, major reviews of research on the impact of out-of-school programs on student outcomes. Three of these reviews focus on programs specifically designed to promote positive youth development (Catalano, et al., 2002; National Research Council, 2001; Roth et al, 1998). The other four reviews have considered programs with additional goals, like improved academic achievement. These four reviews are most relevant to the current interest in after-school programs, and so are discussed below.
Review of Extended-Day and After-School Programs and their Effectiveness identified and described programs with an educational focus that had shown some evidence or promise of effectiveness and/or had the potential for dissemination and replicability (Fashola, 1998). Both experimental and quasi-experimental studies (“well-matched treatment and comparison groups”, p.7) that measured achievement and other outcomes were included. Fashola concluded that a number of the programs looked promising, but few of the collected studies had rigorous designs and almost all of the studies suffered from selection bias, limiting the conclusions that could be drawn with confidence.
The information and issues raised in the Fashola review were useful as a guide to the state of knowledge in the field at the time the report was written, and to suggest areas for further study. However, its current usefulness for guiding after-school programming is limited for three reasons. First, not all the studies included in the review were implemented in the after school context. Some were programs that existed within the normal school day, although the author noted that they had the potential for replication during the after school hours. Second, the evaluations included mentoring and tutoring programs, which are fundamentally very different than what we will define as more traditional after-school programs. Finally, it is not clear the extent to which the review included studies which showed null or negative impacts.
Eccles and Templeton (2002) conducted a review similar to the Fashola (1998) review by casting a broad net in defining after-school programs, including mentoring programs, sexual education programs, or programs with a very low intensity and/or short duration. This review also included both quasi-experimental and experimental design studies. Similar to Fashola, Eccles and Templeton found that the research field in this area is still quite young and inconsistent, citing few experimental studies, little overlap between evaluated outcomes between studies, and the lack of implementation and process data to help understand impact findings. But, they do draw some preliminary conclusions from the experimental and quasi-experimental studies. They suggest that “there is growing evidence that youth programs focused on both prevention and promotion do increase positive outcomes and decrease negative outcomes” (p.172) and that “programs not explicitly focused on academic instruction produce gains in academic achievement, school engagement, and high school graduation rates … (as well as) declines in school-related problem behaviors.” (p.172)
Evaluations of After-School Programs: A Meta-Evaluation of Methodologies and Narrative Synthesis of Findings uses similar inclusion criteria to the above referenced reviews (Scott-Little et al., 2002). The review suggests that after-school programs may positively affect standardized test scores and homework completion, and that programs may have more of an effect for younger and more academically “at-risk” students. Because their review included both experimental and quasi-experimental designs, the reviewers do not make statements of causality in these areas. However, using results from two experimental studies, they suggest that programs can have positive impacts on participants’ social/emotional outcomes. However, these results are from programs that are beyond the realm of the traditional program experienced by the majority of youth; we are left questioning the generalizability of such findings.
Hollister (2003) reviewed the effectiveness of after-school programs in The Growth in After-School Programs: Status, Issues, and Evaluation Impacts. Unlike prior reviewers, he limited his search to experimental design studies. However, Hollister also cast a broad net in the definition of an “after-school program” (including mentoring, tutoring, remedial schooling, and comprehensive services programs). Looking across the ten experimental design studies identified for review, he found that mentoring and tutoring programs have positively affected in-school and out-of-school outcomes, parent involvement and training has been an effective component, and life-skills training curricula may positively effect some out-of-school outcomes. Because of the diverse characteristics of the included programs, these findings are again difficult to apply to the majority of students currently participating in the more traditional programs which may combine academics with recreation and youth programming.
These four recent reviews leave us with three major questions: Would the review conclusions differ if they only included the type of programming that is experienced by more of a majority of youth, i.e. more traditional after-school programs combining academics, recreation, and youth programming? Would the number of studies held for review differ if a systematic and consistent approach were used to identify and select high quality studies? Would the review conclusions differ if only experimental evaluations were used to measure overall program impacts?
Contribution of this review
As the authors cited above have noted, their reviews were limited by the amount of available evaluation work in this very quickly growing field. However, other factors related to their inclusion criteria limited the conclusions they were able to draw. Because prior reviews included studies of both experimental and quasi-experimental methodologies, and/or programs which look quite dissimilar in design and service delivery, it has been difficult to use such work to answer questions about program effectiveness.
This proposed review will differ from prior reviews in two main ways. First, this review will make use of recently released experimental studies in this field that were not captured by prior reviews. Second, this review will limit the inclusion criteria up front, so that the answers to policy relevant questions, like program effectiveness, can be made with more certainty and drawing upon a more homogeneous set of program models.
III. Objectives of the Review
As after-school programs become increasingly popular, there is a strong belief that they will improve outcomes for participants, yet little collective evidence to suggest if programs are moving closer to meeting their goals. To date, there is a limited scientific knowledge base to guide fundamental decision making about program expansion and improvement. Policymakers and program administrators would benefit greatly from knowing whether or not we can reliably answer the following questions and, if so, what the answers are: Does access to after-school programs impact academic, social/emotional, and/or behavioral outcomes for youth? If so, what is the range of impact estimates across key outcomes and program models? For what groups of youth have after-school programs been more and less beneficial? Is the range of variation in the program models and targeting sufficient to draw any generalizations? Among the program models and settings evaluated, do some seem more beneficial to youth than others? What are the distinguishing characteristics of those more and less successful programs?
A systematic approach to the identification, compilation and analysis of well-designed and executed studies measuring the impacts of program participation on student outcomes will allow us to answer these questions as best one can given the available research base, and will also point out the limits of our knowledge.
For those studies that meet criteria for inclusion, this systematic narrative review and meta-analysis will: Describe the programs, evaluation methodologies, outcome measures used and analysis methods. Identify the programs that have shown evidence of effectiveness for academic, behavioral, and/or social/emotional outcomes, and compare and contrast the characteristics of these programs and implementation settings with other evaluated programs. If feasible, use individual program impact estimates from rigorous experimental design studies to determine overall program impacts on outcomes. This core analysis will serve as the “benchmark” which can be augmented by future studies. If feasible, perform an analysis comparing the outcomes of studies of programs with different goals and activities. Policymakers are increasingly interested in which kinds of programs are more effective at improving different student academic, behavioral, and/or social/emotional outcomes. If feasible, identify target groups that have tended to benefit from the programs and compare their characteristics and program experiences with those of youth not showing evidence of benefits. Make recommendations for future evaluations so that studies provide the data which could contribute towards better answering the key questions outlined above.
IV. Methodology
Inclusion criteria
Studies must meet the following criteria to be included in this narrative review and meta-analysis.
Characteristics of the intervention
Programs must operate after-school, and may or may not have a before-school component. They must not be targeted specifically at youth with particular special needs, such as learning disabilities, physical disabilities, emotional problems, or behavioral problems. Summer school programs or programs that have a significant in-school component will not be included. Programs must provide academic, recreational, and/or positive youth development activities. However, the primary means of attempting to achieve positive outcomes in these areas must not be through a one-on-one mentoring or tutoring format. While most of these programs do operate during the after school hours, the design and delivery of such programs assume a much different relationship between program teachers/volunteers and youth, and therefore such programs would fall out of the bounds of this particular review.
In other words, programs chosen for this review will fall under the general heading of more traditional after-school programs than some of the programs captured by prior reviews. These more traditional programs have arguably experienced the greatest amount of growth since funding for programming increased, allowing more districts and schools to offer programs. These types of programs are thus likely attended by a majority of youth in organized after-school programming. Such programs are therefore currently of great interest to policymakers. However, we want to make clear that we expect that there will still be some degree of heterogeneity among program models. For example, some programs may offer more recreational than academic activities, while others might focus on academics for most of the program hours and then only offer recreational activities one day a week.
Programs could operate in a variety of settings – schools, community centers, and religious institutions. But, the evaluation should report results separately for programs operating in different settings. Finally, we will limit our focus to studies of interventions conducted in North America. 4 We will not specifically search for any international studies because the intervention context would likely be much different than that of the U.S. and Canadian programs. However, we will provide a full citation of any international studies we find to aid future reviewers in this field.
Characteristics of the target population
Participants must include youth enrolled in regular public or private K-12 schools; typically, we expect these youth to range in age from 5–19. This large range was chosen so that studies which span a wide range would not be excluded. Also, some potentially relevant programs are designed to prevent risk-taking behavior for students age 13 and above.
Types of studies
In order to be included for consideration in the review, evaluations must use well-implemented experimental designs, and the control group must not have received a similar intervention. Parental consent for study must have been received prior to random assignment. Otherwise, we would assume there to be selection bias among those consenting for study after the random assignment process. Furthermore, description of the methodology must be clear and complete enough that the reviewers can judge the rigor of the implementation of design. Studies should also provide a thorough description of program goals and activities.
We will include only studies published in 1982 or later for two reasons. First, Lauver's (2002) prior search did not identify any studies prior to this date. Second, there was little public support for programs and evaluations in this field before the 1990s.
Study quality
We propose to retain for inclusion those studies which meet the criteria outlined above. Then, specific codes designed to judge quality of the design and implementation of the study will be applied. Only those studies that are judged to have a high likelihood of generating unbiased estimates of program impacts will be included in the statistical reporting on estimated program impacts. Dimensions that will be considered when judging study quality are found in Appendix A.
Outcome measures
We will include studies with academic (test scores, grades), social/emotional (self-confidence, self-esteem, aspiration, locus of control), and/or behavioral outcomes (risk-taking, time use, school attendance). We understand that evaluators may have chosen to measure outcomes that may not have been the specific goals of the programs. However, we don't believe that these outcomes should be excluded from the analyses. Prior research and reviews have used non-experimental research to generate hypotheses about the cross-over between seemingly incompatible program goals and measured outcomes. For example, the review recently conducted by Eccles and Templeton (2002) suggests that programs not explicitly focused on academic instruction can produce gains in achievement, school engagement, and high school graduation rates. Several other recent studies also support the hypothesis that youth development goals can promote academic achievement (National Research Council and Institute of Medicine, 2002). Additionally, many logic models for after-school programs include both academic and social/emotional outcomes, recognizing that the two are likely closely related (HFRP, 2003; Lauver, 2002; Dynarski, et al., 2001).
Upon the reporting of results, the reviewers will take care not to suggest that programs which may not have had academic goals should be held accountable for achieving them. Similarly, programs which may not have had behavioral goals should not be held accountable for achieving them. However, it would be of great policy importance to understand what outcomes are being achieved, and from which types of programs.
Use of quasi-experimental studies
Recent reviews and studies have highlighted the probable certainty of strong bias affecting the estimates from quasi-experimental studies, especially for studies of voluntary participation in programs for children and adolescents like after school programs (Guyat, et al., 2000; Agodini & Dynarski, 2001; Weisburd, Lum & Petrosino, 2001; Wilson & Lipsey, 2001; Glazerman, Levy, & Myers, 2003). However, the field of after-school programming is awash in quasi-experimental studies as funding or feasibility has traditionally limited the possibility of experimental evaluations of this particular intervention.
Further work is being conducted to assess whether we can identify any quasi-experimental methods that accurately replicate experimental results, and under what conditions (Glazerman, Levy, & Myers, 2003). Until the field is more confident in how to control for the possible biases and/or which kinds of quasi-experimental studies, if any, can be reasonably substituted for experimental designs, it seems imprudent to combine reliable with unreliable impact estimates. We will, however, provide references for quasi-experimental studies meeting the following criteria outlined by Glazerman, Levy, & Myers (2003): Methods which use linear regression or propensity score matching. A comparison group drawn in a way that increases the chances of comparability with the program group.
5
Pre-intervention measures of the outcomes used to adjust for program/comparison groups’ initial differences.
Using these criteria to find references to quasi-experimental studies will facilitate possible future work to explore the sensitivity of impact estimates across studies employing different methodologies.
Literature search strategy for identification of appropriate studies
Personal contacts
The field of after-school evaluation is young and quickly expanding after the awards of 21st Century Community Learning Center grants. The Harvard Family Research Project (HFRP, 2003) and Lauver (2002) have extensive knowledge of the current programs that have gone or are currently undergoing rigorous evaluation. The Harvard Family Research Project (HFRP) has recently published an on-line Out-Of-School-Time Evaluation database (HFRP, 2003). The database includes a program and evaluation description of 54 out-of-school studies of summer and after-school programs. Lauver (2002) searched for “high-quality” descriptive and impact evaluations of after-school programs for low-income youth by conducting an extensive search of major databases in January, 2001 (ERIC, PsychINFO, Sociology Abstracts). Lauver also searched the Internet and directly contacted program administrators, evaluators and other colleagues in the field to identify studies. Both sources have gone to great lengths to identify the available studies in this field and have applied well-defined criteria before including studies in their work. Thus, we will assemble our initial collection of studies to consider for inclusion from these reviews. We will also contact experts in the field to assist us in updating our search with the most current evaluations.
Database searches
We will search ERIC, Education Index, PsychINFO, and Dissertation Abstracts for the past 20 years. Keyword searches will likely include the following words, but may be tailored given subject headings or a given database: after-school programs
6
after-school education after-school centers
AND evaluation outcome impact
Many of the studies that are included in the Lauver (2002) review and HFRP database were not published in peer-reviewed journals, but rather were publications of the evaluator or funder. For this reason, journal publication bias may not be a concern. Coding will identify the source of the study and the funding, and these particular issues of potential bias will be considered at the time of analysis.
Most studies that have previously been identified by HFRP, Lauver (2002), and the prior reviews cited above first appeared as project reports, and were not also published in peer review journals. Given this, and a limited supply of time and resources, we will not hand search journals for this review. However, we will document the date that the search of the major databases ended, so that future reviews can capture all journal publications after that date that are then catalogued by the databases. As a result of this strategy, journals not catalogued by the databases will not be captured, but knowledge of the field suggests that we will likely not miss rigorous experimental design studies of after-school programs when drawing such search boundaries.
Prior reviews and reference lists
We will identify studies for inclusion from any prior reviews conducted in this field, and also by searching the reference lists of other studies identified through the search process.
Internet searches
A keyword search using a search engine such as google.com will be included. We anticipate that major new research studies could be identified through this mechanism prior to the results appearing in journals.
Selection of studies
Both primary reviewers will review the abstracts of all eligible studies, and will indicate which studies should be retrieved. Only those studies which appear to meet the above criteria for inclusion will be retrieved. Any disagreements between the reviewers’ recommendations will be resolved by a default decision to retrieve any study that is in question. Both reviewers will independently read the full studies retrieved and recommend which studies should be included in the review. Any disagreements between the reviewers’ decisions will be resolved by the advisor to the reviewers, Dr. Rebecca Maynard.
Study coding strategies
Dimensions that will be considered when coding are found in Appendix B. Both principal reviewers will code the first five studies. Differences in coding will then be resolved in order to establish inter-coder reliability. If the availability of time and resources prevents double-coding of any remaining studies, one researcher will then independently code all the other studies, while a second coder will code a random sample of these remaining studies. Key fields, like outcome measures, will be double-coded for all studies. The reviewers will attempt to contact the authors of studies that are missing key data that is essential for the review.
Data Analysis
Because we have drawn very discrete boundaries around the types of after-school programs and evaluation design which would be included in this review, we anticipate that any meta-analysis conducted will include a modest number of studies. Although the design and delivery of these programs may look quite similar, and we would not be comparing “apples to oranges”, it is likely (given the findings from prior reviews) that the evaluations measured different outcomes in different ways. It therefore may not be prudent to calculate the “effect” of each individual study, and then the “overall effect” of all available studies. In an effort to determine whether after-school programming has a greater effect in any one area, we would instead conduct separate meta-analyses of key outcome areas, like GPA, math scores, reading scores, attendance, and behavior. If a study measures a key outcome in several ways, we will ensure that each study only contributes one data point to the analysis for each key outcome in order to ensure independency of the findings. 7
Some after-school programs have had multiple evaluations over time. In these cases, we will analyze the outcomes by the duration of the follow-up period (e.g., outcomes at the conclusion by one semester, two semesters, one year, two years, etc.). If a sufficient number of studies are available, we will also analyze outcomes of such studies by investigating the change in effect size across time.
We expect to use Comprehensive Meta-Analysis software developed by Biostat to conduct any meta-analyses. However, we also anticipate needing to address issues of different output reporting formats and missing data to support the meta-analysis. We will consult Cooper and Hedges (1994) for guidance on when and how to impute missing data, and Lipsey and Wilson (2001) for assistance using different reporting formats. We will not be able to identify what effect size metric will be used for the meta-analytic calculations until we have identified the outcomes measured and how they have been reported. We plan to report the effect sizes, plot them in a Forest plot, and also report the natural units (means, proportions). Natural units are likely more understandable and relevant to consumers of reviews. 8
Fixed versus Random Effects
Given our knowledge of the experimental evaluations of after-school programs prior to conducting the review, we are fairly confident that we will identify a limited number of studies. Therefore, these studies likely would not represent the universe of possible studies that can be done on after-school programs that share the characteristics of the intervention we put forth for this review. For this reason, we will likely choose a fixed-effects model for all meta-analytic calculations (Cooper & Hedges, 1994).
Treatment of qualitative research
In order to meet the objectives of this review, qualitative data will be an integral part of the analysis. The systematic review will include qualitative research findings from well-designed complementary process, implementation, and operational studies, which inform the associated impact study findings. Observation and interview data will provide details of program goals, activities, connections made to the school day, and program context. The qualitative data will be used to inform how programmatic style, leadership, and context can influence the outcomes of after-school programs.
V. Anticipated Timeframe
VI. Plans for Updating the Review
The authors will update the review upon the release of the second year findings from the national evaluation of 21st Century Community Learning Centers being conducted by Mathematica Policy Research. Then, the review will be updated as warranted, pending funding.
VII. Acknowledgements
This protocol was completed with the help of many colleagues, including the members of the University of Pennsylvania's Graduate School of Education seminar on conducting systematic reviews and meta-analysis, organized by Dr. Rebecca Maynard.
VIII. Statement Concerning Conflict of Interest
Lauver has completed an impact evaluation of an after-school program (utilizing random assignment) that will be reviewed for its potential inclusion in the systematic review of the literature. Lauver will remove herself from the decision to include this research study in the review; Zief and Maynard will code this research study and make decisions regarding its eligibility.
Footnotes
2
These criticisms included program sampling, implementation status of programs, and time frame for data collection.
3
We draw bounds around the programs that are likely more in supply, and of great interest to this review, in Section IV: Methodology.
4
5
The study suggests that bias was lower “when the comparison group was drawn from within the evaluation itself rather than from a national dataset, when it was locally matched to the treatment population, and when it was itself drawn as a control group in an evaluation of a similar program or the same program in a different study site” (Glazerman, Levy & Myers, 2002, p. 47).
6
Exploratory searching revealed that using “extended day” as a key word pulled mostly irrelevant studies on adding more hours to the school day, or more days to the school year.
7
For example, Study 1 may report average grades in the major subjects while Study 2 may report an overall GPA. The reviewers would impute an overall GPA for Study 1 which would be included in the analysis.
8
It should be noted that it may not be appropriate to compare grades and some test scores reported as natural units. For example, an “A” from District 1 may not equal an “A” from District 2. The reviewers will make readers of the review aware of the appropriateness of comparing the outcomes from different reviews, and the potential benefits of using effect sizes as a standardized measure in these cases.
9
This checklist is in draft form, and will be updated throughout the review process.