Abstract
Linked article:
2. BACKGROUND FOR THE REVIEW
The main goal of this review will be to examine the effectiveness of intervention programs in affecting the sexual risk-taking behaviors of adolescents and to explore whether there is evidence to suggest that particular intervention strategies have been effective in the settings in which they have been tried. Studies have shown that teens who become pregnant, and especially those who give birth at a young age, face hardships that can have detrimental economic and social consequences, both in the short-term and as the young mothers make the transition to adulthood (Maynard, 1996; McLanahan, 1994; Moore et al, 1993). While the pregnancy and birth rates for teenagers in the U.S. have declined over the past decade, they remain the highest of all industrialized countries (Alan Guttmacher Institute, 1999). Other industrialized countries such as the United Kingdom, Canada, and Australia also have relatively high rates of teen pregnancies and births (Maynard, 1996). It is therefore important to understand the factors that lead teens to engage in risky sexual behaviors and to examine the effectiveness of interventions that attempt to reduce sexual risk-taking.
There is a great deal of research exploring the antecedents of sexual risk-taking behaviors. Researchers have identified a variety of correlates of sexual-risk taking among teens, including community characteristics; school characteristics; family characteristics; biological factors; psychological factors; relationships with peers, parents, and school; as well as attitudes and beliefs concerning sex (Bearman et al., 1999; Blum et al., 2000; and Kirby, 2001).
It is clear that myriad factors influence adolescents' decisions to engage in risk-taking behaviors. However, it is not clear what strategies will be effective in curbing these behaviors. Many types of interventions designed to reduce teen sexual activity, prevent teenage pregnancy, and childbearing have been implemented over the course of the past few decades, 1 focusing on various combinations of the antecedents identified above. In addition, a multitude of actors have entered the pregnancy prevention arena ranging from schools to community-based organizations to religious organizations (U.S. Department of Health and Human Services, 2000). Policymakers, researchers, and practitioners have engaged in ongoing debates concerning the content and timing of pregnancy prevention programs. Specifically, two major issues have been hotly debated: (1) whether school-based sex education programs should have an abstinence-only focus or whether such programs should also include information and education on contraception protection, and (2) whether pregnancy prevention programs should be aimed at younger versus older adolescents.
There have been a number of reviews of teenage pregnancy prevention programs (see Table 1). While there is substantial overlap in the studies included in these prior reviews, these reviews differ in their inclusion criteria, analytic strategies, and their conclusions regarding the effectiveness of interventions. There is considerable inconsistency in terms of the methodological standards reviewers have applied in their inclusion criteria. For example, some include only experimental design research; some include experiments and quasi experiments; and some include even studies with no matched control group (see Table 2). None of these reviews has specified clear criteria as to what constitutes a well-executed study under particular methodology; and few have followed clear and consistent search criteria. While some have conducted statistical meta-analyses, many provide only a narrative description of study findings. Finally, some of these reviews are outdated and others are too narrowly focused.
Reviews Exploring the Effects of Programs on Adolescent Sexual Risk-taking Behaviors
Key Features of Prior Reviews of Programs Designed to Affect Adolescent Sexual Risk-taking Behavior
A. Methods and Findings of Two Prior Reviews
There are two relatively recent prior reviews that attempt to present a comprehensive review of the effect of teen pregnancy prevention interventions warrant brief discussion. One is a narrative and vote-counting review of pregnancy prevention programs in the U.S. and Canada (Kirby 2001). The other is a review of pregnancy prevention programs in North America that follows procedures similar in many ways to the Campbell Collaboration review methods, including conducting a statistical meta-analysis (DiCenso et al. 2002). Interestingly, the two reviews include overlapping but different studies, and they draw different conclusions regarding the effectiveness of intervention strategies.
1. Kirby (2001).
Commissioned by the National Campaign to Prevent Teen Pregnancy, Kirby (2001) includes over 70 primary studies in his update of a 1997 review. This review covers a variety of intervention types, such as sex education programs, school-based clinics, community-wide pregnancy prevention efforts, and youth development programs. 2
Kirby examines the effect of programs on a range of outcomes related to sexual behaviors, contraceptive use, pregnancy, and birth rates. His stated study inclusion criteria encompass evaluations that rely on either experimental or quasi-experimental designs. He presents the specific characteristics and findings for each included study in tabular form and then synthesizes the study findings in his narrative presentation by means of ‘vote counting’ – reporting and drawing conclusions based on the number of studies that reported statistically significant positive (favorable) effects, negative effects, or no effects.
Kirby (2001) concludes from his review that effective programs are characterized by the inclusion of certain key components. However, these conclusions are based on looking at the common characteristics of the programs that showed evidence of statistically significant impacts. The significant impacts were not necessarily observed across all outcomes. Moreover, it appears that many of the programs that had characteristics of the “effective” models did not show evidence of effectiveness.
2. DiCenso et al. (2002)
Unlike the Kirby (2001) review, DiCenso et al. (2002) limited their review to randomized control trials (RCTs). 3 This review, which includes both published and unpublished studies, has clear, thorough, and documented inclusion criteria. DiCenso and her colleagues conducted statistical meta-analysis looking both at the overall study findings and at findings for sub-groups defined by the characteristics of the intervention. The review concluded that there were no program types that consistently reduced sexual risk taking sexual behaviors of adolescents.
One of the major sources of the difference in the conclusion from this review and that in Kirby (2001) is the fact that Kirby included many more studies, most of which used non-experimental methods. The majority of the significant program impact estimates in the Kirby review were from quasi-experimental studies—a finding that is consistent with a methodological study by Guyatt et al. (2000) that preceded the DiCenso et al. review.
The DiCenso et al. review has three main limitations. First, although the review included only RCTs, it included studies regardless of the quality of the study implementation and, indeed, the authors documented nontrivial quality concerns with most of the studies. The authors created a “quality index” that ranges from 0-4 (4 is highest) based on the following four criteria: (1) appropriate randomization, (2) unbiased data collection, (3) a minimum of 80 percent of the sample included in the follow-up outcome data, and (4) less than a 2 percent difference in attrition rates between experimentals and controls. Only 8 of the 26 studies earned a quality score over 2, suggesting that the internal validity of many of the findings may have been compromised.
In their meta-analytic results, the authors present pooled effect sizes for all of the included studies and do not differentiate between studies that are more likely to be internally valid versus those that are not. Second, the authors chose only to include studies conducted in the United States and Canada. Finally, the authors only included studies for which it was possible to create separate effect sizes by gender, thereby, excluding potentially internally valid studies that did not provide gender-specific information.
B. Benefits of the Proposed Review
The proposed study will improve upon the prior reviews, particularly the Kirby (2001) and DiCenso et al. (2002) reviews, in six ways. First, we will focus the review on a clear and policy relevant set of questions in terms of both the intervention and the outcomes. We will focus on interventions with a primary goal of reducing sexual risk-taking behavior, and studies of interventions that include measures of at least one of three key outcomes measures—(1) sexual initiation, (2) sexual activity and contraceptive use, which we use to construct a measure of pregnancy risk, and/or (3) pregnancy.
Second, we include evaluations operating in a broader set of geographical contexts than have most prior reviews, while bounding the search to encompass research on programs that have operated in developed countries with relatively high rates of teen pregnancy. The clarity of the boundaries will make it efficient to augment the review to include a broader set of geographical contexts at a later date.
Third, we will limit the prospective research base to those studies with a strong potential for generating credible (internally valid) findings. Specifically, we will include in the search inventory only randomized control trials that address the study questions within the geographic and language boundaries for the search.
Fourth, we will evaluate whether the research base is an adequate representation of the programs currently in operation and we will assess the appropriateness of combining effect sizes of different program types. While we do not have the resources to do a systematic and thorough assessment of all programs in operation, we will rely on summary reports by government and non-government entities to inventory strategies aimed at preventing teen pregnancy. We will determine the extent to which there is credible evidence of the impacts of these particular strategies by comparing this inventory with the studies included in our review.
Fifth, we will explore differences in outcomes among clusters of programs defined by seemingly important programmatic features such as whether the program has an abstinence-only focus or includes contraception information. Finally, we will note those studies that have been excluded due to study quality considerations and the primary reasons for their exclusion. 4
3. OBJECTIVES OF THE REVIEW
The purpose of this review is to summarize the evidence regarding the effectiveness of interventions designed to reduce sexual risk-taking behaviors among adolescents. Specifically, this review will explore the following questions: What is the corpus of evidence to judge the effectiveness of programs aimed at reducing sexual risk-taking behaviors among adolescents? If there a sizeable body of evidence: What types of programs have been rigorously evaluated? How representative are these programs to the range of programs that exist? What are the estimated average impacts of programs on sexual activity rates
5
, contraceptive use, and pregnancy rates? What are the estimated average impacts of each of the four main categories of programs: (1) one-time consultations, (2) sex education programs with an abstinence focus, (3) sex education programs with contraception component, and (4) multi-component and/or youth development programs?
6
What are the estimated average impacts of programs targeted at different age groups of youth? What are the estimated average impacts of programs for males and for females?
4. METHODOLOGY
The following sections detail our proposed approach to the review. Section A details the criteria for inclusion and exclusion of studies in the review. Section B describes the search strategy and defines the boundaries within which the search will be conducted. Section C documents the methods generally used in component studies. Section D provides the criteria for determination of independence of findings. Section E details our strategy for coding information from the studies included in the review. Section F outlines our plans for conducting the statistical analysis of overall program impacts and for exploring estimated impacts for subgroups defined by program qualities and the characteristics of the target population. Finally, Section G discusses the treatment of qualitative research.
A. Criteria for Inclusion and Exclusion of Studies in the Review
The following ten criteria will determine whether a study will be included in the review data base for purposes of estimating program impacts:
B. Search Strategy
We propose to include all studies that meet the inclusion criteria outlined above, regardless of publication status, and we will code basic information on all studies regardless of the study quality.
1. The methods and bounds of the search
Our search strategy will make use of electronic data bases, hand searching of journals, internet searches, and personal contacts. For each search, we will maintain a log documenting our procedures and their yield. While our intention is to unearth all relevant studies, time and financial constraints prevent us from conducting extensive hand searches and database extraction. However, we believe that our search plan will uncover nearly all relevant studies, and the transparency of our plan will allow future researchers easily to expand their searches beyond our specified boundaries.
2. Databases
Cochrane Controlled Trials Register Database of Abstracts of Reviews of Effectiveness (DARE) (evidence-based medicine) Dissertation Abstracts Franklin (University of Pennsylvania library book search) Econlit ERIC HealthSTAR Index to Current Urban Documents International Bibliography of the Social Sciences Inter-university Consortium for Political and Social Research (ICPSR) MEDLINE The National Criminal Justice Reference Service Abstracts Database PolicyFile POPLINE PsychInfo Public Affairs Information Service (PAIS International) Sage Family Studies Abstracts Social Science Electronic Data Library Social Science Research Network;Social Services Abstracts Social Work Abstracts Sociological Abstracts
Within each database, we will search on the following (pregnancy or pregnant) AND (evaluation or adolescent or teen or prevention) (sex AND education) (sex education) AND (evaluation or adolescent or teen or prevention) abstinence AND (evaluation or adolescent or teen or prevention) clinic AND (evaluation or adolescent or teen or prevention) (HIV or AIDS) AND (evaluation or adolescent or teen or prevention)
Based on preliminary keyword searches, we believe that these general labels will capture the majority of the studies. The remainder of the studies will be accessible through alternative measures of literature searching, as mentioned below.
Abstracts will be collected for all seemingly relevant studies. If the abstract appears appropriate, then the full study will be obtained and reviewed. For more information concerning study inclusion, see Section 2: “Methods of review” and Section 3: “Selection of trials,” below. Due to timing, financial, and technology constraints, all databases will be searched for documents written or published between January 1, 1992, and December 31, 2002.
3. Hand searching
Many of the databases are not fully updated and electronic searching is not fool-proof. Thus, we propose to conduct a hand search of the past 10 years in the following journals that are highly likely to contain relevant studies: AIDS Education and Prevention, Journal of Adolescent Health; Pregnancy Prevention and Youth; Journal of Adolescent Research; American Journal of Public Health; Journal of Health and Social Behavior; Journal of Sex Research; Family Planning Perspectives. We will add journals to hand-search if we find, through our database searches, that other journals have highly relevant studies.
In addition to conducting key-word searches in the on-line databases, we also will hand search those journals noted above for the years 1992 through present. In light of the number of prior reviews in this area, our financial constraints, and our own familiarity with the research, we are quite confident that hand searching journals prior to the last 10 years will yield few to no new studies. By having a clear “stop date” for our hand searching, others who feel we may have missed important studies through this restriction will be able to conduct a complementary search of earlier journal issues.
4. Internet searches
All relevant government, foundation, professional associations and policy research firm websites will be searched. In addition, keyword searches (see the above list of keywords) will be conducted using search engines such as google.com.
5. Other sources
Personal contacts: An initial library of studies will be assembled from Doug Kirby's collection. In addition, principal investigators in a current ongoing evaluation of abstinence-based programs will be consulted.
Reference lists: We will check reference lists of review papers and of primary studies to uncover additional studies that may be eligible for inclusion.
Search log: A comprehensive log will be maintained that will keep track of all databases searches, key words used, number of hits, and time spent searching. Full citations on all relevant or potentially relevant studies will be maintained in the review database.
C. Methods of review
As we identify studies that potentially are relevant to the review, we will systematically gather details on the study (progressing from title review, to abstract review, to full study review) until we are able to determine with certainty whether the study is or is not appropriate for inclusion.
The initial step is to review titles and abstracts to rule out obviously inappropriate studies. The second step is to collect the full reports for all potentially eligible studies and to assess the appropriateness of the study for inclusion in the review based on reading the full report. Two reviewers (identified by their reviewer code) will do all screening independently. Differences will be resolved by negotiation, including a third party if necessary.
In some instances, the review team will find it necessary to contact the study authors to gather additional information. Such contacts and the supplemental information gathered for the review will be documented.
D. Selection of trials for analysis and reporting on program impacts
The review will include bibliographical information on all potentially relevant studies identified through the screening process. However, only the subset of studies meeting a base set of criteria required to generate credible evidence on program impacts will be included in the descriptive and statistical analyses and reporting of study findings.
As we review each study, we will code the basic information needed to determine whether the study meets the overall inclusion criteria. For those studies that meet inclusion criteria, we also will code the information required to determine quality of study implementation and the information required for the analysis of program impacts for the total sample and for key subgroups/ program types. Ultimately, we will include in the formal analysis estimating program impacts only those RCT studies that meet the criteria noted above.
E. Criteria for determination of independent findings
We will analyze results for the various outcome measures— sexual initiation, pregnancy risk, and pregnancy— separately. We will never pool effect size estimates from overlapping, dependent samples. Multiple independent effect sizes within a study will be included in the following circumstances: (1) if a study reports effect sizes separately by gender; (2) if separate outcomes are available for different cohorts within a study; and/or (3) if separate effect sizes are available across different study sites within a study.
In cases where there are multiple points of follow-up for a given outcome measure, we will focus on the longest follow-up period for which there is adequate retention of the original study sample (at least 60 percent of the initial study sample). If there are an adequate number of studies with longitudinal follow-up, we will examine changes in effect sizes over time. That is, we will estimate and test the significance of systematic changes in effect sizes systematically over time.
In cases where we encounter studies with multiple outcomes for dependent or overlapping samples (e.g., multiple treatments compared against one control group), we will code all of the effect sizes but only include one treatment/control comparison in the meta-analysis. We will randomly select which treatment to include in instances where both treatments are within the subgroup to be pooled.
Pilot programs and their replications will be treated as independent study samples. Replication status will be coded and potentially used in the analysis as a descriptive moderator.
F. Coding and data management
All studies meeting the initial criteria will be coded using an instrument that contains four main sections (see Attachment A). Section A includes all relevant bibliographical information. Section B collects basic information concerning both the study design and intervention criteria. Specifically, this section includes questions about the research design employed, whether there was adequate follow-up of the sample, whether the program served the age group of interest, the geographic location of the intervention, the primary goal of the intervention, and the types of outcomes measured. If all of the criteria in Section B are met, then Sections C and D are coded. Section C contains information concerning the nature and implementation of the intervention, and Section D contains additional information concerning study design, sample characteristics, and outcome data needed to calculate effect sizes.
1. Microsoft Access database
We will use Microsoft Access to code and manage our database. Using such a database system permits us flexibility to add, delete, or modify fields in response to our early coding experience. Moreover, coding forms created using Access are nonproprietary and, thus, easily shared among reviewers.
2. Data quality and coder reliability
Two individuals will code the first five studies. They will resolve differences in coding decisions and clarify coding policies. Following this “training” exercise, the primary researcher will code all of the studies, and a second coder will code a random sample of 20 percent of the studies. If there are more than 10 percent discrepancy in critical fields between the two coders in the random subsample, the remaining 80 percent of the studies will be coded by a second coder and all differences in coding resolved. This two-stage decision is based on our prior experience getting high levels of consistency among coders who have trained together and are trained in systematic review methods.
G. Statistical Procedures and Conventions
1. Software
Descriptive analyses will be conducted in SAS. If statistical meta-analysis is appropriate, analyses will be conducted using Comprehensive Meta-Analysis and SAS software. Prior to applying the Comprehensive Meta-Analysis software, it is necessary that all outcomes are measured as odds ratios, standardized effect sizes, or numbers of cases and sample sizes. The data we are able to extract directly from the evaluation reports are going to vary in their form and in their ability to be transformed into any of these standard formats. Therefore, we will make necessary transformations and formatting changes to the data using SAS subroutines provided by David Wilson (Wilson, 2001).
2. Units for measuring outcomes.
All three outcome measures are dichotomous and, thus, will be reported as percentages and percentage point differences. For studies where insufficient information is available to calculate an effect size, but study criteria are otherwise met, we will code the direction of the effect found for descriptive purposes and to help support and/or refute findings from the meta-analysis.
3. Metrics to remove bias
We will not use any metrics to remove bias. In cases where study author report regression adjusted impact estimates, these will form the basis for our impact estimates.
4. Missing data
We will not impute for missing values of the outcomes. However, we will consider some imputation when we have missing information on sample sizes, but have sufficient information to estimate cell sizes reasonably well. For example, a study may provide baseline sample sizes for program and control groups, sample attrition rates, and follow-up odds ratios but no information on follow-up program and control sample sizes. In such a case, we would use the information available to create estimates of the total number of program and control group members in the follow-up sample and the number who have experienced the outcome event (e.g., pregnancy).
5. Subgroup Analysis
If there are instances where we have adequate numbers of studies reflecting different program qualities and implementation conditions, we will conduct a moderator analysis to explore whether it is reasonable to combine the studies of various sample and program subgroups. The analysis will be conducted employing random effects assumptions. The findings of such a moderator analysis will be descriptive of the differences in measured impact across the subgroups represented in the studies. Differences between subgroups cannot be interpreted as evidence of causal relationships between the subgroup identifiers and the strength of program impacts. Rather, they may offer support for hypotheses regarding differential program effectiveness that could be tested through subsequent studies.
Five types of measures will be coded and potentially examined in the statistical analysis: (1) the comparison treatment environment (e.g., control group receiving no treatment versus receiving “usual” services); (2) the intervention components (e.g., sex education only; multi-component programs, without clinic services, clinic based or linked program); (3) qualities of the intervention (e.g., duration of intervention; whether the intervention was theory-driven); (4) the target populations for the intervention (e.g., high risk groups versus lower risk groups; middle school age versus high school age); (5) method variation (e.g., unit of randomization, attrition). Each of these subgroup identifiers has been identified in theoretical models as potentially important in determining program impacts.
We likely will not have more than 50 studies in our meta-analysis and, therefore, will be limited in terms of the number of subgroups we can examine at any one time. However, by gathering the information on all of these subgroups, where it exists, we will lay the foundation for future such analyses if the number of studies increases adequately.
6. Sample Weighting and Sensitivity Analysis
The analysis will be conducted using inverse variance weights. Where appropriate, we will use Hedge's g to correct for small sample sizes.
Sensitivity analysis will be conducted to determine whether effect sizes vary due to the following reasons: (1) pooling varying definitions for a similar outcome (for example, “contraception use” is measured in a variety of ways across studies), (2) comparing results by study sample attrition levels, (3) assuming different variance estimates when there is not enough information to adjust for clustering in clustered randomized trials. If necessary, additional sensitivity analyses will be conducted if other methodological issues arise that may impede our confidence in the estimated pooled effect size estimates.
7. Assessing heterogeneity
It is anticipated that there will be considerable diversity among the types of and settings for the programs represented by the evidence base. Therefore, random effects models likely will be most appropriate. Homogeneity tests will be conducted to confirm whether to use a fixed or random effects analysis. Results from such tests will be reported in the final review. If homogeneity estimates are statistically significant and it is not possible to explain variance in the pooled effect size using a few basic study characteristics, then pooled estimates will be based on random effects models.
H. Treatment of Qualitative Data
We will code key qualitative information about the program interventions, intervention setting, and study sample. This information will be used for descriptive purposes.
5. TIMEFRAME
April 2002 — July 2004: searching for published and unpublished studies April 2002 — Pilot testing of inclusion criteria and study coding May 2002 — Relevance assessments May - June 2002— Study coding and data collection (of all Kirby studies that meet review criteria) June - July 2003 – preliminary analysis using Kirby studies only (comparing findings to Kirby's narrative review) July 2003 – August 2004 — continued study coding August - September 2004— Statistical Analysis September 2004 – August 2005 — Preparation of report
6. PLANS FOR UPDATING THE REVIEW
This review, once completed, will be updated every two years to include additional study results. The primary author will take the lead in updating this review.
7. ACKNOWLEDGEMENTS
This protocol was completed with the help of many colleagues including: Doug Kirby, Jenny Ehrle, Jane Reardon-Anderson, and members of the University of Pennsylvania's Spring 2002 Education 900 class on conducting systematic reviews, which included the following participants Heather Cohen, Connie Keefe, Jeff Kralik, Andrew Sparks, John Weathers and Susan Zief.
8. STATEMENT CONCERNING CONFLICT OF INTEREST
Neither Scher nor Stagner has any conflict of interest related to this review. Maynard conducts experimental design studies in this area. However, none of her completed studies addresses questions that would qualify them for consideration for this review.
Footnotes
1
Throughout this protocol, we use the terms teenage pregnancy and childbearing to refer to unplanned or mistimed teenage pregnancies and births. In the past two decades, it has been relatively uncommon for teens in the United States and most similarly developed countries to intentionally become pregnant and give birth (
).
2
Notably, some of the intervention programs included in the review were not designed with the specific goal of reducing teen sexual risk-taking behavior. Rather, they are programs with other primary goals that also happen to report estimated impacts for sexual risk-taking outcomes.
3
A meta-analysis by examined differences in impact estimates generated from pregnancy prevention program evaluations based on different methodologies. This study concluded that there were meaningful differences in the estimates of program effectiveness associated with study design. This fact influenced the decision by DiCenso et al. to restrict their analysis to RCTs. Clustered randomized trials were also included and, when the authors of the original studies failed to adjust for clustering, DiCenso et al. made good faith efforts to adjust the findings for clustering.
4
Depending on the richness of the excluded study set, future reviewers may be interested in conducting methodological research to understand the relationship between study designs and other quality features and estimates of program impacts.
5
Since the primary interest for this systematic review is in preventing teen pregnancies, sexual activity will be defined as heterosexual intercourse. Future reviewers who are interested in HIV prevention may choose to define sexual activity more broadly.
6
This analysis and that in the next question depend on there being multiple evaluations of the various types of programs.
7
The most compelling research on this issue is reported in a systematic review of methodological studies that have attempted to replicate the results of experiments through various statistical matching and econometric analysis techniques (Glazerman et al. 2002). Studies, such as Fraker and Maynard (1987), Wilde and Hollister (2002) and Agodini and Dynarski (2001), are the types of studies included in this research synthesis. Furthermore, impact estimates of teen pregnancy prevention programs suggest that there are strong method biases favoring the weaker study designs ().
8
Programs may serve teen parents. However, the main focus of the program should be on delaying sexual activity and reducing pregnancy risks among nonparenting teens. Studies of programs that did not include pregnancy prevention as a primary goal, but that subsequently analyzed sexual behavior and pregnancy outcomes will not be included in the review.
9
This restriction is to omit programs that are aimed primarily at older adolescents such as college students or post high-school youth, as the needs and services for these groups differ considerably from the population of youth at large. Programs will be included in the review if their intention is to serve school-age adolescents, even if a minority of the sample members served are not within the 11 through 18 age range.
10
Other outcomes of interest include births, abortions, sexual knowledge, and sexual attitudes/beliefs. However studies including one or more of these secondary outcomes will only be included in the review if the study also includes at least one of the three primary outcomes.
ATTACHMENT A TEEN PREGNANCY PREVENTION RESEARCH SYNTHESIS DATA CODING INSTRUMENT
