PROTOCOL: School-based Interventions to Reduce Dating and Sexual Violence: A Systematic Review

Types of research designs

Only studies that have a well-defined control group will be included in the review. These may include wait-list controls, treatment-as-usual, and straw-man designs. Studies that compare a dating violence program to another dating violence program (i.e., treatment vs. treatment designs) will not be included. We will include studies where participants are assigned at the individual, group, school, district, or state level. Specifically, we plan to include the following designs:

1.

Randomized controlled trials: Studies in which individual participants, classrooms, or schools were randomly assigned to control and treatment conditions.

We will also note that randomized experiments have been considered paramount to quasi-experimental designs due to a theoretical lack of selection bias. A burgeoning literature insists, however, that quasi-experimental designs, given appropriate pre-test observations, have the ability to produce similarly efficient and unbiased estimates of a treatment effect (Shadish & Cook, 2009). As it is often infeasible to randomly assign students to conditions, we have chosen not to exclude studies based on lack of random assignment. As a result, we believe that including such studies will increase the total study sample size and provide a more complete picture of the literature. To ensure that theoretically high risk of selection bias does not bias the average effect size, we will conduct sensitivity analyses that remove potentially biased studies and re-calculate the average effect size.

Types of participants

The population of interest is 4^th-12^th grade students. Studies that include participants outside this range must provide summary statistics for the age groups of interest. For instance, a study might implement a program for students in 11^th and 12^th grade as well as freshman and sophomore college students. A study that provides summary statistics for the reviewed population, in this case 11^th and 12^th graders, will be included. However, studies that merely provide study-level statistics that fail to disaggregate information will be excluded.

Types of interventions

The intervention of interest is a school-based prevention program designed to reduce teen dating violence. The intervention may also seek to change other outcomes (i.e., bullying perpetration, sexual harassment, etc.); however, a clear goal, as provided by the authors, must state that the program sought to explicitly reduce teen dating violence behaviors, change attitudes supportive of teen dating violence, increase bystander intervention to reduce perpetration, or increase peer support for victims of dating violence. Studies that merely measure these as secondary outcomes will be excluded.

Types of outcomes

The primary outcomes of interest are:

1.

Attitudes supportive of teen dating violence behaviors

Outcome information may be reported via self-report questionnaires, teacher reports and observations, or researcher reports and observations. Examples of measures may include modified versions of the Conflicts Tactics Scale (Straus, Hamby, Boney-McCoy, & Sugarman, 1996), which assesses the frequency of use and type of behaviors used when dealing with conflict; dating violence knowledge, which can include “true” or “false” questions on definitions of abuse, resources for help, etc. (Taylor, Stein, Mumford, & Woods, 2013); and measures of victimization, which ask how many times incidents of sexual and nonsexual violence have occurred while on a date (Foshee et al., 2000). The outcome's metric is often determined by the primary author and is therefore difficult to hypothesize prior to coding.

Types of time points

We plan to synthesize outcomes measured directly after intervention (i.e., post-test) and at follow-up. The first post-program measures will be considered outcomes that follow the intervention, and any subsequent measures will be considered as follow-up effects. In essence, two simultaneous syntheses will be presented. This technique allows for greater understanding of the programs' effects both short- and long-term while guarding against non-independence of effect sizes. Therefore, time will not be used as a moderator (as is sometimes the case). Instead program effects will be calculated as outcomes immediately following the completion of a program, and also as outcomes sustained for a period after a student has ended a program.

Types of settings

The review will include studies conducted within middle and high schools. Studies that utilize community centers or other locations outside the brick-and-mortar schools will be excluded. We will include, however, all types of schools (i.e., publicly or privately funded). After-school programs conducted at the school will be included.

Example of included study

Weisz and Black (2001) implemented a school-based intervention in an urban middle school to change students' knowledge about and attitudes toward dating violence. The authors recruited seventh-grade students by offering a $5 McDonald's gift card (n = 44). A control group was formed by randomly selecting students from the same school who chose not to participate in the program (n = 20). This study constituted a non-randomized design.

The authors assessed pre-test equality using an independent t-test; the results indicated no significant differences at pre-test. Results at post-test indicted significant increases in knowledge about and attitudes toward sexual violence for students who received the intervention (gs = .20 & .17, respectively).

Electronic searches

Relevant studies will be identified using electronic database searches, government policy databanks, and Internet search engines. We will place neither language nor date restriction on the search. We plan to search from 1960 – February 2013.

The following “traditional” databases will be searched:

1.

ERIC

The traditional databases include international publications. However, we will also search CBCA Education, the British Education Index, and the Australian Education Index for citations outside the US.

The following “grey literature” databases will also be searched:

1.

scientific.thomson.comdatabases

Search terms

Search terms will be created using relevant key words that represent the studies of interest. Table 1 lists each of the relevant search terms. These terms will then be arranged to produce a Boolean search phrase for each combination. An example of the search phrase that will be used in ERIC is presented in Appendix A.

Searching other resources

Two further procedures will be conducted to ensure search breadth. First, the authors will scan and screen bibliographies of included studies. Hammerstrøm, Wade, and Jørgensen (2010) showed that this procedure has the ability to produce substantial additional studies even with the most robust searches of electronic databases. Second, the authors will contact high-profile researchers in this field. These contacts will ensure that relevant published studies are included and may engender the inclusion of “file-drawer” datasets yet unpublished (Rosenthal, 1991). We will identify high-profile researchers by identifying who has published extensively within the field of teen dating violence, and identifying who has received funding to explore teen dating violence prevention programs. Finally, we will search websites of foundations and organizations that aim to decrease the prevalence of sexual and dating violence among youth for studies that may not have been included from previous search efforts. Examples of foundations and organizations include Centers for Disease Control and Prevention, National Institute of Justice, Love is Respect, Robert Wood Johnson Foundation, and the American Association of University Women.

To identify potential grey literature outside of indexed databases, we will also search Google, Google Scholar, and Bing search engines. This will locate conference abstracts, government documents, and other online material. We plan to include any studies that meet the inclusion criteria listed above regardless of source.

Targeting specific journals

One member of the review team will hand-search the Journal of Counselling Psychology and Prevention Science starting in 2000 to locate any additional studies or references. These journals were selected because they had high initial citation counts relative to the purpose of this review.

Screening procedures

The screening process will occur in two distinct phases. First, LD and JRP will independently screen each title and abstract obtained from the search procedures described above for inclusion (see Appendix B). Each reviewer will code each citation according to pre-determined inclusion criteria. This information will be stored in an EXCEL database. Disagreements will be handled by discussion and consensus agreement with a third reviewer (DLE). The decisions available to the reviewer are: 1) Yes, include for full article scan, 2) Unclear so include for full scan, 3) Unclear but do not include (include reason), and 4) No, this article should be eliminated (include reason). Citations that meet the inclusion criteria will be retrieved for full review using the University of Illinois and Loyola University Chicago library resources.

Second, each of the first and second authors will also independently screen the full articles for inclusion (see Appendix C). As with the previous procedure, we will screen the studies using the inclusion criteria, keeping track of that information in an EXCEL database. Should the citation be excluded at this stage, the reviewer will provide a brief description of the reason for dismissal. Again, a discussion and agreement will be reached for each disagreement using a third reviewer.

Extraction of study information

At least two reviewers (LD, JRP) will independently code all of the included studies. The codebook (Appendix D) details the variety of study characteristics that will be coded. The codebook will be operationalized in a Microsoft Access database. Electronic coding is preferable to hand-coding because it reduces data entry errors (Cooper, 2010). Microsoft Access will be utilized because of the hierarchical nature of data extraction. For instance, a study may include multiple outcomes nested within multiple treatment groups. Access allows the reviewers to assign this multiplicity of information to one study, in turn limiting the amount of redundant coding.

All studies will be double-coded by an independent member of the review team (DLE or TDP) and inter-rater reliability (i.e., percentage match) will be calculated. Disagreements will be discussed and a consensus code used.

Assessment of risk of bias in included studies

The review team will assess the methodological quality of studies using the risk of bias tool developed by the Cochrane Methods group (Higgins, Altman, & Sterne, 2011). The risk of bias tool assesses study quality on nine indicators. The review team will not exclude studies based on the risk of bias assessment because this procedure has been shown to substantially bias meta-analytic results (Juni, Witschi, Bloch, & Egger, 1999). Rather, we will use the results of the risk of bias assessment as a categorical moderator (see moderator section below). In addition to the risk of bias assessment, we will also use attrition information as additional sources of information about study quality.

It should be noted that, because the risk of bias was developed for medical interventions, certain aspects of the tool may not be reported by the primary authors. It is uncommon for social science researchers to employ allocation concealment techniques and all but impossible to blind participants to psycho-educational interventions. Given that all studies are missing one of the nine indicators, however unlikely, we will not count that indicator toward a studies' overall level of bias. Instead we will use pre-test information to evaluate studies for risk of selection bias. We will calculate pre-test effect sizes to test for equivalence of all outcomes reported, regardless of whether the outcome will be included in the review. Should more than half of the reported outcomes differ, the study will be labelled “high risk” and sensitivity analyses will be conducted by removing such studies. In addition, we will note if treatment and control groups differ on demographics variables or on any outcome measures of interest. Again, should demographic differences exist, we will conduct sensitivity analyses where these studies are removed from the synthesis.

Effect size calculation

Effect sizes will be extracted from each study using relevant summary statistics. Traditional effect size calculation procedures will follow those laid out by Lipsey and Wilson (2001) and Hedges and Olkin (1985). David Wilson's online effect size calculator (2013) as well as the software program Comprehensive Meta-Analysis (Borenstein, Hedges, Higgins, & Rothstein, 2005) will be utilized to calculate each effect size.

Effect size synthesis

Prior to synthesis, we will determine the appropriate effect size metric. Should the majority of studies provide a continuous measure of the effect, we will convert all discrete effect sizes to the continuous metric first. We hypothesize this will be the case. However, should most studies utilize discrete data, we will convert all effect sizes to discrete. The conversion procedures, as mentioned previously, will be performed via Wilson's (2013) online effect size calculator.

The review will utilize a random-effects model for effect size synthesis. This choice is preferable given that the effect sizes derive from a theoretical population of effect sizes (Borenstein, Hedges, Higgins, & Rothstein, 2010). Thus, the random-effects model accounts for the distributional effects by including a between-groups variance component in addition to the within-group variance. The effect size and confidence interval therefore reflect a more conservative estimate relative to fixed-effect models. It should be mentioned that the fixed-effect estimate will not be reported.

The synthesis calculation is represented below:

where w_i represents the random-effect weight from study i and ES_i is the effect size from study i.

A z-test and confidence interval is calculated by estimating the standard error of the average weighted effect size. This calculation is represented below:

where w_i again represents the random-effects weight from study i.

Assessing heterogeneity

Following the average weighted effect size calculation, the authors' next procedure will be to assess for heterogeneity of effect sizes. Traditionally, heterogeneity is assessed using the Q statistic; this statistic is calculated using the following formula:

where W_i is the random-effects weight for study i, ES_i is the effect size for study i, and ES represents the overall mean effect size. The calculated Q statistic is distributed as a chi-square with k – 1 degrees of freedom and k is the number of effect sizes (Hedges & Olkin, 1985). We also plan to estimate and report values for τ ² and I ² to assess for heterogeneity in the effect sizes.

Moderator analyses

Given a statistically significant Q statistic, the authors will conduct moderator analyses to assess variation in treatment effects. Lipsey (2009) suggested utilizing three types of moderators: extrinsic, methodological, and substantive. Extrinsic variables are represented by the study's unchangeable characteristics, for instance date of publication, or published and not published. Methods variables can be represented by random vs. non-random assignment. Length of intervention or intervention location constitutes substantive moderators.

We plan to investigate the impact of 13 moderator variables. Following the suggestion of Lipsey (2009), they have been subdivided into 3 broad categories:

1)

Extrinsic a)

Date of Publication: Trikalinos and Ioannidis (2005) showed that treatment effects may diminish over time as more sophisticated methodological approaches and designs are utilized. It is therefore important to test this assumption.

Borenstein, Hedges, Higgins, and Rothstein (2010) explicated several options for assessing conditional effect size differences. A distinction must first be made between categorical and continuous moderators. Categorical moderators will be assessed using ANOVA-like procedures that compare between- and within-level variance. Continuous moderators will be empirically assessed using meta-regression.

Publication bias

Publication bias will be controlled for and assessed using various methodological and statistical techniques. First, the authors will take pains to locate and include studies not published via traditional methods. Sutton (2005) showed bias consequences associated with the exclusion of un-published studies; overall, studies that fail to find positive intervention effects tend to remain un-published. As a result, the meta-analytic estimate of the average weighted effect size is upwardly biased. Including un-published studies prevents this bias.

Second, the authors will evaluate empirically the presence of publication bias using funnel plot asymmetry in addition to Duval and Tweedie's (2000) Trim and Fill procedure. Funnel plot asymmetry is a simple graphical technique that plots the study's standard error against its effect size. Asymmetry detects potential bias of the outcomes. The trim and fill procedure's is also to locate potentially missing effect sizes, usually ones due to a non-positive treatment effect. The technique estimates an imputed effect size and recalculates the average weighted effect size. Although this procedure is not robust to criticism, it can be useful for sensitivity analysis purposes.

Handling of missing data

Missing data is bound to occur in the meta-analytic context. Pigott (2009) suggested multiple methods for handling the inevitable situations. One practical solution often requires the meta-analyst to estimate an effect size given less than ideal summary statistics. These situations offer require little more than the correct software packages. We will use this technique when summary statistics are not available to calculate an effect size.

Another pragmatic solution will be to contact the primary author for further information. This will be especially useful for studies conducted and published recently. We will use this option when an effect size cannot be calculated or when moderator information is missing.

Sensitivity Analyses

We will observe the distribution of effect sizes to check for potential outliers. Should an effect size rest more than 3 SDs from the mean, we will remove the effect size and recalculate the mean effect size.

We also plan to estimate the correlations among the moderators. Should substantial covariation exist (i.e., r > .6), we will conduct additional sensitivity analyses to address potential interactions among the moderators.

Finally, any quasi-experiments that fail to use one of the pre-determined acceptable methods will be removed from the total sample and the average effect size will be re-calculated.

Software

Wilson's (2013) online effect size calculator will be utilized to calculate effect sizes. The stand-alone software Comprehensive Meta-Analysis (CMA; Borenstein, Hedges, Higgins, & Rothstein, 2005) will be used to calculate all meta-analytic estimates. CMA will also be used to produce data tables and graphs.