PROTOCOL: Effectiveness of Early Family/Parent Training Programs in Preventing Antisocial Behavior & Delinquency

3.1. Criteria for inclusion and exclusion of studies in the review

Following the earlier Bernazzani and Tremblay (2001:92) review and the more general Campbell reviews, the scope of this review is experimental and quasi-experimental studies including pre-post evaluations of family programs. Studies lacking control groups were excluded as they cannot help differentiate intervention effects from other effects including developmental effects. The preliminary eligibility criteria are as follows:

Types of Studies: Only studies employing random assignment or quasi-experimental designs were included. Studies lacking control groups were excluded. ¹⁰

3.2. Search strategy for identification of relevant studies

Several strategies were used to perform an exhaustive search for literature fitting the eligibility criteria. First, a key word search was performed on an array of online abstract databases (see lists of keywords and databases below). Second, reviewed the bibliographies of four past reviews of EFPT programs (Mrazek and Brown, 1999; Tremblay, LeMarquand, and Vitaro, 1999; Bernazzani et al., 2001; Farrington and Welsh, 2007). Third, we performed forward searches for works that have cited seminal studies in this area. ¹³ Fourth, we performed hand searches of leading journals in the field. ¹⁴ Fifth, we searched the publications of several research and professional agencies (see list below). Sixth, after finishing the above searches and reviewing the studies as described later, we e-mailed the list to leading scholars knowledgeable in the specific area. These scholars were defined as those who authored two or more studies that appear on our inclusion list. These experts referred us to studies we may have missed, particularly unpublished pieces such as dissertations. Finally, we consulted with an information specialist at the outset of our review and at points along the way in order to ensure that we have used appropriate search strategies.

The following databases were searched:

Criminal Justice Periodical Index

The publications of the following groups were searched:

Washington State Institute for Public Policy

The following agencies’ publications were searched and the agencies will be contacted if necessary:

Home Office (United Kingdom)

The following keywords were used to search the databases listed above:

“Parent Training” and “childhood” or “pre-school” and “delinquency” or “conduct disorder” or “antisocial behavior” or “aggression” or “physical aggression” or “behavior problems”.

Several strategies were used to obtain full-text versions of the studies found through searches of the various abstract databases listed above. First, attempted to obtain full-text versions from the electronic journals available through the John Jay/CUNY library research port as well as those from the University of Maryland and the University of Louisville. When electronic versions were not available, we used print versions of journals available at the library. If the journals were not available at the university libraries, we used of the Interlibrary Loan Office (ILL) to try to obtain the journal from the libraries of other area schools. When these methods did not work, we contacted the author(s) of the article and/or the agency that funded the research to try to obtain a copy of the full-text version of the study.

3.3. Description of methods used in the component studies

The studies included in this review will use methodologies that are variations of a treatment versus comparison group research designs with a post-test measure and possible follow-up measurement points. In all cases, the participant samples will be families and children, a sample of who will participate in the program and a sample who will not participate in the program. The studies varied with respect to the method of constructing the comparison group. The studies also varied with respect to the degree to which they employed statistical controls to reduce the threat of selection bias.

All studies included in the review included a post-program measure of antisocial behavior, delinquency, or crime. These included dichotomous indicators or more differentiated indicators that indicated the specific types of offenses or the frequency of offenses committed. A few studies reported on other outcomes, such as school performance. We did not code these other outcomes, but documented all reported outcomes.

3.4. Criteria for determination of independent findings

Some studies reported multiple findings on different outcomes and/or different samples. In the case of independent samples, the results will be treated as separate findings and all such results will be included in the analysis. Other studies reported on outcomes for the same sample. For cases such as this with multiple findings from the same sample, each was examined independently to decide how to either combine the findings or to choose the one that best represents the study (recognizing that we do not wish to select the measure with the largest or smallest effect). Our final decision here was to pool together the separate effects into a single summary estimate because (1) different studies measure different antisocial and delinquent behaviors in different ways with respect to item wording, combined scales/indices, and time period referenced, and (2) we were primarily interested in the overall effect of the EFPT program on antisocial behavior/delinquency in general. (Note: We do indicate that future evaluation studies parcel out and specifically focus on the effect of EFPT on unique antisocial/delinquent behaviors). It is the case that many interventions were designed to deal with a specific outcome, but some also targeted some secondary problems and reported outcomes for these as well. In these cases the effect size for the primary outcome was reported. To be sure, different outcome contrasts were analyzed separately, and thus the independence issue was addressed. The issue, of course, is one that deals with multiple measures of the same construct or the same measure at multiple time points. Yet, we selected the best measure of a construct and analyzed these separately. So, for example, most studies had the primary outcome measure of interest be mother-reports of child delinquency. If other outcomes were also examined in the reviewed study (i.e., teacher reports of child delinquency), we analyzed these separately. ¹⁵

3.5. Details of study coding categories

All eligible studies were coded (see coding protocol attached in Appendix A) on a variety of criteria (including details related to them) including:

Reference information (title, authors, publication etc.)

Dr. Jennings will independently code each eligible study and consult with Dr. Piquero when questions arise in order to determine the final coding decision. ¹⁶

3.6. Statistical procedures and conventions

Meta-analytic procedures were used to combine data from studies. For eligible studies, with enough data present, effect sizes were calculated using the standardized measures of effect sizes as suggested in the meta-analytic literature (Lipsey and Wilson, 2001). We also used the standardized mean difference effect size or odds-ratios, depending on the outcome construct. Mean effect sizes were computed across studies and we used a correction such as the inverse variance weight for computing the associated standard error. ¹⁷ Though we examined the Q statistic to assess heterogeneity of effect sizes across studies, it is our initial assumption that study random factor in our analysis, that is, that there are meaningful differences across the studies affecting the size of the effect. As such, we implemented a random effects model for all analyses involving effect sizes. This is the case because EFPT strategies are diverse and they are oftentimes designed to ameliorate different types of problems. In this context, we believed that a mixed effects model will be most appropriate in analyzing the effectiveness of EFPT programs.

We had originally hoped to examine contextual or moderating features of these programs. Though it was difficult to know at the outset, we thought it important to explore the differential effects of EFPT programs across different outcomes and across different types of treatments. Our goal was to assess this using the analog to the ANOVA method of moderator analysis (Lipsey and Wilson, 2001) for categorical moderator variables and meta-analytic regression analysis for continuous moderator variables or analyses involving multiple moderators. We also initially recognized that such analyses would be dependent on the number of studies that are available for inclusion in the meta-analysis. For reasons discussed above with respect to different outcomes, we did not continue further on this issue, but identified it as an important future research effort.

Methodological quality will also be examined. For example, we will report the results of experimental and quasi-experimental studies combined and separately. We also plan to conduct moderator analysis of other method variables, such as attrition, similarity at baseline, or other method variations that are important within the more general prevention literature.

Finally, publication bias is a concern in every meta-analysis. As such, we use traditional methods to test for the sensitivity of the findings to publication bias in the experimental and quasi-experimental studies. These methods include a comparison of the mean effect size for published and unpublished studies and a trim-and-fill analysis. We recognize that it may be difficult to rely upon traditional quantitative methods to deal with publication bias.

3.7. Treatment of qualitative research

Qualitative studies will not be included in the current review.

3.8. How study quality will be assessed?

Although we recognize that assessing study quality can be both an objective and a subjective exercise, we attempt to assess the quality of the studies in terms of research design, sample bias, and attrition bias, and make notation of study quality at various points in the review.