Protocol for a Systematic Review: Teach For America (TFA) for Improving Math,Language Arts,and Science Achievement of Primary and Secondary Students in the United States: A Systematic Review

Types of study designs

This review will include primary studies with research designs that, when implemented well, are capable of generating data that can be used to make generalizable causal inferences about the effects of TFA on student academic outcomes. Eligible designs that meet these criteria are randomized controlled trials (RCTs), regression discontinuity designs (RDDs), single-case designs (SCDs), and quasi-experimental designs (QEDs). However, we limit the eligible designs for this review to RCTs, where random assignment is used to form intervention and comparison groups, and QEDs, where non-random methods such as matching or other statistical methods are used to form a counterfactual group that is comparable to the intervention group on measured characteristics. RDDs and SCDs will be excluded from this review since statistical methods for incorporating RDD and SCD data into meta-analyses are, to the best of our knowledge, not well established. For example, the Campbell Collaboration Methods Policy Brief is silent on the statistical synthesis of the RDD and SCD. Furthermore, the nature of TFA interventions and the results of our cursory literature search indicate that RDDs and SCDs are rare. Research designs that lack a comparison group, such as single-group “pretest/posttest” designs, will be excluded from the review. It is well established in the methodological literature and in the practice of educational research that designs without a comparison group cannot rule out a competing explanation for observed differences between intervention and comparison groups on an outcome (Shadish, Cook, and Campbell, 2002).

Types of participants

We will include studies with participants who are K-12 students with TFA corps members, TFA alumni, and non-TFA teachers in rural and urban public schools in the United States. At the time of the intervention, the teachers in the treatment condition must be TFA corps members or TFA alumni; the control condition must include non-TFA teachers who have never participated in TFA. Non-TFA teachers may vary in their years of teaching experience and certification status. During the time frame of the study, all students must have a teacher who meets the eligibility criteria for TFA teachers or for non-TFA teachers. We will exclude studies that focus on Teach for America's Early Childhood Initiative, defined as initiatives that start prior to kindergarten, since early childhood studies are outside the policy relevant scope of this review.

Types of interventions

The TFA intervention condition will include TFA corps members, who are serving their two year commitment, TFA alumni who have completed the two year program but continue to teach, or both. The non-TFA comparison condition will include teachers who have never participated in TFA. These teachers must not have received preparation or training in programs associated with TFA. Teachers in the non-TFA comparison condition can vary in their certification status: traditional, alternative, emergency and uncertified. To be included in the review, the study must include a TFA condition (as described) and a non-TFA condition (as described). Studies that create an intervention group by bundling the TFA corps members or alumni with teachers trained in other alternative teacher preparation, such as the New York Teaching Fellows Program, will be excluded from the review. The reason is that when TFA is bundled in this way it is impossible to disentangle the effect of TFA from the effect of other alternatively prepared teachers in the intervention group.

Types of outcome measures for students

The review will include studies with at least one academic student outcome in math, English language arts, or science domains. Student outcomes in other non-academic (or non-cognitive) domains will be documented in the coding guide but will not be reported in the review. Multiple types of outcome measures will be included, although our experience with reviewing the TFA literature indicates that the primary types of outcome measures we will encounter will be state assessments, end-of-course assessments, and other standardized assessments.

State assessments, end-of-course assessments, and other standardized assessments are eligible for inclusion in the review provided that they were administered as intended. Non-standardized assessments, such as researcher-developed assessments, are eligible for inclusion however the study must provide evidence that the measure (1) has face validity and reliability, (2) is not over aligned with the intervention, and (3) administered in the same way for both intervention and comparison groups. The first criterion is that the measure has face validity and sufficient reliable. A description that shows that the measure is clearly defined and measures the construct it is supposed to measure can serve as evidence for face validity in this review. Reliability evidence may come in the form of internal consistency, test-retest reliability, or inter-rater reliability. The second criterion is that a study must provide evidence that the measure does not closely resemble aspects of the intervention. For example, the measure should not have items or materials that intervention teachers have access to through their TFA training materials but that comparison teachers do not. The third criterion is that eligible outcome measures must have been used the same way in the treatment and comparison conditions.

Type of outcome measures for teachers

The review will include studies that meet the outcome inclusion criteria for students and have at least on teacher outcome on teacher leadership which is a key mediator between TFA teacher training and student achievement in the TFA theory of change. Additional teacher outcomes that will be included in the review are content knowledge, years of teaching experience, or overall academic ability.

Teacher Leadership. There is no single definition, however. One that comes closest to aligning with the TFA framework is teachers who take on leadership roles and additional professional responsibilities such that leadership roles and decision making responsibilities extend beyond the school or district administrative team to the teacher. Reliable and valid measures designed to tap into this construct will be eligible for the review.

Content knowledge. To be eligible for the review, an outcome should measure a teacher having a solid background in a subject or content area as exhibited by a college minor or major in the subject or content area such as math, reading, or science.

Teaching experience. To be eligible for the review, an outcome should measure the total number of years of classroom teaching experience in the field.

Academic ability. To be eligible for the review, an outcome should tap into the construct of of academic skills as measured by SAT scores, ACT scores, grade point average, or selectivity of the college attended.

Validity criteria for student and teacher outcomes

When reviewing outcome measures according to the three criteria, we will apply the definitions in the WWC Procedures and Standards Handbook Version 3.0, page 16, section 4. For example, thresholds for the psychometric properties that determine the reliability of an outcome measure will be based on the WWC Evidence Standards that require (a) internal consistency of 0.50 or higher; (b) temporal stability/test-retest reliability of 0.40 or higher; or (c) inter-rater reliability of 0.50 or higher. Teacher outcomes such as years of teaching experience that do not have psychometric properties must show evidence of being collected consistently across groups in the study.

Duration of follow-up

One school year is the minimum dosage that study participants must have had in order for the study to be included in the review. All durations of follow-up above or equal to the minimum dosage will be included in the review. In the meta-analysis we will control for study to study differences in the follow-up period by meta-analyzing studies with the same follow-up periods (studies with one-year follow up outcome will be meta-analyzed on that outcome together, studies with two-year follow-up outcome will be meta-analyzed on that outcome together, and so on). Studies will be excluded if the minimum dosage is less than one school year or if the treatment and comparison groups do not have a comparable dosage.

Types of settings

The review will include studies that take place in K-12 public schools, including charter schools, in the United States. Limiting the setting to K-12 public schools helps ensure that the review will generate evidence that informs the TFA policy debate. Privately funded schools, early childhood education programs, higher education programs, adult education programs, and alternative schools, such as correctional programs, will not be included in the review.

Searching Electronic Databases

For the main electronic database search, we will use the ProQuest search engine available through the University of Pennsylvania's library web portal. This search engine provides access to and individual searching of 52 databases that index studies published formally and informally in a range of disciplines including education, economics, psychology, and sociology. Using this engine, we plan to search the following databases:

ERIC,

We also plan to search additional databases not covered by ProQuest. These are JSTOR, Academic Search Premier, and Education Next/Full Text.

When searching electronic databases, we will use search terms in consultation with our PENN library liaison at PENN GSE, using the following four domains:

Program names for Teach for America. (Searching only Title and Abstract).

Based on these domains, we will employ search terms that connect the domains with the Boolean “AND” operator. Within domains, we will use the Boolean “OR” operator in order to search multiple keywords. The search terms will be similar to the following:

(TI (“Teach for America”) OR AB (“Teach for America” OR “TFA Corps”))

AND

(“academic achievement” OR “success*” OR “grade level” OR “grading” OR “academic ability” OR “Attainment” OR “failure” OR “educational indicator*”)

AND

(“kindergarten” OR “elementary school*” OR “primary school*” OR “high school*” OR “public school”)

AND

(“random assignment” OR “randomized experiment” OR experiment* OR “experimental design” OR “control group” OR “non-experiment” OR “non-experimental” OR “quasi-experiment” OR “quasi-experimental” OR “comparison group” OR “matched comparison group” OR “matched comparison” OR “matched groups” OR “statistical matching” OR “propensity score matching” OR “systematic review” OR “review” OR “meta-analysis” OR “research synthesis” OR “research review”)

Publication Date = 1 Jan 1994 – 2015

Due to controlled vocabulary differences across databases, the main search term will change for each database. We will consult the database thesauruses in order to build the final search terms for each database.

We will search each database separately and tailor our strategy for each. For example, we will look up search words in each database thesaurus to see which descriptors are available, and make use of grade level and publication type filters. Results from our literature searches and other searches described next will be stored and managed using RefWorks online bibliographic software.

Searching Grey Literature

In addition to the main database searches, there will be a five-step grey literature search that involves 1) searching grey literature databases, 2) manually searching targeted websites, 3) searching conference presentation databases, 4) searching existing reviews, and 5) searching Google. The database searches will use PolicyFile, PsycExtra, and OpenGrey.eu. The manual grey literature searches will include general websites for organizations that conduct research across many areas of education (Table 1) as well as targeted websites for organizations that have a focus on teacher education or TFA research (Table 2). In order to make sure that relevant conference presentations are included in the meta-analysis, we will search the EditLib and Index of Conference Proceedings databases for conference abstracts using search criteria similar to what will be used for the main database search. We will also search existing reviews in order to refine the search strategy and check references for studies that should be included. Existing reviews will be identified through the main database searches and the grey literature searches as well as through searching the Campbell Library. >Lastly, we will do an advanced Google search where we use criteria similar to the main database search and screen the first twenty pages of results.

Soliciting Previous Authors of TFA Studies

Based on studies we retrieved from our cursory/background searches, we will develop an email list of all researchers who authored an effectiveness study. We will also develop an email template that briefly describes the C2 systematic review on TFA, provides a bibliography of all effectiveness studies identified from our literature search whether or not these studies are eligible for our review, and requests that study authors refer us to 1) any effectiveness studies not in the bibliography or 2) any authors not in the bibliography who may be aware of TFA studies that have not been formally published, or 3) both.

Hand Searches of Journals

The limited resources and personnel prevent us from conducting a comprehensive hand search of social science journals where TFA may be published. Moreover, our search of bibliographic databases that indexed grey literature, grey literature databases, and our outreach to previous authors of TFA studies should identify many of the studies, if any, not identified in our search of electronic databases. That said, consistent with guidance from the C2 Information Retrieval Policy Brief, we will have master level graduate students on our review team manually scan the table of contents of the most recent two journals that, based on our bibliography of effectiveness studies, are most likely to publish effectiveness studies on TFA. We will conduct this hand search in December 2015 towards the end of the review to compensate for any lag before articles are indexed by the databases.

Goal of the Meta-Analysis

The primary goal of the meta-analysis is to address review questions 3 through 10 by estimating the average effect of TFA on student academic outcomes, quantifying its precision (95% confidence intervals), evaluating whether the effect is real or due to chance using the actual p-value compared to the alpha level. Questions 10 is descriptive, and will report–for studies included in the meta-analysis–what authors reported on TFA's cost effectiveness.

Data Extraction Using Excel Based Coding Guides

Two reviewers will extract data from the articles independently and code methods, participant characteristics, intervention characteristics, and outcomes into the coding guide in Appendix A. Issues will be resolved in conference. If still unresolved, then they will be resolved in consultation with PI. If further information (e.g., missing data) is required regarding study data in order to conduct appropriate analyses of outcomes or to establish baseline equivalence, the first author of the study will be contacted. Should requested data be unavailable, the study will still be reported, but not included in the final meta-analysis. Information extracted from the studies eligible for review will be displayed in an appendix table of the review that report study characteristics for studies included and excluded in the statistical synthesis.

Meta-Analysis

Individual study effects will be synthesized statistically using Comprehensive Meta-Analysis software version 3.0 (CMA 3.0). The software allows for over 100 different data entry formats for effect size calculations. The choice of which effect size computation to use depends on three key factors: (1) the measures of the outcome variable(s), (2) the designs of studies being reviewed, and (3) the statistical analyses that have been reported. Depending on the study results, we will compute effect sizes from one of two types described next.

Standardized Mean Difference. TFA effectiveness studies reporting continuous outcomes will be summarized using standardized mean differences where different measures have been used to measure the same outcome. When the outcome has been measured using the same measure, a weighted mean difference will be used.

Odds Ratio. Although unlikely, TFA effectiveness studies may report binary data in which mean outcomes are compared in the experimental and control (or comparison) groups. If so, we will use CMA 3.0 to convert odds ratios, risk difference, or risk ratios to standardized mean differences with 95% confidence intervals for inclusion in the Meta-Analysis.

Combining Effect Sizes. Using CMA 3.0, we will convert all effect size indices to Hedge's g which is a standardized mean difference with a small sample size bias correction factor, it is unbiased for both small and large samples. When substantively feasible, effect sizes will be averaged across studies by using an inverse variance weighting of the individual effect sizes to account for differences in sample sizes for individual studies. This weighting will result in the individual effect sizes of larger n studies being given more weight in the combined effect size. We will calculate this effect size using a random effects model, which assumes that the effect size in each study is a sample estimate from a different population, and it estimates the parameter for that population. The estimates differ from study to study owing differences among the study population parameters (between-studies variation) and to sampling of different subjects within the study populations (within-study variation). More importantly, from a policy perspective, results from the random effects model allow for inferences from the sample of studies to the population of studies from which the set was sampled. In this case, results of the data synthesis can be extrapolated beyond the studies in the set. However, we will also calculate effect sizes using fixed effects to provide a basis for comparison—this is easy to do with the CMA software. For both random effects and fixed effects models, individual study effect sizes and average effect sizes across studies will be reported with confidence intervals and corresponding p-values using a Forrest Plots.

Homogeneity Analysis

CMA 3.0 will allow us to quantify the amount of heterogeneity in the individual effect sizes that comprise the average effect size. This involves empirically distinguishing between variation in the individual effect sizes that is due to sampling error and variation in the individual effects that is due to true differences among studies. This can be done using two homogeneity statistics—I² and Q with the former preferred because it is not dependent on sample size and does not lead to inferential errors due to low statistical power.

Sensitivity Analysis

We will test the robustness of the conclusions drawn from the statistical synthesis through a sensitivity analysis on publication source and influential studies (or outliers with respect to effect size). To evaluate the possibility of publication bias, we will use funnel plots to assess the relationships between effect size and study precision. Such a relationship could be due to identifiable biases or to less obvious systematic differences between studies. To increase confidence in the robustness of the results of the data synthesis to overly influential studies (or outliers), we will conduct a “one-study removed” meta-analysis (provided there are at least three studies in the analysis) aimed at examining whether the results are sensitive to the inclusion or exclusion of particular studies. This analysis will be implemented in CMA 3.0 and estimates the overall effect size for all studies included in the meta-analysis with “one study” removed.

Publication Bias

Mean effect sizes of studies retrieved from peer reviewed sources will be compared to mean effect sizes of studies retrieved from unpublished sources (e.g., dissertations, government reports, and conference presentations). To evaluate whether the overall estimate of TFA's average effect is affected by publication bias, we will address three questions using methods recommended by Borenstein, Hedges, Higgins, Rothstein (2009, p. 277 – 291). -

Is there evidence of bias?

To address the first question, we will use a funnel plot to determine whether the studies are symmetrically distributed around the mean effect size within the funnel from top to bottom. To address the second question, we will use Rosenthal's Fail-safe N and Orwin's Fail-safe N with the second, an improvement on the first, by allowing us to 1) determine how many missing studies would bring the overall effect to a specified level other than zero and 2) specify the mean effect in the missing studies as some value other than zero. To address the third question, we will use Duval and Tweedle's Trim and Fill. This approach allows us to produce our best estimate of the unbiased effect size, although it is highly dependent on the assumptions of the model for why studies are missing–assumptions that we will state in the systematic review. These methodological approaches will be implemented with statistical validity using CMA 3.0.

Moderator Analysis for Categorical Study-Level Variables

The moderator analysis will be used to address research questions 7 – 9. We plan to code all moderators specified in these questions as categorical except for average teacher experience in the study–it will be coded as continuous. For categorical moderators, we are interested in “exploring” through a post-hoc analysis whether study effect sizes vary by the categories of the moderator. This analysis will be conducted for each moderator separately.

We assume that the studies within each moderator category do not share a common effect size. Under this assumption, we will use a random effects statistical model to weight the effect sizes in each moderator category and estimate the average effect size for each category. A separate estimate of Tau-squared (i.e., the true between study effect size variance) should always be estimated under random effects (Borenstein, Hedges, Higgins, and Rothstein, 2009). However, if there are fewer than five studies in a category, which is highly likely for this topic area, this estimate can be imprecise and we will use a pooled estimate. Using a pooled estimate makes more sense because the increased accuracy we get by pooling more studies is likely to exceed any real differences between groups (Borenstein, Hedges, Higgins, and Rothstein, 2009).

To evaluate whether average effect size differs by moderator category, when the moderator has only two categories, we will use the Z-test. When the moderator has more than two categories, we will use a Q-test based on analysis of variance. Although the multiple hypothesis tests across all the moderators could raise the “real” Type I error well above the “nominal” Type I error rate (or alpha level), we will not apply a multiple comparison correction to the alpha level because we are conceptualizing this post-hoc analysis as exploratory research to inform future research on TFA and not to determine whether TFA works or is effective. However, in analyses where such a correction is warranted, we will alert the reader to consider this correction who could use the results to inform their own statistical power analysis for a new RCT or QED on TFA or similar intervention.

Moderator Analysis for Continuous Study-Level Variables

For the one moderator (average years teaching experience) coded as continuous, we will use bivariate meta-regression with the effect size for an academic outcome as the dependent variable, and the moderator as a predictor. In this model, multi-collinearity among predictors is not a concern because there is only one. The meta-regression will be estimated using a random effects model (the model used in the main effects analysis) with a Z-test of the relationship between the two variables (e.g., effect size and predictor). To quantify the relationship between the two variables, we will report all of the meta-regression parameter estimates, standard errors, p-values, and variance explained index (e.g., R-squared) analogous to regression in primary studies.

Reporting Results and Risk Bias

Results from the main effect, subgroup, and moderator analyses will be reported according to the research questions these analyses are designed to address. Results will be reported using forest plots with study sample sizes, effect sizes, 95% confidence intervals, p-values, tests of homogeneity, and model choice of fixed or random effects. Moderator analyses will be reported with three important methodological caveats.

First, the analysis is post-hoc and should be interpreted as exploratory to inform future research.

Primary studies will be assessed for quality, risk of bias, and assigned a rating of meets design quality standards with or without reservations based on the criteria (Table B1 in Appendix B)

Missing Data and Author Queries

In research studies, authors sometimes do not report results from some of the analyses that were mentioned in the study. If any study mentions but does not report information that we need for our statistical synthesis, then we will send a letter to the study authors in an effort to obtain the relevant information that was mentioned. The authors will have two weeks to respond, but can request more time if needed.