Abstract
The aim of this Campbell systematic review is to assess the effectiveness of volunteer tutoring programs for improving the academic skills of student enrolled in Kindergarten through to 8th Grade (K-8) in the USA, and to further investigate for whom and under what conditions tutoring can be effective.
Only randomized field trials (in which a treatment was compared to a control group that did not receive the treatment) published from January 1985 to August 2005 were included in our review. The results of the review were based on the data from 1,676 study participants in 28 study cohorts in 21 research articles or reports.
The analysis of these studies – most of which included relatively small samples – showed that volunteer tutoring programs can positively influence language and reading outcomes for students. This review does not suggest that there are any particular volunteer tutoring models that should be recommended for immediate adoption for schools and districts across the country. Rather, we can conclude from the analysis that these programs can positively influence important reading and language sub-skills for young students.
The results are substantial – approximately one-third standard deviation. In the end, the results of this analysis should serve as one important piece of evidence used by policymakers and educators who are deciding whether to employ volunteer tutoring as a strategy to improve academic skills for young students.
Abstract
Background
Volunteer tutoring programs are intended to improve student performance, provide mentorship, and improve student self-esteem, as well as behavior. Despite the best of intention and effort, schools are not certain which volunteer tutoring programs are most effective. Therefore, we contend that a rigorous analysis of the extant literature regarding volunteer tutoring programs can provide schools with information about the most effective types of tutoring programs.
Objectives
To assess the effectiveness of volunteer tutoring programs for improving the academic skills of student enrolled in grades K-8 in the USA, and to further investigate for whom and under what conditions tutoring can be effective.
Search Strategy
A range of electronic social science databases were searched including Academic Search Premier; Primary Search; Professional Development Collection; Middle Search Plus; Psychology and Behavioral Sciences Collection; PsycINFO; Sociological Collection, ERIC (Education Resources Information Center), and Proquest Digital Dissertations.
Selection Criteria
Only randomized field trials (in which a treatment was compared to a control group that did not receive the treatment) published from January 1985 to August 2005 were included in our review. We only included studies that included academic impacts, although other impacts could be part of the study (e.g. behavioral, emotional). We also only included programs that were used for students in grades K – 8, and programs where adult, non-professional (volunteer) tutors were used.
Data Collection & Analysis
Details on the methods and procedures (i.e., design, analysis, outcome measures), the intervention (i.e., duration, setting), and the subject samples (i.e., age, gender, ethnicity) were coded for each study to allow for analysis of outcomes for different types of volunteer tutoring programs and for different study characteristics. After we identified all outcomes measured in the included studies, we selected the appropriate effect size metric for the meta-analysis. We used Hedges’ unbiased estimate g of the standardized mean difference effect size statistic (the difference between the treatment and control group means on an outcome variable divided by their pooled standard deviations) for each outcome measure. When means and standard deviations were not reported, we attempted to estimate the effect sizes using the procedures recommended by Wilson and Lipsey (2001). When studies had multiple measures of an outcome, we computed a pooled mean of effect size.
Main Results
The results of the review were based on the data from 1,676 study participants in 28 study cohorts in 21 research articles or reports. The analysis of these studies – most of which included relatively small samples – showed that volunteer tutoring programs can positively influence language and reading outcomes for students. We began by examining the overall effects of volunteer tutoring on student reading outcome measures. Twenty-five studies assessed reading measures of one type or another. The average effect of volunteer tutoring programs on reading outcomes for elementary students is 0.23. After removing one outlier study which disproportionately influenced by the overall result, we found an average effect size of 0.30. We also found several significant results in the meta-analyses of specific academic domains. The outcomes where volunteer tutoring programs made a significant difference were Reading Global (effect size = 0.26), Letters and Words (effect size = 0.41), Oral Fluency (effect size = 0.30), and Writing (effect size = 0.45). We found positive, but not significant, effects of volunteer tutoring on Reading – Comprehension and Mathematics.
When we analyzed the reading outcomes separately by study characteristics, we found no significant difference in effect size by tutor type, grade level, or program focus. Highly structured tutoring programs had a significantly greater effect on global reading outcomes than programs with low structure, but not on the other outcome types. The difference in effect sizes between studies published in journals and non-published studies was not statistically significant. Other tests of publication bias also suggested the included studies were an unbiased sample.
Reviewers’ Conclusions
This review does not suggest that there are any particular volunteer tutoring models that should be recommended for immediate adoption for schools and districts across the country. Rather, we can conclude from the analysis that these programs can positively influence important reading and language sub-skills for young students. The results are substantial – approximately one-third standard deviation. In the end, the results of this analysis should serve as one important piece of evidence used by policymakers and educators who are deciding whether to employ volunteer tutoring as a strategy to improve academic skills for young students. As educators across the country work to meet adequate yearly progress goals in state accountability systems, and as they seek affordable ways to offer additional services to students at risk of not meeting annual academic goals, it would be worthwhile to consider structured, reading-focused volunteer tutoring programs as strategies to improve reading and language skills.
Linked article:
Synopsis
Based on increasing demands for accountability in schools, school administrators are looking for ways to increase the academic performance of students including tutoring. Currently a variety of tutoring approaches are used including volunteer tutoring, peer tutoring, cross-age tutoring, and one-on-one tutoring. The evidence base related to volunteer tutoring has grown in recent years; consequently, this review is based upon only those evaluations of volunteer programs in which tutees were randomly assigned to a treatment or control group. We found 21 studies (with 27 different study cohorts in those studies because several studies provided separate reporting on multiple cohorts) reporting on randomized field trials to guide us in assessing the effectiveness of volunteer tutoring programs.
1. Background
In many cultures, the oldest form of teaching was provided to children by in-home tutors or private instructors (Shanahan, 1998). Tutoring remains a popular form of instruction worldwide, and the effectiveness of tutoring as a pedagogical method has been documented extensively in various strands of the educational literature (see, for example, Cohen, Kulik, & Kulik, 1982; Fashola, 2001; Wasik & Slavin, 1993).
During the 1970s, U.S. schools began relying more on peer tutoring (also known as student-to-student or cross-age tutoring) as a way to efficiently use scarce resources in a period of teacher shortages (Rekrut, 1994). The next two decades also witnessed an increased interest in tutoring programs staffed by adult volunteers for a variety of reasons: (1) increased public concern with the quality of education after the National Commission on Excellence in Education's release of “A Nation at Risk” in 1983; (2) rising interest in community service in the 1990s, and (3) the encouraging results from effective yet costly programs that employ professional tutors. By 1987, the National Research Council estimated that there were over one million volunteer tutors who donated an average of four hours per week in the nation's public schools. The survey found that three-fourths of public elementary schools in the United States reported the involvement of volunteers, with schools having an average of 24 volunteers (Michael, 1990).
More recently, the growth of these university partnerships has accelerated due to the America Reads Challenge, a nationwide tutoring initiative launched in 1997 by President Clinton. At that time, nearly 800 universities and colleges throughout the nation had already pledged to commit work-study slots for college students to serve as tutors for elementary school children (White House, 1997). By 1999, nearly 1,200 colleges and universities committed to placing work-study students as tutors in public schools. President Clinton's 1999 budget proposal also included $140 million to establish programs matching university-based mentors with students in schools that had very high dropout rates and high concentrations of poor students. As a result of the America Reads Challenge, state leaders have become increasingly interested in providing tutoring programs for elementary school children, and numerous local tutoring initiatives are now receiving increased support (U.S. Department of Education 1996; 1997).
Despite the increased interest in and support for tutoring programs over the past few decades, the expansion of programs that use non-professional, adult volunteer tutors has yet to be matched by a supporting research base. In the 1970s and early 1980s, most tutoring research in the United States focused on the impacts of peer or cross-age tutoring (Shanahan, 1998). In 1982, Cohen, Kulik, and Kulik published a well-known and often-cited review of this early body of tutoring research in the American Educational Research Journal. The authors noted that hundreds of reports on tutoring had been written by teachers and researchers, some based on scientifically-sound, experimental design evaluations, and others more informal and subjective. According to the authors, the four major reviews of the research up to 1982 used “relatively informal narrative and box score techniques” (p. 234), and each concluded that peer tutoring can help improve the academic success of young students.1
During the 1980s and 1990s, researchers began focusing more on specialized interventions aimed at improving the academic achievement of the lowest-achieving children, most notably Reading Recovery and Success for All (Shanahan, 1998; Wasik & Slavin, 1993). Both programs include one-on-one tutoring by professional tutors and are perceived as effective by many in the research community (Wasik, 1998). However, because the expense of employing professional tutors limits the number of children who can be served by these interventions, several programs have been created in recent years that use adult volunteers or para-professionals as tutors. Some of these programs are responses to the accountability measures established by the No Child Left Behind Act of 2001, which encourage education administrators to implement new programs aimed at increasing student performance. As administrators encourage the use of tutoring programs in their schools, they may be doing so without solid evidence of what types of impacts specific tutoring programs can realistically produce. To begin understanding the effects of tutoring programs, we examined the existing literature for reviews.
Previous work has identified four major reviews of research on the impact of volunteer tutoring programs on student outcomes: Topping and Hill (1995), Wasik (1998), Shanahan (1998), and Elbaum et al. (2000).
In 1995, Topping and Hill provide interesting background on the research related to volunteer tutoring in a chapter contributed to the book, Students as tutors and mentors (Goodlad, 1995). The chapter presents a review of the evaluation research around the world related to the effectiveness of college students as tutors for schoolchildren.
Wasik (1998) reviewed studies of 17 programs that used volunteer tutors to help improve students’ reading abilities. While the evidence suggested that volunteers can indeed help many children improve their reading skills, the results varied considerably across programs. Furthermore, only two of the 17 programs reviewed compared students’ achievement with that of a control group, which makes the results less certain. Wasik's findings are consistent with those of the National Research Council's 1998 report, Preventing Reading Difficulties in Young Children, which concluded that there is “[no] evidence confirming that [volunteers] are able to deal effectively with children who have serious reading problems” (p. 238). Nevertheless, it is likely that there are other evaluations not included in either of these non-systematic reviews, which could have altered the findings.
In contrast, Shanahan's (1998) review of the research on volunteer tutoring found that, despite many limitations, these programs can indeed be effective in improving student achievement. However, Shanahan offers little detail about the methodology used in his review, and, as with the reviews listed above, the results of several important studies may well be excluded.
Most recently, Elbaum et al. (2000) reported on a meta-analysis in the Journal of Educational Psychology focused on the effectiveness of one-to-one tutoring programs for improving reading ability in elementary students at risk for reading failure. The authors reviewed 29 studies involving 42 samples of students between 1975 and 1998 and found that trained tutors can help students improve in reading skills. This review is helpful, but only focused on reading as a subject matter and included
The limited and often-conflicting evidence on the effectiveness of volunteer tutoring programs leaves us with many questions. First and foremost, is there good evidence to encourage policymakers and school leaders to continue to pursue volunteer tutoring as a possible strategy for improving the academic skills of young students? How do differences in participants (e.g., age, gender) impact the effectiveness of the programs? And how do differences in tutoring programs (e.g., program structure, program focus, types of tutors) affect the results?
2. Objectives
The objective of this systematic review was to gather, summarize, and integrate the empirical research on volunteer tutoring programs in order to help policymakers, educators, parents, and other stakeholders understand whether this type of intervention might be an effective tool for improving academic skills for elementary students.
3. Methods of the Review
Criteria for Considering Studies for Review
Types of Studies
Only randomized field trials were included in the review. Quasi-experimental studies that employ treatment and control groups matched on pre-tests of key outcome variables were not included in this review. Pretest-posttest studies, or those in which a treatment group is compared to another treatment group, were not included. Studies published before 1985 were not included. Only English-language studies of programs conducted in the United States were considered, due to the limited resources for this review. Furthermore, we excluded studies of programs that were especially designed to address the needs of students with limited English proficiency (LEP), because such specialized programs are not representative of most volunteer tutoring programs for elementary and middle school students.
Types of Participants
Only studies of programs involving adult, non-professional tutors were included. Although these tutors were almost always referred to as “volunteers” in the literature, those programs that pay a small stipend to tutors (such as undergraduate tutors who are tutoring as part of a work-study program) were also included. Several of the programs in this area are those that train parents to tutor their own students; such programs are included in this review, and are coded for separate examination in the subgroup analysis. In terms of the tutees, only studies of programs that serve students in grades K-8 (elementary and middle school) were considered, since this is the population typically served by volunteer tutoring programs, and because such programs are fundamentally different than those provided to high school students.
Types of Interventions
The interventions featured regular tutoring sessions with an academic focus for at least one month in duration. The duration restriction was included due to a belief that programs lasting only a few days were qualitatively different than longer programs with sustained exposure. With regard to intervention focus, we did include interventions with other components in addition to an academic focus (such as behavior modification); however, the evaluation had to focus on at least one academic outcome measure.
Types of Outcome Measures
The original intent of the review was to consider all outcome measures related to student achievement, including distal outcomes (ones the program are actually intended to influence, such as school grades or standardized achievement measures), as well as proximal outcomes (intermediate measures that might be influenced by tutoring and then might lead to improved outcomes in the future, such as student attendance rates). However, the review yielded very few studies that analyzed school grades or attendance rates; rather, most studies focused on various standardized assessments of math and reading skills, or “authentic” measures of reading and writing skills. As a result, our review focused on these outcomes.
Many of the included studies, particularly those with a reading focus, examined several outcomes and it was the task of the reviewers to categorize those outcomes within our classifications of outcome measures. We organized the outcomes into six broad categories based on different concepts described within the retained articles. Listed below are the six classifications and the outcome measures that fit within each class. (Table 11 in Section 12 presents each class of outcome measures along with the individual outcomes from each study that are included within the broad class.)
The outcome measures that the reviewers included within this class are generally subtests within the general reading standardized assessments (examples of standardized assessments with reading subtests include the Gates MacGinitie Reading Test, the Wide Range Achievement Test reading section, the Comprehensive Test of Basic Skills, the Woodcock-Johnson Psycho-Educational Battery, and the Test of Word Reading Efficiency). Examples of decoding outcome measures include Word Identification, Word Attack, Letter Identification, DIBELS fluency assessments, Word Recognition, and Vocabulary tests. Also included in this category were subtests focused on such topics of consonant sounds, short vowels, digraphs and combinations, sight words, and nonword decoding. Some of these measures are not standardized; for example, the Morris et al. 1990 study utilizes a word recognition outcome that is not related to any standardized assessment.
While there are numerous ways that these categories could be divided, the goal was to develop a reasonable number of categories that included similar types of outcome measures. In this category, the focus is on the particular sub-skills necessary for young students to become fluent readers.
Search strategy for Identification of Studies
For this systematic review, titles of studies on volunteer tutoring programs were identified using several methods. First, the reviewers searched the C2-SPECTR, EbscoHost Research Database using the following databases: Academic Search Premier; Primary Search; Professional Development Collection; Middle Search Plus; Psychology and Behavioral Sciences Collection; PsycINFO; Sociological Collection, ERIC (Education Resources Information Center), and Proquest Digital Dissertations.
The initial search included the following key words in various combinations and truncations: “Volunteer or mentor or tutor* or tutorial programs,” and “elementary or primary education or middle school students or junior high school students or early intervention,” and “control or random or experiment or evaluation or program not peer.” In order to find any articles that have yet to be updated in the electronic databases, we conducted online reviews of the table of contents of several major journals that are most relevant to our study, including: Education Next, Education Policy Analysis Archives (EPAA), Educational Evaluation and Policy Analysis (EEPA), Reading Research Quarterly (RRQ), and Review of Educational Research (RER) for the years 2003-2005. Only studies conducted in the United States with native English speakers were included in the review. The article search was restricted to studies conducted in the United States only for the sake of policy relevance. That is, interventions employed in educational systems in other countries would not likely fit well within the education policy environment in the United States.
The resulting list of articles was augmented by other research studies referenced in four widely-cited reviews on volunteer tutoring listed above: Elbaum et al., 2000; Wasik, 1998; Shanahan, 1998; and Topping and Hill, 1995. The reviewers also consulted with several sources in order to refine the search process, including an information specialist or librarian, a reading specialist, and the Campbell Collaboration's Information Retrieval Policy Brief (Rothstein, Turner, & Lavenberg, 2004). Studies were retrieved primarily from the University of Arkansas Library System, Interlibrary Loan, University Microfilms, and the databases listed above. All study titles and inclusion decisions were documented and managed using Excel software in order to maintain accuracy and consistency among the reviews. When possible, PDF files of all articles were saved in a central network folder; hard copies of these and print-only articles were also kept on file.
The list of study titles generated in this process was then narrowed through a review of the studies’ abstracts by at least two reviewers. Once the abstracts were retrieved and reviewed, both reviewers reviewed the full text of all studies chosen. Studies that passed the full-text initial review were passed into the full-coding stage. After these studies were fully coded (each coding involved at least two reviewers), the final set of studies that met all inclusion criteria was then analyzed and the results were synthesized. If the two reviewers arrived at different conclusions during the coding process, the coders reconciled whether or not to keep the article. The reconciliation process consisted of a meeting in which each coder explained the rationale for retaining or rejecting the article until agreement was reached. If agreement were not reached, the coders would default to retain the article. In the final stage, the lead reviewer settled any disagreements.
With respect to study design, it is commonly accepted that randomized designs are the strongest designs upon which to base causal inferences. Initial exploration of the available literature uncovered numerous randomized field trials. Choices in the review were based upon the premise that we should use the most reliable evidence available. Had we found only a handful of randomized field trials, we would have then chosen to include high-quality quasi-experimental designs. However, because we identified nearly thirty study cohorts within more than 20 studies or reports based on randomized field trials, we made the decision to exclude quasi-experimental designs from the meta-analysis.
Selection of Trials
As shown in table 1 below, there were 1,437 total studies initially identified based on the search terms in search databases – some of these studies were duplicates from multiple searches. Of the total studies identified, there were 969 unique study abstracts to be reviewed.
Original Search Terms and Yields
The list of study titles was generated from electronic database searches, hand searches, and the review of reviews. Abstracts were collected for each title and two coders read each abstract to determine whether the study met the inclusion criteria. The reviewers eliminated abstracts based on the following guide: The article reported a meta-analysis The study did not employ an experimental design (e.g., case study, narrative, brief analysis/report about 1-2 pages in length, or qualitative study) The study was conducted prior to 1985 The tutees resided outside the United States or were non-English speaking (i.e., ESL, LEP) The tutors were peer or professional tutors (i.e., not volunteer adult tutors) The students fell outside the parameters of grades K-8 The focus of the intervention did not have at least one academic outcome The intervention duration was less than four weeks, or one month The student population was specialized (i.e., deaf, blind)
For a study to be eliminated from the pool, the abstract had to include information clearly indicating that the study met one of the exclusion criteria described above. If not enough information was provided in the abstract, the reviewers could not exclude the study and thus passed it to the next round for further analysis.
A majority of articles during this round were excluded for the following reasons: the studies did not employ an experimental design; the tutors were not volunteers; the tutees were not English speaking or resided outside the United States; or the tutees fell outside the parameters of K-8 grades. The screening stage narrowed the pool of studies a great deal; only 233 of 969 studies passed the initial screening of article abstracts.
The two reviewers then put the remaining 233 articles through a full-text initial review. The initial evaluation of the full text was focused on the introduction and methods section of the article. The purpose of this step was to achieve a fuller understanding of the article without having to fully code every retained article. The reviewers first examined the article to make sure the article reported findings of a study rather than a simple program description. Second, we reviewed the “Methods” section to ensure the students were randomly assigned to a treatment and control group. Finally, the reviewers also eliminated articles based on the general exclusion criteria described above as the article introductions often provided better information than was provided by the abstracts. This stage also resulted in a substantial narrowing of the study pool. Only 56 articles passed this stage and were moved into the full coding stage.
Finally, the final full coding of the 56 retained articles also resulted in the exclusion of more than half of the remaining studies. In the end, only 21 studies were included in our meta-analysis, although we had 28 study “cohorts” from those articles since a few of the articles had multiple cohorts requiring separate coding. For example, the Cobb 2000 article reported results for each of three grades separately and did not provide any pooled data; thus we included the results of this study as three separate Cobb 2000 study cohorts.
A total of 35 articles were excluded during the final full-coding phase. Of the 35 excluded articles, 30 were eliminated due to the following reasons: intervention duration; not a true randomized field trial study design; program intervention implemented outside the U.S.; no relevant academic outcomes; tutoring was not face-to-face; and the use of professional rather than volunteer tutors. The other 5 articles excluded from the meta-analysis were eliminated due to quality concerns with the statistics computed in those studies. The 5 studies in question reported insufficient statistics for us to include in our meta-analysis.
We had initially planned to include quasi-experimental designs, but after identifying 21 studies which employed randomized designs (the strongest designs upon which to base causal inferences), we excluded the quasi-experimental designs from the meta-analysis.
Assessment of Methodological Quality
The quality of each study (and its reporting) was assessed according to several characteristics, including: 1) the transparency of the study; that is, the clarity with which the investigators reported the random assignment procedures; 2) the integrity of the random assignment design and whether investigators addressed violations of the design; 3) the existence of high levels of attrition of either tutees or tutors from the sample initially randomized; and 4) the existence of substantial problems with respect to treatment fidelity. The quality of each study was assessed by all four reviewers during coding review sessions.
Of the 21 studies included in the meta-analysis, none had clear problems with student assignment to treatment and control conditions and none had any evidence of problematic attrition. Questions with fidelity were raised in a few of the included studies and these concerns are noted in Table 4, which summarizes the details of each of the included studies. However, the reviewers did not eliminate any of these studies from consideration.
A final study quality problem was related to reporting of study statistics. Six of the excluded studies failed, at least in part, due to the failure to report adequate descriptive statistics so that the reviewers could compute standardized mean difference effect sizes. Two of these studies in particular (Compton, 1992; Meier & Invernizzi, 2001) did report inferential statistics that would have allowed for the computation of the necessary descriptive statistics; however, the values presented were not consistent. The Compton dissertation had a total sample size of 483, but reported standard deviations that did not match either the reported standard errors or the reported t value and that would have yielded an effect size of nearly 2 standard deviations. The Meier and Invernizzi article reported means but no standard deviations, and the F values reported had degrees of freedom inconsistent with sample sizes in the study.
Data Management & Extraction
Of the 234 articles initially reviewed, we collected 109 articles from online sources, 66 articles were available in the University of Arkansas library on microfiche, microfilm, or in bound periodicals, 58 articles (including dissertations) were requested by inter-library loan, and 1 dissertation was uncollected when the library determined it had exhausted all possible sources to locate a circulating copy. After collecting an electronic or paper copy of each article, we extracted the information from each article using a designed coding form. The entire review team created the coding form. All study coding, data management, and meta-analysis was done using Excel software.
Data Synthesis
We used Hedges’ unbiased estimate g of the standardized mean difference effect size statistic (the difference between the treatment and control group means on an outcome variable divided by the pooled standard deviations for the post measure) for each outcome measure. When means and standard deviations were not reported, we estimated the effect sizes using the procedures recommended by Wilson and Lipsey (2001). When both pre and post measures were available, we subtracted pretest group differences, which in most cases were minimal because of the requirement of random assignment. If a study reported only adjusted posttest means or posttest means we computed the treatment-control difference.2 In either case, we used the pooled standard deviation of the posttest scores as our denominator in computing the effect sizes d. To get unbiased estimates of the population effect size, we divided by the approximation of Hedges and Olkin (1985): 1 – 3/(4N – 9).
Some of the studies employed a variety of outcome measures to assess program effectiveness. Because math outcomes are qualitatively different from verbal outcomes, we did not calculate the “effect” of each individual study or the “overall effect” of all available studies. However, we did calculate the overall effect on reading measures. To accomplish this, we computed an overall reading effect size for each of the 25 study cohorts for which some type of reading outcome was assessed. Next, to determine whether an intervention had a greater effect in any one area, we conducted separate meta-analyses of key outcome areas, including standardized overall reading, standardized letters and word skills, standardized reading comprehension, measures of oral reading fluency and of writing, as well as standardized math performance. If a study measured a key outcome in several ways, we averaged the effect sizes of the measures to ensure that each study only contributed one data point to the analysis for each key outcome and that no study was unduly “weighted” in the meta-analysis.
Homogeneity Analysis
The homogeneity analysis test determines if variations in the effect sizes are due to sampling error or other factors. The decision to use a fixed effects model or random effects model is based on the homogeneity analysis. The analyses of the overall effects and of the six key outcomes revealed Q-statistics that were not large enough to allow us to reject the null hypothesis of homogeneity. That is, the variability across effect sizes did not exceed what would be expected based on sampling error (Lipsey & Wilson 2001). Therefore, data syntheses in our study employed fixed effects models.
Sensitivity Analysis
Using the Comprehensive Meta Analysis ® software, we tested the extent to which our main results were sensitive to any one study's inclusion in the meta analysis. The “one study removed” analysis presents the average standardized mean difference of all remaining studies after each study, in turn, is removed from the analysis. In the end, the sensitivity analysis revealed that one study, with a very large sample, had a disproportionate impact on the meta-analytic outcomes. Consequently, all results based on the reading outcomes reported here exclude the Ritter-00 study from the sample (according to the sensitivity analysis, the Rittter-00 study did not have a disproportionate impact on the mathematics global outcome; thus, the Ritter-00 study was retained in the sample for the meta-analysis of the mathematics outcomes).
Subgroup Analysis
Subgroup analyses were conducted to compute differential mean effect sizes based on various program characteristics, including: 1) types of tutors; 2) age of tutees; 3) highly structured vs. unstructured programs, and 4) publication source. That is, we examined whether published studies were more likely to show positive program effects. These subgroup analyses were conducted only on the four reading outcomes reviewed here as the writing and math outcome domains only had six and five studies, respectively.
Publication Bias
The publication bias was measured with the “trim and fill” procedure, where the Funnel Plot was visually inspected for differences. Additionally, we examined the possibility of publication bias by using publication type as a moderator variable.
Incomplete Reporting of Study Data
The retained randomized field trials for our study provided enough information for effect sizes to be computed.
PostHoc Subgroup and Moderator Analysis
As part of the meta-analysis, we conducted some subgroup and moderator analysis to determine if differences in effect sizes existed due to characteristics within the studies. The moderator analysis included: 1) type of tutor (parent, college, community); 2) grade 1 or other; 3) level of program structure; and 4) publication type.
Description of Program Characteristics for 28 Study Cohorts Included in Meta-Analysis
4. Results
Description of Included Studies
In the end, the search yielded 21 unique articles, reports, or dissertations; there are 28 unique “study cohorts” or “studies” identified in these 21 reports, as some reports focused on multiple cohorts analyzed separately. Each of the 28 studies is described in some detail in Table 4 in Section 9. The evidence base described here relies upon a sample of 1,676 study participants, 873 of whom were in the tutoring treatment groups and 803 of whom were in the control groups.
Reading global: evidence based on 14 studies with a total sample size of 819. Reading letters and words: evidence based on 15 studies with a total sample size of 798. Reading comprehension: evidence based on 8 studies with a total sample size of 546. Reading oral fluency: evidence based on 12 studies with a total sample size of 635. Writing: evidence based on 6 studies with a total sample size of 228. Mathematics global: evidence based on 5 studies with a total sample size of 643.
There is overlap among the studies and samples described above. That is, many of the same studies with reading global outcomes, for example, also assessed outcomes in the reading oral fluency domain.
For example, the Howard Street Tutoring Program, as described by Morris, Shaw and Perney (2000), was classified as structured: “The 3:00-4:00 p.m. tutoring period is carefully planned and work filled, with very few disruptions….a typical 1-hour tutoring lesson takes the following form …” (pp. 136-137). In another case, Vadasy, Jenkins, Antil, and Wayne (1997a) describe an early version of the Sound Partners program “The intervention was a set of 100 after-school lessons, each 30 min long.to be used by tutors to teach phonological and early reading skills to first-grade students” (p. 31).
Alternatively, the Start Making a Reader Today (SMART) program was decidedly unstructured. “Tutors are provided with a broad framework to use during sessions, rather than specific techniques” (Baker, Gersten, & Keating, 2000, p. 497). They also suggest it is easier to recruit tutors if they are not expected to acquire specialized instructional skills. The paired reading approach and repeated reading approaches, where students select their own materials to read are also classified as non-structured (e.g. Miller, 1998; Weiss, 1989).
In our sample of 28 study “cohorts”, 15 were from programs classified as highly structured (study sample = 919), while 13 were not (study sample = 757).
Overall Effect Sizes Across Studies for All Outcomes
We began by examining the overall effects of volunteer tutoring on student reading outcome measures. Twenty-five studies assessed reading measures of one type or another. In eight of these 25 studies, the overall reading score was based on a single reading measure; we computed an average effect size based on multiple reading outcomes for the remaining 17 studies that employed multiple reading outcome measures. Our analysis indicates that volunteer tutoring interventions of the type reviewed here have a significant positive effect on the verbal skills of participating students. Figure 1 shows that the average effect of volunteer tutoring programs on reading outcomes for elementary students is 0.23 using a fixed effects model.
We then conducted sensitivity analyses to examine whether the average effect of volunteer tutoring on reading was disproportionately influenced by the result of any single study. We found that the Ritter-00 study had a sample that comprised approximately 25 percent of the total sample and was more than twice the size of the next largest study sample. The one study removed analysis highlighted in Figure 2 shows that the Ritter-00 did have a substantial negative impact on the average effect size; removing this study increased the average effect size to 0.30. Moreover, the program evaluated in Ritter-00 was unique from the other programs in that there was not a strong academic focus in the program. Given the disproportionate influence of the Ritter-00 study on the average effect size and the unique nature of the program evaluated in Ritter-00, we decided to exclude the study from the domain-specific and subgroup analyses that follow. Figure 3 presents the forest plot and average reading effect size of 0.30 for the set of 24 studies that excludes the Ritter-00 study. Using a 95 percent confidence interval for this effect size, we find that the average effect of volunteer tutoring on reading outcomes ranges from 0.18 to 0.42. The test for heterogeneity produced a Q-value that was not statistically significant (Q = 17.29, p = .80); thus, all subsequent results are reported using a fixed effects model.
Next, we examined the effect of volunteer tutoring programs on the following specific academic domains (as described in the methods section above): reading global, reading letters and words, reading comprehension, reading oral fluency, writing, and mathematics global.
Reading Global
Thirteen studies assessed outcome measures within the reading global domain; these studies included 195 tutored students in the analysis. As the forest plot for Reading Global indicates (see Figure 4 in Section 10), eight of the studies have positive effect sizes while six have negative effect sizes. However, only two of the most positive effect sizes are statistically different from zero. Overall, the average effect size for this outcome domain is +0.26, an effect that is statistically significant.
Reading Letters and Words
Fifteen studies assessed outcome measures within the letters and words outcome domain; these studies included 403 tutored students in the analysis. As the forest plot for this outcome indicates (see Figure 5 in Section 10), all but two of the studies have positive effect sizes. While only three of these positive effect sizes are statistically different from zero on their own, when all the results are pooled, the effect size is +0.41, an effect that is statistically significant.
Reading Comprehension
Eight studies assessed outcome measures within the reading global domain; these studies included 293 tutored students in the analysis. As the forest plot for Reading Comprehension indicates (see Figure 6 in Section 10), five of the studies have positive effect sizes while only three have negative effect sizes. Only two of these positive effect sizes are statistically different from zero, and the overall effect size is +0.18, an effect that is not statistically significant.
Reading Oral Fluency
Twelve studies assessed outcome measures within the letters and words outcome domain; these studies included 336 tutored students in the analysis. As the forest plot for this outcome indicates (see Figure 7 in Section 10), ten of the twelve studies have positive effect sizes. While only two of these positive effect sizes are statistically different from zero on their own, the pooled effect size for this outcome is +0.30, an effect that is statistically significant.
Writing
Only six studies assessed outcome measures that the reviewers classified within the writing domain; these studies included 111 tutored students in the analysis. As the forest plot for Writing indicates (see Figure 8 in Section 10), all six of these studies have positive effect sizes. While only one of these positive effect sizes is statistically different from zero on its own, the pooled effect size for this outcome is +0.45, an effect that is statistically significant.
Mathematics Global
Only five studies assessed outcome measures that the reviewers classified within the mathematics domain; because these studies were dissertation with large sample sizes, these five studies included a total of 292 tutored students in the analysis. As the forest plot for Mathematics Global indicates (see Figure 9 in Section 10), three of these studies have positive effect sizes and two have negative effect sizes. Only two of the positive effect sizes are statistically different from zero. When these five studies are pooled together, the overall effect size for this outcome domain is +0.27, an effect that is not statistically significant.
Mean and Standard Error of Effect Sizes for All Study Outcomes
* Significantly different from zero, favoring the treatment over the control group, p < .05.
Analysis of Impacts for Subgroups of Studies on Reading Outcomes
Next, we examined the possibility of differential effects of different types of volunteer tutoring programs on the reading outcomes. We focus here only on these “subgroup” effects in which there are at least 3 studies in each subgroup. Subgroups examined are described above and include: types of tutors, grade level of tutees, program structure, and publication type.
None of the outcomes had a significant difference in effect size by tutor type. That is, programs using parent, college age, or community tutors did not differ significantly in their effectiveness. Similarly, programs that included Grade 1 were not significantly different from programs for higher grades in their effectiveness. The only significant subgroup difference we found was that highly structured programs had a significant advantage over programs with low structure on the global reading outcome, with an effect size of .59 for structured programs and .14 for unstructured. The other reading outcomes did not differ significantly by amount of program structure. It should be noted that there were only three studies classified as highly structured that used global reading outcomes, and all three studies had the same lead author (Vadasy et al., 1997a, 1997b, 2000).
To assess the possibility of publication bias, we conducted the “trim and fill” procedure, which trims excessively large studies and imputes small studies which may be missing. Figure 10 shows that the 24 studies conform neatly and symmetrically to the shape of the funnel plot, suggesting that there is not a large effect of publication bias on our results. Additionally, the trim and fill procedure was conducted, but there were no excessively large studies to trim. Thus, the observed overall effect size is likely based on an unbiased set of studies.
As an additional test of the possibility of publication bias or “file-drawer bias”, we distinguished studies published from dissertations or other unpublished works. The overall trend revealed in this meta-analysis does not indicate publication bias. That is, in each of the reading domains examined here, the effect sizes from the studies published in journals were not significantly larger than those presented in dissertations and unpublished studies.
Mean and Standard Error of Effect Sizes, by Publication Type
* Significantly different from zero, favoring the treatment over the control group, p < .05.
5. Discussion and Conclusions
The objective of this systematic review was to gather, summarize, and integrate the empirical research on the effectiveness of volunteer tutoring programs. The current research base is strong – relatively to that of other educational interventions – in that the review uncovered more than 20 randomized field trials. However, this good news is tempered by the fact that most of these studies employ small samples. Nineteen of the 28 study cohorts in this meta-analysis included 25 or fewer students in the tutoring group; only three of the study cohorts had full study samples (treatment plus control) of more than 100 students. In the end, these 28 study cohorts in these 21 articles reporting on randomized field trials included a full study sample of 1,676 students (873 tutored students and 803 control students). In cases such as this in which there are many studies, most with small samples with very little power to detect program effects, the strength of metaanalysis is that the results of the small studies are pooled and the statistical power is enhanced.
There were numerous academic outcomes assessed in the 21 articles that employed experimental designs to investigate the effectiveness of volunteer tutoring programs. First we analyzed the overall effect of volunteer tutoring programs on all reading outcomes and found a positive and statistically significant positive effect of 0.30 standard deviations. We then grouped the specific outcomes into six domains. Two of these domains were very broad and employed standardized assessments of general skills in reading and math: reading global and math global. The other four outcomes were less broad and focused on specific sub-skills related to reading and language: letters and words, reading comprehension, reading oral fluency, and writing.
The central goal was to examine whether a volunteer tutoring intervention represents a potentially effective strategy for improving academic skills for young students. The answer, according to the existing set of randomized field trials, is a qualified yes. Participation in a volunteer tutoring program results in improved overall reading measures of approximately one-third of a standard deviation. With respect to particular sub-skills, students who work with volunteer tutors are likely to earn higher scores on assessments related to letters and words, oral fluency, and writing, as compared to their peers who were not tutored. The effect sizes connected to these outcome domains were relatively consistent, ranging from 0.26 to 0.45 (See Table 2 above).
A secondary goal was to assess whether certain programs are particularly effective. As for the secondary goal, the review reveals that the programs are unique and the individual studies are based on small samples. Furthermore, the programs are small enough so as not to be replicable. Thus, the relevant question may not be “which of these programs are most effective?” Rather, the more important question is “what are the characteristics of effective programs?”
To address this question, the reviewers computed differential mean effect sizes based on various program characteristics, including: 1) types of tutors; 2) age of tutees; and 3) highly structured vs. unstructured programs. These subgroup analyses were conducted only on overall reading and on the four specific reading domains.
However, for the most part, effects were not significantly different across these subgroups. Nonetheless, we can derive some lessons from the characteristics of the programs analyzed. The majority of the studies reviewed here evaluated reading-focused programs delivered to primary age students. Programs did not have to be highly structured to have positive effects, nor did they have to use a particular type of person as a tutor.
In the end, the results of this analysis should serve as one important piece of evidence used by policymakers and educators who are deciding whether to employ volunteer tutoring as a strategy to improve academic skills for young students. As educators across the country work to meet adequate yearly progress goals in state accountability systems, and as they seek affordable ways to offer additional services to students at risk of not meeting annual academic goals, it would be worthwhile to consider structured, reading-focused volunteer tutoring programs as strategies to improve reading and language skills.
It is also worth noting that the research base for volunteer tutoring, although based mostly on studies with small samples, is useful precisely because there are so many studies that employ experimental designs. This illustrates the power and utility of meta-analysis. While many of the individual studies, standing alone, do not show significant program effects, the overall effect is relatively large and statistically significant in five of the seven outcome domains examined here. As a result, the evidence base in the field benefits from small randomized field trials in which data are reported thoroughly and carefully.
The results of this analysis also reveal that we know very little about the effectiveness of these types of interventions at improving math outcomes. This is disappointing given the important role that early numeracy skills play in later math achievement for students in elementary and middle school. Given this lack of information, it would be useful for educators to develop and implement volunteer tutoring programs focused on early math skills while researchers worked collaboratively to evaluate the effectiveness of these programs. In the end, both practitioners and researchers would be better off as we would begin to learn more about whether this type of intervention might in fact be beneficial for students struggling with early math skills.
6. Plans for Updating the Review
The authors will attempt to update the review approximately every two years. To produce as comprehensive of an update as possible, the researchers plan to conduct more rigorous reviews of the grey literature, contact additional experts in the field of volunteer tutoring, and include studies conducted outside the United States. The researchers also plan to revisit our decision to exclude studies that use quasi-experimental designs.
7. Acknowledgments
The initial investigation into the field of volunteer tutoring was supported under the “Evaluation of the West Philadelphia Tutoring Project” conducted by faculty and staff at the University of Pennsylvania from 1998-2000. The Smith Richardson Foundation provided funding to initiate this review of volunteer tutoring effectiveness. The draft protocol for the pilot review was completed with the help of many colleagues, including participants in the University of Pennsylvania's Graduate School of Education seminar on conducting systematic reviews and meta-analysis, led by Dr. Rebecca Maynard.
The systematic review has been completed at the University of Arkansas, with the financial support of the Campbell Collaboration. The results of this study do not necessarily represent the official opinion or policy of the Smith Richardson Foundation, the University of Pennsylvania, or the University of Arkansas.
8. Potential Conflict of Interest
Gary Ritter was not involved in the decision whether to include the Ritter 2000 study.
9. Included Studies: Characteristics and References
Summary of Characteristics of Included Studies3
Reference List – Included Studies (21 articles)
10. Figures: Forest Plots for All Key Outcomes
Heterogeneity Statistics for a Fixed Effects Model: Q = 27.20, df = 24, p = .30, and I squared = 11.75
Funnel Plot of Standard Error by Hedges's g for Overall Reading (24 studies, Ritter-00 excluded)
11. Additional Tables
Level of Structure of Various Programs
Components of Effect Size Outcomes by Outcome Domain by Study
12. Excluded Studies: Characteristics and References
Summary of Characteristics of Excluded Studies
Internal Sources of Support
University of Arkansas
External Sources of Support
The completion of this review during 2005 was supported through a grant from the Campbell Collaboration—Education Review Group. Earlier support for this project was provided by the Smith Richardson Foundation under grants to the University of Pennsylvania and the University of Missouri.
The draft protocol for the pilot review was completed with the help of many colleagues, including participants in the University of Pennsylvania's Graduate School of Education seminar on conducting systematic reviews and meta-analysis, led by Dr. Rebecca Maynard.
The initial investigation into the field of volunteer tutoring was supported under the “Evaluation of the West Philadelphia Tutoring Project” conducted by faculty and staff at the University of Pennsylvania from 1998-2000.
The systematic review is now being conducted at and funded by the University of Arkansas. The results of this study do not necessarily represent the official opinion or policy of the Smith Richardson Foundation, the University of Pennsylvania, or the University of Arkansas.
Acknowledgments
Thanks to Rebecca Maynard for oversight and guidance in getting this review started and throughout. Thanks to Geraldine Macdonald and Jeff Valentine for helpful comments and suggestions on this review. This review benefited from technical advice from Mark Lipsey, Chad Nye, Herb Turner, and Robert Boruch.
Potential Conflict of Interest
Gary Ritter was not involved in the decision whether to include the Ritter 2000 study.
Footnotes
1
The four reviews are: Devin-Sheehan, Feldman, & Allen, 1976; Ellson, 1976; Fitz-Gibbon, 1977; and Rosenshine & Furst, 1969.
2
Of the 28 study cohorts analyzed here, 12 were analyzed using posttest scores adjusted for pretest differences (equivalent to gain scores); 3 were analyzed using posttest scores statistically adjusted for pretest differences (using ANCOVA); 12 analyzed based on posttest scores only as no pretest scores were provided; and 1 was analyzed based on a mix of posttest only scores and adjusted posttest scores. The one common denominator was, in fact, the denominator of the effect size calculation. In each case, the denominator of the effect size statistic was the pooled posttest standard deviation.
3
The full citations for these 28 studies are presented in a reference list following the table.
4
d = t sqrt(1/nE + 1/nC) (Glass, McGaw, & Smith, 1981)
5
This study reported only the gain scores and their standard deviations, but no pre/post correlation. Thus, we were unable to derive the posttest pooled standard deviations. To estimate the posttest pooled standard deviation for our effect size computations, we used the gain score standard deviation, which assumes a pre/post correlation of 0.5 and equal pre/post standard deviations.