Abstract
Chile features a universal school choice system, in which a government voucher provides families an opportunity to send students to public or private schools of their choosing. Since its implementation in 1981, the amount of the voucher was flat without adjustments for family income, creating incentives for schools to enroll students from economically advantaged families. In 2008, a policy change adjusted voucher values by the poverty level of students and the proportion of poor students attending each school. We evaluate the effect of this policy on primary school students’ standardized test scores, using time-distributed fixed effects models. We find a positive and significant effect of the means-tested voucher policy on Math and Language achievement. The effect is much larger among private-voucher schools serving poor children, and it increased over the years after the policy change, suggesting that schools require some time to realize the benefits of the policy. Our findings show that moving from a flat to a means-tested voucher improves achievement and equality.
The Coleman Report (Coleman et al. 1966) was a vast and path-breaking empirical research project that changed the ways in which researchers and policy-makers evaluated the role of schools in educational opportunity and inequality. Before the report, schools were mainly evaluated according to the resources allocated to students’ education. After it, the evaluation of a school depended on its outcomes, i.e., the gains in students learning, the rate of student graduation, and the students’ later performance in the labor market. In fact, the study pioneered the assessment of students through tests (Hanushek 2016; Alexander and Morgan 2016).
The Coleman Report fueled intense attention to school effects on student learning that remains relevant today. The report was commonly presented as evidence that between-school variation in resources—teacher characteristics, class size, infrastructure, per pupil expenditure, and so forth—has a relatively weak correlation with student achievement, and that families were the most important determinant of student outcomes. Still today, there is debate over the relevance of schools and families to students’ outcomes and, more generally, the factors that shape students’ results. Early studies motivated by the pioneering work of Coleman involved estimations of schools’ educational productivity that linked output—educational achievement—with educational inputs and family and student characteristics (see for example Hanushek 1997).
Research also suggested that the relevance of school and family resources may vary across countries depending on the level of economic development. Heyneman and Loxley (1983) were the first to posit that, in low- and middle-income countries, variation in the quality of school resources and teachers could matter more than family inputs for academic achievement in primary school and that relevance of school and teacher quality increased as the income level of the country decreased. Reviews of the so-called school effects in low- and middle-income countries suggest that several factors are significant in less-developed contexts, including basic items such as instruction time, textbook availability, teacher absence, and specific teaching methods (Fuller and Clarke 1994; Hanushek 1995; Glewwe et al. 2013).
In addition to drawing attention to the relative importance of school and family inputs, the Coleman Report brought to the fore wide inequalities in achievement between groups defined by race and ethnicity, socioeconomic background, and region. In the early twenty-first century, the racial and socioeconomic achievement gaps remain substantial and vast socioeconomic and racial and ethnic inequalities persists in educational attainment (Belley and Lochner 2007; Fryer and Levitt 2006).
This article is motivated by the two main issues underlined by the Coleman Report: on one hand, the relevance of school-level factors on students’ achievement, including the potentially strong role that school resources may play in low- and middle-income countries; on the other hand, the substantial achievement gaps between advantaged and disadvantaged groups in a given society. We address these issues with analysis of a major policy change in school financing aimed at improving equality of educational opportunity in Chile. We investigate whether focusing resources on schools serving disadvantaged students improves their educational outcomes and overall equality of educational achievement.
The Chilean Voucher System
Chile offers a unique setting to address these questions. In the early 1980s a military regime undertook sweeping reforms in many Chilean markets. The educational system was not an exception. A country-wide school voucher system was implemented. In this system, the government established a universal school voucher system that paid a flat, per-student subsidy to public schools and to private schools that did not charge tuition (private schools that did charge tuition received no vouchers and families paid tuition in full). Families were free to send students to the public- or private-voucher schools of their choosing. The universal voucher system paved the way for private sector participation in publicly financed education. By 1990, the proportion of students attending public schools had dropped from almost 80 to 60 percent and those attending private-voucher schools had increased to more than one-third. By 2014, 55.6 percent of school-age children were enrolled in private schools funded by government vouchers, 36.8 percent were in public schools, and 7.6 percent attended private-paid schools.
School voucher systems implemented around the world vary in terms of institutional design (West 1997; Patrinos 2002; Gauri and Vawda 2003), and these design specificities determine whether school choice improves outcomes (Epple and Romano 2012). Four institutional features are relevant in the Chilean case: the amount of the per-student voucher, rules about admission and expulsion of students, alternative sources of school financing, and teachers’ regulations (Mizala and Torche 2012). At inception, the Chilean voucher was flat; that is, it did not vary with family socioeconomic resources (González, Mizala, and Romaguera 2004). Furthermore, private-voucher schools (but not public schools) were allowed to select students at will. No additional tuition funds paid by parents were permitted, but a change implemented in 1993 allowed primary and secondary private-voucher schools (and secondary public schools) to charge “add-on” fees to families to supplement the government voucher, under a withdrawal schedule that reduced the subsidy as parental fees increased. This system, known as “shared financing” (financiamiento compartido), expanded rapidly from 16 percent of the voucher sector enrollment in 1993 to about 80 percent in 1998, stabilizing thereafter. Over the past two decades, subsequent administrations have increased the real value of the voucher and targeted assistance on schools serving the most deprived populations. However, the central features of the voucher system have remained unchanged.
Critics of the Chilean voucher system argue that the institutional features of the system provide incentives for private-voucher schools to select socioeconomically advantaged students who have, on average, higher educational performance and are less demanding in terms of resources, rather than to increase their value added in terms of educational achievement. In fact, the studies that examine test score gains in voucher schools compared to public schools generally find positive but very small or insignificant effects (Mizala and Romaguera 2000; McEwan and Carnoy 2000; Anand, Mizala, and Repetto 2009; Lara, Mizala, and Repetto 2011). At the same time, the school choice policies implemented in Chile are associated with substantial socioeconomic inequalities in educational achievement and socioeconomic segregation between school sector and across schools within sector (Torche 2005; Hsieh and Urquiola 2006; Mizala and Torche 2012; Valenzuela, Bellei, and de los Ríos 2013).
The cost of providing education depends not only on the cost of educational inputs, but also on the socioeconomic context in which education must be provided. If students at a given school are likely to live in poverty, live in single-parent households, or have poorly educated parents, then obtaining a given performance level will imply higher costs. Schools need to be compensated for this additional cost to reduce incentives to “cream-skim” advantaged students as a competitive strategy (Duncombe and Yinger 2000; Reschovsky and Imazeki 2001). Indeed, the theoretical literature on voucher concludes that one way to ameliorate inequality and segregation is to design a voucher value as a function of family income, or to restrict the vouchers to poor families. For example, Epple and Romano (1998) claim that less-able students will need more financial assistance and vouchers will need to be income-dependent to avoid an increase in ability segregation. Hoxby (1996) argues that vouchers are particularly important for poor households, and that private-school vouchers should be means-tested. Bearse, Glomm, and Ravikumar (2000) state that if uniform and universal vouchers lead to more socioeconomic segregation, one obvious policy response is to use means-tested or targeted vouchers.
Following these insights, a major reform, known as “preferential school voucher” (Subvención Escolar Preferencial, or SEP), was implemented in 2008 in Chile. The reform amounts to transforming the flat voucher system into a means-tested one. It establishes an additional per-student subsidy for economically disadvantaged students, designated as “priority students,” and an extra subsidy for schools with a high proportion of disadvantaged students. When launched in 2008, the reform applied to pre-K through fourth grade, and it has expanded by one grade per year—to fifth grade in 2009, sixth grade in 2010, and so on—such that the policy reached all grades through the twelfth grade in 2016.
Schools’ enrollment into the means-tested voucher program is voluntary. In 2008, the first year of the reform, as many as 93 percent of elementary public schools enrolled in the program and by 2011, 99 percent of public schools had enrolled. In contrast, only 51.7 percent of elementary private-voucher schools enrolled in the first year. More private-voucher schools joined the following years, and by 2014, 72.4 of them were enrolled in the program. The reform substantially increased the value of the voucher for priority students, with schools receiving between 58 to 68 percent more funds for each priority student compared to their nonpriority peers, depending on the concentration of priority students in the school (Figure 1).
Value of the Means-Tested Voucher, 2005–2014
The intended objectives of the means-tested school voucher reform were to improve educational quality and to offer equal educational opportunity by focusing on the most vulnerable students. This analysis examines whether the reform resulted in an increase in students’ scores on standardized achievement tests, whether this effect is concentrated among poor students or expands to their advantaged peers, and whether the effects on achievement change over time. Standardized test scores are relevant outcomes because they predict high school dropout, college entry, and freshman retention rates.
The following questions guide our analyses: What is the average effect of the means-tested voucher policy on student test scores in schools that took advantage of the policy? Does the effect vary by the socioeconomic characteristics of the school? Does the effect vary over time after uptake?
A central challenge in detecting the causal effect of the policy on the schools that took it is unobserved selectivity. Given that schools’ enrollment in the program was optional those that enrolled can differ systematically from those that chose not to enroll. To alleviate bias emerging from unobserved selectivity we create a panel of schools between 2005 and 2014 and use fixed-effects at the school level and year. In this setting, we rely on within-school over-time variation to identify the effect of the means-tested school voucher policy on students’ achievement. While most analyses using fixed effects consider only an average effect post–policy implementation, we empirically test the hypothesis that schools may have taken time to fully adjust to the new policy environment and realize its potential benefits (e.g., Rauscher 2016) by implementing time-distributed fixed effects (Dougherty 2006).
Means-Tested School Voucher
The main objective of the 2008 preferential school voucher reform was to transform a flat voucher system into one that varied according to the students’ poverty level. To this end it establishes an additional per-student subsidy for economically disadvantaged students (priority students), roughly the poorest 40 percent of the student body, and an additional, much smaller, subsidy for schools with a high concentration of priority students. Criteria to define poor students targeted by the program are determined by the Chilean Ministry of Education and include family enrollment in a social assistance program targeted at families in extreme poverty; families belonging the lowest 33 percent of the income distribution by standard national socioeconomic classification regulations; families enrolled in public health insurance serving those without income; or families ranked below a cutoff based on a socioeconomic index including total family income, parents’ education, rural residence, and the poverty level of the county of residence.
To enroll in the means-tested voucher program, schools have to sign an Equality of Opportunity and Educational Excellence Agreement (Convenio de Igualdad de Oportunidades y Excelencia Educativa), wherein they commit to enrolling all students who apply regardless of their prior or potential academic performance, not charging add-on fees to poor students, retaining poor students regardless of their academic performance, and achieving improvements in students’ performance, especially for poor students. To fulfill this last condition, they are required to implement a four-year “school improvement plan” in exchange for the additional funds received. The extent of autonomy and support granted to enrolled schools by the Ministry of Education depends on the average test scores, adjusting for the socioeconomic status of the student body. The Chilean government classifies schools into five socioeconomic strata according to parents’ education, parents’ income, and the proportion of students deemed as “socioeconomically vulnerable” in the school. If the school performance is at or above the median for other schools with similar socioeconomic characteristics, schools are classified as “autonomous”; if performance is below the median for their socioeconomic group, they are classified as “emergent”; and if emergent schools fail to meet the quantitative goals required by the program within four years, they are deemed “in recovery.” The agreement is valid for four years and can be renewable for a similar amount of time.
Schools that do not meet these requirements see the resources associated with the means-tested voucher partially or totally withheld. In extreme cases, they can also lose their public funding and administrators could be forbidden from participating in the management of private-voucher schools or public schools. Because of these requirements, the means-tested school voucher reform both increased school resources and enhanced accountability. When it was implemented, the means-tested school voucher was the only source of public funding for education that required schools to fulfill institutional goals and improve academic performance.
The value of the means-tested voucher depends on the grade attended by the priority student, with primary-level students receiving more funds, and it is calculated on the basis of the average attendance rate of the student during the prior three months. The additional poverty concentration subsidy depends on the proportion of poor students enrolled in the school. Schools start receiving an additional concentration subsidy when poor students make up more than 15 percent of the student body, and the concentration supplement increases with the proportion of disadvantaged students, up to 60 percent (Table 1).
Monthly Value of the Means-Tested Voucher and the Additional “Concentration Voucher” (2014 US$).
SOURCE: Ministry of Education data.
NOTE: The amount of the voucher is calculated using the value of the Unidad de Subvención Educacional of March 2014 (exchange rate Ch$563.84 per dollar).
This amount corresponds to the voucher for primary education.
This amount corresponds to the voucher for secondary education (ninth to twelfth grade).
The means-tested voucher reform substantially increased the funds that schools received for enrolling socioeconomically disadvantaged students. In 2014 the monthly voucher per pre-K–fourth grade student was US$108 for a nonpriority student. It rose to US$171 for a disadvantaged student attending a school with less than 15 percent disadvantaged students and to US$183 if the priority students attended schools with 60 percent or more disadvantaged students (Figure 1). In 2011, the policy was modified in several ways. Benefits were extended to secondary school students; the value of the means-tested voucher increased by about 20 percent (see Table 1), and schools were allowed to use the funds more flexibly, in particular by lifting restrictions in the use of resources for personnel expenditures such as hiring additional teachers, paying overtime to teachers, or financing professional development for school leaders. It is important to clarify that investments financed by SEP resources are not earmarked and could benefit both priority and nonpriority students.
Research suggests a positive effect of the reform on students’ test scores (Correa, Parro, and Reyes 2014; Villarroel 2012; Carrasco 2014; Neilson 2013 Murnane et al. 2017). Correa, Parro, and Reyes (2014) use a difference-in-differences approach and a market-level analysis to identify schools that entered the program and received funds from 2009 to 2011 (treatment group) and those that chose not to participate in this program during this period (control group). Defining each municipality as an individual market, they also compare the changes in test scores after the implementation of the reform in a market where a different number of schools signed the agreement. Under both methodologies, they find a positive effect of the reform on private-voucher schools’ academic results.
Villarroel (2012) combines matching methods with a difference-in-differences approach to compare test results of private-voucher schools that joined the program to those that did not participate between 2007 and 2010. Like Correa, Parro, and Reyes (2014) they consider two points in time. Valenzuela, Villarroel, and Villalobos (2013) follow up on this study and suggest that the effect of the SEP policy has been largely concentrated in schools with better performance and serving a large number of disadvantaged students. Using an interrupted time series approach, Carrasco (2014) compares trajectories of schools that joined the SEP program with different control groups during 2005 to 2011. He finds that treated schools experienced gains in standardized Math and Language achievement test scores compared to the control groups. Both Neilson (2013) and Murnane et al. (2017) offer a general equilibrium analysis of the effect of SEP on overall test scores, test scores of poor students, and the income gaps in test scores. Both analyses show substantial increases in the test scores of poor students and a narrowing of the achievement gap between these students and the rest of the population. Given their focus on a general equilibrium outcome, they do not distinguish trends among schools that took SEP and those that did not.
These studies suggest that the means-tested voucher reform had a positive impact on students’ test scores. However, comparisons between two discrete time-points preclude the understanding of effects of the reform over time. Furthermore, the aggregate effects captured may hide substantial heterogeneity in the effects of the policy, both in terms of socioeconomic status of the students served by the school, and in terms of time elapsed since uptake. We expand on these findings to assess the effect of the means-tested voucher on private-vouchers schools, its socioeconomic heterogeneity, and variation over time.
Data and Methods
We create an annual panel of private-voucher schools between 2005 and 2014. The panel combines information about standardized test scores in Math and Language for fourth graders obtained from the Measurement System of Educational Quality (Sistema de Medición de la Calidad de la Educacion, SIMCE) with information based on a survey to parents that accompanies standardized score testing, and with administrative data from the Ministry of Education. Administrative data include information about schools, their location, enrollment, number of “priority students,” and the year in which schools took the SEP program. Given that virtually all public schools took the SEP program when it was launched in 2008, yielding minimal variation in the treatment in subsequent years, we focus on private-voucher schools, among which there is a more gradual entry into SEP.
The main objective of this analysis is to examine the effect of enrollment into the SEP program on average test scores at the school level. Given that schools that opted to take the program might be systematically different from those that did not in terms of unobserved factors, it is not possible to simply compare schools across treatment categories. For example, the decision to take the policy may be based on the socioeconomic composition of the school, competitiveness of the market where the school is located, or quality of school leadership. To alleviate the problem of unobserved selectivity, we implement a regression model with fixed effects for school and year, as follows:
where Test Scoremit identifies the mean school test score in subject m (m = Math, Language) for school i in year t. Tests are given in November of each year, which corresponds to the end of the school year. SEPit is a dichotomous variable identifying whether school i joined the SEP program at the beginning of the school year t (SEP is effectively an absorbing state, as there have not been transitions out of the program).
By design, the fixed effects estimator compares the average school-level test scores across all years before the policy was implemented to the average test scores across all years after implementation, making the implicit assumption that the effect of the policy is homogeneous over time after the absorbing state has been entered. This assumption is likely unrealistic. Developments such as adjusting to and gaining experience with the new policy, or optimizing the use of additional resources emerging from the policy, may result in an effect that changes over time. A flexible nonparametric way to assess the change in effects over time is known as time-distributed fixed effects (Dougherty 2006). This model replaces the treatment dummy SEPit with a set of dummies SEPitp where p is the number of years since uptake of SEP, yielding the following formula:
where n is the number of years postimplementation of the policy, which takes a maximum value of seven.
Findings: The Effect of Transitioning from a Flat to a Means-Tested Voucher on Achievement
Before moving to the core of the analysis, Table 2 compares the socioeconomic characteristics of schools that enrolled in the SEP program by year of enrollment with those that remained unenrolled. We examine the socioeconomic composition of schools based on mother’s education, father’s education, and family income percentile, all measured in 2006–2007 prior to the implementation of the policy.
Private-Voucher School Socioeconomic Characteristics and Average Fourth-Grade Test Scores, by Year of SEP Uptake
Based on these indicators, schools that entered the SEP program in 2008, the year the program was launched, served families that were, on average, more disadvantaged that those attending schools that did not join the program. Schools that entered in subsequent years served gradually more socioeconomically advantaged populations. Still, the schools that had not taken the program by the last year of observation served significantly more advantaged populations than those that entered the program at any point. For example, schools that had not taken the policy by 2014 had a mean mother’s years of schooling of 12.9 in 2006/07, compared with 11.5 years of schooling for those that entered in 2011–2014, 10.5 years for those entering in 2010, 10.2 years in 2009 and 10 years in 2008. In terms of family income percentile, schools that had not taken the program ranked in the 73rd percentile, compared to the 49th, 36th, 30th, and 28th percentiles for schools entering in 2011–2014, 2010, 2009, and 2008, respectively.
Table 2 also displays the mean standardized test scores in 2006/07 at the school level by year of entry into the program. The pattern is very similar to the socioeconomic composition. Schools that entered SEP in 2008 had the lowest mean test scores in 2006/07 with an average of −0.92 in Math and −0.96 in Language. Those entering over the following years have gradually better scores, and schools that are not in the program have substantially higher scores with 0.33 in Math and 0.19 in Language. As a preliminary foray into the effect of the SEP program, average test scores are also presented for 2014, six years after the implementation of the policy. Even if disparities between schools based on year of entry into the program remain, the dispersion between schools had declined substantially, with the schools that entered the program earlier experiencing the most gains. The narrowing of disparities across schools is not a simple artifact of declining overall dispersion in test scores, as it can be seen, the test scores’ standard deviation declines only marginally between 2006/07 and 2014. These descriptive findings suggest that the SEP policy may have had a positive effect on test scores, particularly for schools serving disadvantaged populations that were more likely to enroll in the policy early.
Moving to the first research question, Table 3 presents fixed effects models assessing the effect of SEP on fourth-grade Math and Language standardized test scores at the school level among private-subsidized schools. Model 1 includes school and year fixed effects only, and model 2 adds controls for socioeconomic attributes of the school (mean family income, father’s schooling, and mother’s schooling). Figure 2 presents the focal parameter estimates from Table 3, capturing the effect of the SEP program on standardized test scores, along with 95 percent confidence intervals. The main finding from model 1 is that SEP has a positive and significant effect on students’ test scores. On average, schools gain about 0.14 standard deviations in both Math and Language fourth grade scores after taking the SEP. Adding controls for schools’ socioeconomic composition in model 2 alters these parameter estimates minimally, suggesting that change in socioeconomic composition of schools triggered by SEP does not drive the effect of the policy (see Figure 2).
Effect of SEP Program on Fourth-Grade Math and Language Standardized Test Scores: Fixed Effects Models, Private-Voucher Schools in Chile, 2005–2014
p < .10. **p < .05. ***p < .01.
Effect of SEP Program on Fourth-Grade Math and Language Standardized Test Scores, Fixed Effects Models, Private-Voucher Schools in Chile, 2005–2014
Is the effect of the SEP program large? To offer a benchmark, we can compare these effects with the effect of other voucher programs and educational interventions elsewhere. Rouse (1998) estimates that participation in the Milwaukee Parental Choice Program increased Math scores by 0.08 to 0.12 standard deviation each year. Evidence from the Tennessee STAR experiment indicates that cutting class size by one-third increased achievement by roughly 0.2 standard deviations (Krueger 1999; Chetty et al. 2011), while estimated standard deviations of achievement impacts across teachers and schools range from 0.1 to 0.2 (Chetty, Friedman, and Rockoff 2014; Angrist et al., forthcoming). Studies of effective charter schools show annual score gains between 0.2 and 0.4 standard deviations (Abdulkadiroglu et al. 2011; Angrist et al. 2012; Curto and Fryer 2014). For the Chilean case, we can compare the effect of SEP with two major educational interventions. Bellei (2009) finds that the full–school day program increases high school students’ test scores by 0.05 to 0.07 standard deviations in Language and by 0.00 to 0.12 standard deviations in Math. Contreras and Rau (2012) assess the impact of a teachers’ collective incentives program known as the National System of School Performance Assessment (Sistema Nacional de Evaluación del Desempeño, SNED) on students’ academic performance; they find an effect of 0.14 and 0.25 standard deviations in Language and Math test scores, respectively. These figures indicate that the effect size of the SEP program is similar or larger than the effect of major educational interventions in Chile and abroad.
The second research question addresses the socioeconomic heterogeneity of the effect of participating in SEP. As mentioned, the explicit objective of the SEP policy was to favor poor students and reduce achievement gaps. If the additional resources are effective in improving test scores, we should observe that schools at the bottom of the socioeconomic scale should experience larger benefits from the policy. Alternatively, it is possible that schools serving more advantaged students benefit more from the additional SEP resources even without the additional “concentration supplement,” if the socioeconomic advantage of the population they serve translates into stronger management skills, or they are able to hire or retain more capable personnel (e.g., Ladd and Fiske 2009).
To assess economic heterogeneity in the effect of SEP, we created a socioeconomic index based on the school’s average household income, mother’s schooling, and father’s schooling for all private-voucher schools in years 2006/07 (prior to the implementation of the SEP program). These three indicators were combined by means of principal component analysis, and the first component was extracted to create the socioeconomic index. We then divided the index into quintiles, and allowed the effect of SEP on test scores to vary by quintile. The analysis reveals substantial variation in the effect of participating in SEP across socioeconomic quintiles (see Table 4 and Figure 3). For Math, the positive effect reaches about 0.44 standard deviations among the poorest fifth of schools, 0.21 standard deviations for the second poorest quintile, and then becomes not statistically different from zero for private-voucher schools serving more affluent families. The pattern is similar for Language test scores. The positive effect reaches 0.41 standard deviations for schools serving the poorest 20 percent of students, and 0.22 for schools serving students in the next quintile, and is not significantly different from zero among the private-voucher schools serving the remaining 60 percent of the student population.
Effect of SEP Program on Fourth-Grade Math and Language Standardized Test Scores, by School Socioeconomic Quintiles: Fixed Effects Models, Private-Voucher Schools in Chile, 2005–2014
p < .10. **p < .05. ***p < .01.
Effect of SEP Program on Fourth-Grade Math and Language Standardized Test Scores by School Socioeconomic Quintiles, Fixed Effects Models, Private-Voucher Schools in Chile, 2005–2014
These findings show that the policy was particularly beneficial for the schools serving the most disadvantaged students, an effect aligned with the objectives of reducing socioeconomic achievement gaps. It is a concern, however, that no positive effect is found for schools in the top 60 percent of the socioeconomic distribution of private-voucher schools.
The fixed effects model captures an average effect of the policy by comparing the school average test scores across all years after the SEP reform was implemented with the average across all years prior to the policy. However, the effect of the policy is likely to evolve over time, driven by processes such as gaining experience with the new policy environment, learning about how to allocate additional resources, or peer effects of the additional pedagogical support and training received by some teachers at the school level. We estimate a distributed fixed effects model that flexibly captures changes in effect across years since uptake to allow for temporal change in the effects. This model estimates the effect of the policy each year after uptake, using as a baseline for comparison the average test scores in the years prior to the policy.
Results from the distributed fixed effects models are reported in Table 5, and focal parameter estimates along with 95 percent confidence intervals are reported in Figure 4. As can be seen, the effect of the SEP policy increases substantially across years postimplementation for both Math and Language test scores. In the case of Math, the effect is consistently positive across years but insignificant for the first year post-treatment, then reaches 0.08 standard deviations in year two after the school enrolls in the means-tested program, 0.17 standard deviations in year 3, 0.35 in year 4, stabilizing around .40 standard deviations in subsequent years after policy implementation. Results are very similar for Language test scores, with an initial effect of only 0.06 standard deviations the first year, which increases to reach about 0.30 standard deviations in year four, stabilizing thereafter. This pattern of small improvements immediately after policy uptake followed by very large effects after a few years strongly suggests that schools need some time to fully adjust and realize the benefits of the policy.
Effect of SEP Program on Fourth-Grade Math and Language Standardized Test Scores across Years since Enrolling in the Program: Time-Distributed Fixed Effects Models, Private-Voucher Schools in Chile, 2005–2014
p < .10. **p < .05. ***p < .01.
Effect of SEP on Private-Voucher School Average Fourth-Grade Math and Language Test Scores by Year since Enrolling in the SEP Program
Conclusion
Chile implemented a universal school voucher system in the early 1980s. In this system, the government paid schools a flat, per-student subsidy that did not vary by family socioeconomic status. Also, private-voucher schools—but not public-voucher schools—were allowed to select students at will and to charge add-on fees. These institutional features of the Chilean voucher system appear to have contributed to socioeconomic inequalities in educational achievement. In 2008, to improve educational quality and reduce socioeconomic stratification of achievement in the school system, a major policy change transformed the flat voucher system into a means-tested one, adjusting the amount of the voucher by the poverty level of the student and the proportion of poor students attending the school.
This analysis evaluates the impact of this means-tested voucher known as “preferential school voucher” (SEP) on test scores at the school level, how those effects evolved over time, and the heterogeneity across private-voucher schools that served students of different socioeconomic statuses in Chile. To alleviate bias emerging from unobserved heterogeneity among schools, we created a private-voucher school-level panel between 2005 and 2014, and a fixed effects model at the school level and year.
Findings indicate that private-voucher schools gain, on average, about 0.14 standard deviations in fourth-grade Math and Language scores after taking up the SEP policy. This effect size is similar or larger than the effect of major educational interventions in Chile and abroad. Moreover, this result is not affected when controls for schools’ socioeconomic composition are included in the model, suggesting that change in socioeconomic composition of schools does not drive the effect of the policy. Our analysis also reveals substantial variation in the effect of participating in SEP across school’s socioeconomic quintiles. For Math, the positive effect reaches about 0.44 standard deviations among the poorest fifth of schools, 0.21 standard deviations for the second poorest quintile, and then becomes not statistically different from zero for schools serving more affluent populations. The pattern is similar for Language test scores; the positive effect reaches 0.41 standard deviations for schools in the poorest quintile, 0.22 for those in the second poorest, and is not significantly different from zero among the schools serving the remaining 60 percent of the student population. This finding shows that the policy was particularly beneficial for private-voucher schools serving the most disadvantaged students, reducing socioeconomic achievement gaps. Findings from distributed fixed effects also indicate that the impact of the policy increases substantially across years postimplementation for both Math and Language test scores, with insignificant gains the year immediately after implementation and gains that reach about one-third of a standard deviation in both Math and Language scores after four years (a dynamic that might be obscured by methodological approaches that focus on time-averaged effects). This finding suggests that the educational system in general, and schools in particular, require some time to fully adjust to policy reforms and realize the benefits of the policy, and it may be generalizable to many educational policies that require organizational adaptation and mobilization of resources (e.g., Rauscher 2016). These results are consistent with the stated objectives of the reform, and show substantial gains from moving from a flat to a means-tested voucher in terms of overall achievement and equality of educational opportunity. These findings highlight the relevance of considering design when examining the consequences of voucher systems.
It is important to note that the outcome estimated is a treatment-on-the-treated effect, that is, the effect of joining SEP among schools that joined the program. This effect is different from a general-equilibrium approach in which the focus is on the effect of the policy on all students or all disadvantaged students, regardless of whether the school they attended took the program (e.g., Nielson 2013; Murnane et al. 2017). Both kinds of effects are relevant for policy-makers: Our analysis of schools that enrolled in SEP focuses on the gains or losses among the eligible population that took up the program and for whom effects of the policy are expected and intended. Given potential spillover and externality effects, the population-wide effect is also of interest, although its mechanisms are less immediately apparent.
Even if we had found uncontroversially positive treatment effects of a means-tested voucher among schools that took up the policy, it should be emphasized that this policy is not a panacea. First, some of the gains in test scores may have been due to an increase in low-performing students missing the test, as a school’s strategy to respond to the new accountability regime (Quezada-Hofflinger and von Hippel 2017). Second, as suggested by the analysis by Valenzuela, Villarroel, and Villalobos (2013) and Dominguez (2014), making the Chilean universal voucher means-tested has not reduced socioeconomic segregation in the school system. One of the reasons is that schools serving more advantaged populations, and charging higher add-on family contributions, opted out of the SEP policy (see Table 2), thus reducing the choice set of more disadvantaged parents. If policy-makers wish to reduce socioeconomic segregation in the school system, policies prohibiting or substantially restricting the use of add-on funds might also be needed. In January 2015, an inclusion law was approved in the Chilean Congress that gradually eliminates add-on fees. These resources are being replaced by, among others measures, extending the preferential school voucher to middle-class students. Furthermore, given that school segregation is closely patterned to residential segregation—even in a school system that does not restrict enrollment to local catchment areas—addressing segregation requires looking beyond the educational system.
Footnotes
NOTE:
We are grateful to Catalina Canals for excellent research assistance and to Margot Jackson and Susan Moffitt for their thoughtful suggestions. The first author acknowledges financial support from PIA-CONICYT FB0003.
Alejandra Mizala is director of the Center for Advanced Research in Education (CIAE) and a professor with the Center of Applied Economics, Department of Industrial Engineering at the University of Chile. Her research interests include the impact of the Chilean school choice system on educational outcomes, schools’ and teachers’ evaluation, and gender equality in education.
Florencia Torche is a professor of sociology at Stanford University. Her research interests include inequality and mobility, social demography, and sociology of education.