In Search of Value Added in the Case of Complex School Effects

Abstract

In traditional studies on value-added indicators of educational effectiveness, students are usually treated as belonging to those schools where they took their final examination. However, in practice, students sometimes attend multiple schools and therefore it is questionable whether this assumption of belonging to the last school they attended can be made. Furthermore, the schools attended earlier by students might have long-term effects on their subsequent performance. Using data from Dutch primary and secondary schools, multiple membership models and cross classification multilevel models were estimated to explore the effects of student mobility and long-term primary school effects on the estimated value added of secondary schools. Long-term effects of primary schools did not change the estimated value added of secondary schools. On the other hand, allowing students to be a member of multiple secondary schools changed the estimated effectiveness of these schools especially for schools in the middle range of effectiveness.

Keywords

school effectiveness value added secondary education student mobility multiple membership multiple classification (MMMC) models multilevel chained equations imputation

Introduction

Value-added indicators of school effectiveness are increasingly used in educational accountability systems to compare schools in their effects on students’ achievement. Value-added indicators promise to distinguish the schools’ effectiveness more fairly by making a proper correction for differences in intake of students. Value added can be defined as “an indication of the extent to which any given school has fostered the progress of all students in a range of subjects during a particular time period in comparison to the effects of other schools in the same sample” (Sammons, Thomas, & Mortimore, 1997, p. 24). Usually, the students’ prior educational achievement, student background characteristics, and compositional variables of the student population are used as covariates in statistical models to account for differences in intake of students between secondary schools in order to achieve a fair comparison. The statistical techniques to estimate these value-added indicators have been developed since the 1980s (Aitkin & Longford, 1986; Goldstein, 1997; Hill & Rowe, 1996, 1998; Raudenbush & Bryk, 1986; Raudenbush & Willms, 1995; Willms & Raudenbush, 1989). The assumption behind these value-added indicators is that the effectiveness of schools can be considered as a latent trait, which can be measured through the performance of students within schools, just as estimating a latent trait in students can be achieved through a careful analysis of items of a test.

Making a fair and valid comparison between the effectiveness of schools could also mean a control for the effects of previously attended schools, besides the usual covariates in the analysis of value added. Student mobility between secondary schools and long-term effects of primary schools are examples of previously attended education that might influence a students’ performance during secondary education. Student mobility across schools and long-term effects of primary schools traditionally are not incorporated in the estimation of value added of secondary schools. In the traditional value-added models, the students are treated as belonging to just one secondary school; the school where they took their final examination. Therefore, estimated value added derived from these traditional models might include influences of previously attended schools. Ignoring effects of previous education in the estimation of value added might lead to bias in the estimation of the latent school effectiveness trait. Further developments of methods and software made it possible to refine the value-added indicators of school effectiveness and to allow for modeling educational data with imperfect hierarchical structures (Browne, Goldstein, & Rasbash, 2001; Fielding & Goldstein, 2006; Hill & Goldstein, 1998).

The association of student mobility (Engec, 2006; Strand & Demie, 2007) and long-term effects of primary schools (Goldstein & Sammons, 1997; Pustjens, Van de Gaer, Van Damme, Onghena, & Van Landeghem, 2007) with student achievement in secondary education has been studied before. However, research on the effects of these phenomena on the estimation of school effects in secondary education is rare (Leckie, 2009). The present study investigates the degree to which model specifications with respect to student mobility and long-term primary school effects influence estimates of secondary school effects by means of multiple classification and multiple membership multilevel modeling (Browne et al., 2001; Fielding & Goldstein, 2006). The validity of the estimated value added of secondary schools is our main interest of this study rather than the phenomena of student mobility and long-term primary school effects themselves. Modeling these imperfect hierarchical structures might have important consequences for the development of value-added indicators for purposes of educational accountability. The following research question will be addressed in the remainder of this study. To what extent does modeling student mobility and long-term primary school effects lead to differences in estimated value-added indices in secondary education?

The following sections will give a brief overview of literature concerning effects of student mobility and long-term primary school effects on student achievement during secondary education and the relation with the estimation of value added for secondary schools. In this section, a brief overview will be given of the methodology and results from previous studies. After that, the data, variables, and analytical strategy are described, including a short description of Dutch secondary education. Data from schools in Dutch secondary education will be used as an example for estimating value-added indicators while controlling for student mobility and long-term primary school effects. In the final section, the empirical results will be presented and discussed.

Multiple Membership Models and Student Mobility

Student mobility, also known as student transfer or school mobility, is defined as the movement of students between schools or educational institutions, once or multiple times, at other times than the normal age at which the students start or finish their education at a school (Strand & Demie, 2007). In this definition, the change from a primary school to a secondary school is not a part of student mobility. Student mobility is known for its negative effects on student performance both on reading and mathematics (Engec, 2006; Hattie, 2009; Mehana & Reynolds, 2004; Strand & Demie, 2007; Temple & Reynolds, 1999).

The mobility of students between schools is mentioned as one of the many problems in estimating value added (Goldstein, 2001; Keeves, Hungi, & Afrassa, 2005; Roeleveld, 2003). In traditional value-added indicators of school effectiveness, the students are treated as belonging to the school where they take their final examination. In these traditional value-added models, all effects of previously attended schools are attributed to the final school regardless of whether the student changed schools. The regression formula for the hierarchical model for estimating traditional value-added indicators in which student i (Level 1) is nested within the final secondary school j (Level 2) is given below in Formula (1). In this formula, Y_ij is the dependent variable; usually, test or examination scores at the end of a formal stage of education are used in the estimation of value-added. The average performance of students in the sample is represented by the intercept, γ₀. Usually, a set of control variables is included to explain parts of the variability in Y_ij at the student and school level. Control variables are given by γ_hx_hij in the formula. In such models, the residuals at the school level (U_0j) and the student level (R_ij) are assumed to be independent and with a population mean of 0 and a constant variance. The assumption behind this traditional value-added model is that after a careful correction of differences between schools in their intake of students, the remaining between-school variance reflects differences between schools in effectiveness. The residual at the level of the secondary school is then considered to be the estimate of a school’s value added.

Y_{i j} = γ_{0} + \sum_{h = 1}^{q} γ_{h} x_{h i j} + U_{0 j} + R_{i j} .

First attempts were undertaken by Goldstein, Burgess, and McConnell (2007) and Leckie (2009) to model student mobility in the estimation of value-added indicators through the use of multiple membership multilevel models. In multiple membership models, students (Level 1) can be nested in multiple schools for secondary education (Level 2; Browne et al., 2001). The effects of the multiple secondary schools on the students’ progress can be weighted (w_ih), for example, by the time the student attended the school. In Formula (2), the multiple membership model is presented. In the notation chosen here, unlike in usual multilevel models, the index i, denoting the student, is not regarded as being nested in some higher level unit, so that the values of i may range from 1 to the total number of students in the data set. In a traditional multilevel model with students strictly nested in schools, for each student there is only one school h with w_ih = 1, and for all others w_ih = 0. In this case, the secondary school last attended will get the full weight, whereas all other schools get zero weight. In multiple membership models, this is not the case, but in practice still most values of w_ih will be 0.

Y_{i} = γ_{0} + \sum_{h = 1}^{q} γ_{h} x_{h i} + \sum_{h = 1}^{N} w_{i h} U_{0 h} + R_{i} .

In a study in British primary education, Key Stage 2 (comparable to third until sixth grade in the United States), a multiple membership cross-classified model was used for the combined analysis of the effects of student mobility and primary school attended (Goldstein et al., 2007). In a comparison a correlation of .98 was found between a model that took account of student mobility and prior education and the traditional value-added model. Goldstein et al. (2007) therefore suggest that ignoring student mobility and effects of earlier education does not appear to alter the rankings of schools on their posterior value-added estimates. However, they also show that ignoring student mobility leads to a downward bias of the estimated variance at the school level.

In a study in British secondary education, between Key Stage 2 and General Certificate of Secondary Education (GCSE, comparable to 7th until 11th grade in the United States), the effects of both student mobility and long-term primary school effects on the estimation of value added of secondary schools are investigated (Leckie, 2009). In this study, primary schools were included as crossed random effects. Similar to the findings of Goldstein et al. (2007), Leckie showed that the schools appear to be more important if student mobility is modeled through multiple membership models. In other words, ignoring student mobility leads to an underestimation of the between–secondary school variance.

Multiple Classification Models and Long-Term Effects of Previous Education

Long-term primary school effects on the learning progress of students during secondary education can be analyzed through the use of multiple classification models. In multiple classification models students can be nested in groups on more than one dimension (Fielding & Goldstein, 2006). In these kinds of models, students are both nested within secondary schools and within a primary school. The random factors secondary schools and primary schools can both be seen as populations of interest. Primary and secondary schools are called crossed random factors, because not all students from the same primary school attend the same secondary and not all students from the same secondary school attended the same primary school.

In Formula (3), a multiple classification model is presented for students i, nested within secondary schools j, and also nested within primary schools k. Secondary schools and primary schools are crossed random factors in this type of model. Compared with the traditional value-added model in Formula (1), the random effect of the primary school, indicated by W_0k, is just added to the formula. The usual assumption made is that the primary school effects are independent of the other random effects. The interpretation of the primary school effects is similar to other random effects, namely as representing the part of the variability in the dependent variable that is accounted for by primary schools. Similar to the previous models, covariates can be included in the model to control for differences in intake of students between secondary schools. After the inclusion of covariates indicating the performance or ability of students at the end of primary education, the residual between–primary school variance is assumed to reflect long-term effects of primary schools.

Y_{i (j, k)} = γ_{0} + \sum_{h = 1}^{q} γ_{h} x_{hij} + U_{0 j} + W_{0 k} + R_{ij} .

The small body of literature concerning long-term effects of previous education shows consistent small long-term effects of previous schools on the subsequent performance of students (Goldstein & Sammons, 1997; Sammons, Nuttall, Cuttance, & Thomas, 1995; Tymms, 1995; Tymms, Merrell, & Henderson, 2000). However, results concerning the persistence of primary school effects during secondary education are inconsistent (Bressoux & Bianco, 2004; Creemers, Kyriakides, & Sammons, 2010). Differences between the results of the studies might arise from methodological differences between studies with respect to the inclusion of the teacher level or the department level and the period over which the long-term primary school effect is measured.

Small effects of primary schools, persisting during the entire duration of secondary education were found in British secondary education (Goldstein & Sammons, 1997; Sammons et al., 1995). Small long-term effects of primary schools on student achievement in secondary education were found in Flanders (Pustjens et al., 2007; Snijders & Bosker, 1999). However, the small long-term effects of primary schools on performance of students in secondary education in Flanders decreased rapidly during the first years of secondary education (Pustjens et al., 2007). The inclusion of schools for primary education as a crossed random factor in a multiple classification model for the analysis of the effectiveness of secondary schools led to a great reduction in the estimated between secondary school variance in British secondary education (Goldstein & Sammons, 1997). Students from effective primary schools also tend to do well at the end of secondary school. Serious distortions of the results of estimated value added might appear when no adjustments are made for previous education in the estimation of value added in secondary education. Because of these long-term primary school effects, it was suggested that adjustments should be made not only for prior achievement but also for all previous education to find a better estimation of both short- and long-term school effects (Kyriakides & Creemers, 2008).

Method

Subjects

The data used here were collected as part of a national longitudinal study in secondary education in the Netherlands, the “Cohort Studies in Secondary Education” (Dutch abbreviation: VOCL). The study concerned students who entered the first grade of Dutch secondary education in the Netherlands (comparable to the seventh grade in the United States) in the year 1999, also called the VOCL’99 cohort. The total cohort consists of a sample of approximately 20,000 students. This sample has been considered as representative of the schools and students in the Dutch secondary education (Kuyper & Van der Werf, 2003b). The data in the VOCL’99 cohort were derived from several sources and on several occasions (Kuyper & Van der Werf, 2003a).

For the current study, we selected a subsample from the VOCL’99 cohort based on the following criteria: Identification variables had to be available at the student level, secondary school level, the primary school attended, and the students took their final national examinations in the prevocational secondary education theoretical track (VMBO tl). Furthermore, as a result of data requirements for a correct estimation of the multiple membership models, only those mobile students were included for whom examination results were available for the delivering and receiving secondary school. Both students who finished secondary education in the nominal time (4 years) and students who lagged behind one year (5 years) were included in the sample. The Dutch secondary education system consists of multiple differentiated school tracks, for which the students are selected at age 12 years on the basis of their scholastic aptitude. The VMBO theoretical program is one of the 4-year vocational programs preparing students for senior secondary vocational education. The subsample consists of 3,658 students in 185 secondary schools. Because of student mobility, the number of secondary schools is more than the original sample of 100 secondary schools. These students stem from 1,292 different primary schools. The number of feeder primary schools for one secondary school ranged from 1 to 75 with a mean of 9 schools.

In this subsample, only student mobility within the VMBO tl track was allowed. Within the VMBO tl track, 94% of the students (3,438) took their final national examination in the same school where they started their school career in secondary education. The remaining 6% of students was mobile during secondary education at least once. Of this group, 213 students attended two secondary schools and 5 students attended three secondary schools. These five students who attended three schools all changed schools after the first year in secondary education.

Variables

Outcome variable

The overall mean score on the final national examination was used as the dependent variable in the multilevel regression analysis, which ranges between 3 and 9. For the majority of students, who finished secondary education in the nominal time, results from the final nation examination in spring 2003 were used. For the students who lagged behind one year, the scores on the final national examinations of spring 2004 were used.

Explanatory variables

Halfway through the seventh grade, the “Cito-entry” test took place. This test was developed by CITO, the Netherlands Institute for Educational Measurement. This test contains the following parts: Dutch language, mathematics, and information processing. For our study, the total score on the test was used as an overall measure of prior achievement. The total test had a reliability (Cronbach’s α) of .90 (Kuyper & Van der Werf, 2003b), and the range of scores on this test was between 13 and 60 points.

Another indication of students’ prior scholastic aptitude is given by the advice that primary school teachers provide to parents at the end of primary education. The advice is stated in terms of the most appropriate school track for the student in secondary education. The advice consists of nine categories and can range between a more individualistic track in prevocational secondary education and the preuniversity school track. In the analysis the advice of the primary school teachers is used as a continuous variable.

Socioeconomic status was measured by the highest educational level completed by one or both of the student’s parents. This variable consisted of six categories (coded as 2 to 7), ranging from only primary to postgraduate education. In the analysis, socioeconomic status was used as a continuous variable. Descriptive statistics of the covariates are presented in Table 1.

Table 1.

Descriptive Statistics of Variables Used for the Estimation of Value Added Models.

	Average	SD	N	Percentage
Dependent variable
Final achievement	6.40	0.67	3,658
Explanatory variables
Prior achievement	35.45	7.37	3,469
Advice from primary school teacher	4.03	0.99	3,262
Socioeconomic status	5.41	1.22	3,459
Ethnicity (native)				84.5
Ethnicity (minority)				15.5

Information about the ethnic origin of students was gathered by asking the parents in which country they were born. Students’ ethnicity was operationalized as a dichotomous variable with the categories native (coded as 0) and minority (coded as 1). Only if both parents and the student were born in the Netherlands was the student considered to be indigenous; in all other cases, the student was considered to be a minority student.

Methods of Analysis

Multiple imputation through multilevel chained equations

The VOCL’99 cohort contained many predictors of the students’ final achievement. For almost all predictor variables scores were missing for some pupils. The often used method of listwise deletion of cases with missing values is wasteful of information and can lead to biases in results (Graham, 2009) and therefore we employed the multilevel chained equations technique (Snijders & Bosker, 2012; Van Buuren, 2011) using all available information.

In total, there were 2,088 complete cases and 1,570 pupils with one or more missing values on the predictor variables. The missingness is mostly not strongly associated between variables. First, an initial random imputation was done to obtain a first complete data set, based on a suboptimal but reasonable imputation in which the multilevel structure was ignored. The continuous variables, prior achievement, age, intelligence, advice, and socioeconomic status were randomly imputed based on a multivariate normal distribution jointly with the completely observed measure of final achievement. The variables second language, gender, age, intelligence, and living in a problem neighborhood functioned as auxiliary variables for the imputation. For the dependent variables and the predictor variables with missing values, the main relations were investigated using the provisionally imputed data set using multilevel analysis. This led to the imputation models, using the following rules: (a) significant variables and group means of significant variables were included, (b) if X-mean was a significant predictor for Y, then Y-mean was included as predictor of X, (c) implausible predictors were dropped, and (d) unimportant predictors were dropped for binary dependent variables to improve convergence. We constructed 25 data sets with imputed values. Results reported in the following tables are the syntheses of 25 analyses run on these imputed data sets. For parameter estimates and standard errors, the combination rules of Rubin (1987) were used. The imputation uncertainty between imputed data sets appeared small because the estimated coefficients of the control variables hardly differed from each other, when the 25 data sets are considered. The missing fractions range between .014 (ethnicity) and .102 (advice), which indicates that at most 10.2% of the information in a variable was lost because of the missingness. Deviances reported are averages across the 25 imputed data sets, and because of the Bayesian estimation method and the imputations, the deviance differences are to be used cautiously as indications of relative model fit.

Multiple membership and multiple classification multilevel models

Besides the traditional value-added analysis used in school effects studies, three alternative models will be analyzed in which deviations from the strict hierarchical structure are allowed (Snijders & Bosker, 1999). Prior achievement, socioeconomic status, advice, and ethnicity are included as covariates in all value-added models. In the second model, the effects of student mobility are included in the analysis using a multiple membership model. In this multiple membership model, the weights given to each school are based on the proportion time spent in each school. The weight is equal to 1 for all 3,438 students who did not change schools during secondary education. For the remaining students the nonzero weights for individual schools vary between 0.2 and 0.8. The total weight for each student is 1. An overview of the mobility and weights in the sample is presented in Table 2.

Table 2.

Overview of Mobility and Weights for Secondary Schools.

Weights per secondary school	Number of students	Percentages of students
1	3,438	94.00
0.2, 0.8	31	0.85
0.25, 0.75	17	0.46
0.40, 0.60	15	0.41
0.50, 0.50	49	1.34
0.60, 0.40	26	0.71
0.75, 0.25	48	1.31
0.25, 0.50, 0.25	1	0.03
0.40, 0.40, 0.20	2	0.05
0.50, 0.25, 0.25	1	0.03
0.60, 0.20, 0.20	1	0.03

The third model simultaneously analyzes the effects of primary schools on students during secondary education, through a multiple classification multilevel model (Hill & Goldstein, 1998). The final model will take both effects of primary schools and student mobility into account when estimating the value added of secondary schools, by means of a multiple membership multiple classification model (Fielding & Goldstein, 2006).

The estimation of multiple classified and multiple membership models runs into important computational limitations in existing maximum likelihood approaches (Browne et al., 2001). All of the models in this study are therefore estimated using Markov chain Monte Carlo based algorithms from the MLwiN 2.22 software package for multilevel modeling (Browne, 2009; Rasbash, Steele, Browne, & Goldstein, 2009). Starting values for the fixed parameters are estimated from simpler models using a maximum likelihood approach in MLwiN. In these models, grand mean centering was applied for all continuous covariates.

Results

Modeling Value-Added Estimates of School Effectiveness

In Table 3, the results of the empty models are presented for all types of value-added models. From the traditional value-added model, it can be seen that the average examination score is 6.36 and the total variance 0.467. Of this variance, 14% is associated with the secondary schools. Intraclass correlations of similar magnitudes were found in previous studies in Dutch secondary education (Luyten, 1998; Veenstra, 1999).

Table 3.

Results From Empty Models.

	Traditional model		MM model		MC model		MMMC model
	Par.	SE	Par.	SE	Par.	SE	Par.	SE
Fixed effects
Constant	6.36	0.03	6.360	0.03	6.36	0.03	6.36	0.03
Random effects
Crossed random effect
Primary schools					0.006	0.005	0.006	0.005
Level 2 random effect
Secondary schools	0.067	0.014	0.064	0.014	0.066	0.014	0.065	0.014
Level 1 variance
Students	0.400	0.010	0.401	0.009	0.395	0.010	0.395	0.010
Model fit
Deviance	7025.7		7035.0		6977.4		6984.1

Note. MM = multiple membership model for modeling student mobility; MC = multiple classification model for including primary schools as a crossed random factor; MMMC = multiple membership multiple classification model for modeling student mobility and primary schools simultaneously; Par. = parameter estimate.

In the multiple membership model, in which students are allowed to be a member of multiple secondary schools, there is a marginal decrease in the between-school variance, going down from 0.067 to 0.064, and an increase in the deviance with 10 points. This indicates that the multiple membership model does not seem to get meaningfully closer to the data than the traditional multilevel model.

The results in which the available information on the primary schools previously attended by the pupils is taken into account are presented in the multiple classification model. Of course, the average examination grade remains the same, but now we see some small changes in the variance components. The variance between secondary schools marginally decreases to 0.066, and the within-school variance decreases somewhat as well, as now the primary schools take up a variance component of 0.006. The decrease in deviance is 7025.7 − 6977.4 = 48.3, highly significant in a chi-squared distribution with df = 1. However, the covariates such as prior achievement (start secondary education or end of primary education) are not yet included in these models and therefore, the variance on the secondary school level cannot be seen as representing the net between school differences but rather represents the gross secondary school effects.

The results of the multiple membership multiple classification model, in which student mobility and long-term primary school effects are estimated simultaneously are not very different from those of the multiple classification model. The deviance of this empty multiple membership multiple classification model is slightly higher than for the multiple classification model. This is possible because of the Markov chain Monte Carlo algorithm, suggesting that the model may have converged incompletely, and that this model, being more complicated, is harder to estimate than the earlier estimated models.

For the results presented in Table 4, the predictor variables prior achievement, socioeconomic status, advice, and ethnicity were included in the models to estimate value added. The four predictor variables all have highly significant effects, indicating that pupils with higher entry test scores, with higher recommendations from their primary school teachers, and from more affluent families have higher average examination scores. Moreover, pupils from ethnic minorities have lower examination results than pupils from the Dutch majority group. The results of these fixed effects are consistent over the four value-added models.

Table 4.

Results From the Multiple Multilevel Models for Estimating Value Added.

	Traditional value added model		MM value added model		MC value added model		MMMC value added model
	Par.	SE	Par.	SE	Par.	SE	Par.	SE
Fixed effects
Constant	6.39	0.02	6.40	0.02	6.39	0.02	6.40	0.02
Prior achievement	0.032	0.002	0.032	0.002	0.031	0.002	0.031	0.002
Socioeconomic status	0.068	0.010	0.068	0.010	0.068	0.011	0.068	0.011
Advice	0.054	0.010	0.053	0.010	0.054	0.010	0.053	0.010
Ethnicity	−0.071	0.028	−0.072	0.028	−0.071	0.029	−0.073	0.028
Random effects
Crossed random effect
Primary schools					0.003	0.003	0.003	0.003
Level 2 random effect
Secondary schools	0.034	0.008	0.033	0.007	0.033	0.008	0.033	0.007
Level 1 variance
Students	0.333	0.008	0.334	0.008	0.330	0.008	0.331	0.008
Model fit
Deviance	6350.0		6366.1		6326.2		6338.6

Most important, however, are the estimates of the variance components. Comparing the models with and without predictor variables, all variance components have decreased because of the inclusion of the predictor variables. For the multiple classification model, the between-pupils within schools variance decreases from 0.395 to 0.330. The between–secondary school variance (0.034) is almost half its original estimate (0.066), which also turns out to be the case for the between–primary school variance: from 0.006 this decreases to 0.003. The remaining between–secondary school variance indicates that secondary schools do appear to have a value-added effect on pupil achievement measured at the final examination. But primary schools, given the achievement levels attained by pupils at the end of primary education and given their family background, have only a marginally lasting effect as measured 4 or 5 years later at the secondary school examinations. However, the decrease in deviance between the traditional value-added model and the multiple classification value-added model, 6350.0 − 6326.1 = 23.9, is still highly significant in a chi-squared distribution with df = 1. The effects of the multiple membership modeling of student mobility on the coefficients of the model are even smaller after the inclusion of predictor variables.

Comparing the Traditional Value-Added Model with the Multiple Classification and Multiple Membership Models

The specific aim of this study was to investigate whether modeling the effects of the various imperfect hierarchical structures would affect the estimated value added of secondary schools. Correlations among residuals, value-added scores, represented by secondary schools for the various models are presented in Table 5. Despite the very small but significant long-term effects of primary schools, the inclusion of the multiple classification in the model does not appear to change the estimated value added of secondary schools. When the ranks of secondary schools according to their value added are considered, the inclusion of primary schools as a crossed random factor leads to a maximum shift in ranks of 10 places in rank order compared with a traditional value-added model.

Table 5.

Correlations Between School-Level Residuals From the Various Multilevel Models for Estimating Value Added.

	Traditional model	MM model	MC model
MM model	.880*
MC model	1.00*	.879*
MMMC model	.881*	1.00*	.880*

p < .001

The results are somewhat different for the multiple membership model, in which students are allowed to be a member of multiple secondary schools. A correlation of .88 was found between the estimated value added of secondary schools in a traditional model and in a multiple membership model. A scatterplot of the estimated value added in a traditional model and a multiple membership models is presented in Figure 1. From this scatterplot, one can see that especially the schools in the middle range differ with respect to their value added for both models. The estimated value added of the most and least effective schools in the sample is relatively stable over both models. If schools were compared in ranks for the traditional and multiple membership model, more than 50% of the schools change 10 places on the rank order or more.

Conclusion and Discussion

The main focus of this study was to investigate the degree to which model specifications with respect to student mobility during secondary education and long-term effects of primary schools influence the estimation of value added for secondary schools. Traditional studies in school effectiveness research and several educational accountability systems apply multilevel models in which the students are strictly nested within schools. However, there is some evidence of long-term effects of previously attended schools (Pustjens et al., 2007) and students may attend more than a single school during a formal period of schooling. These long-term effects and student mobility might bias the estimated value added of secondary schools if they are ignored in the analyses.

Value-added indicators, which are frequently used in educational accountability systems, should be valid but not unnecessarily complicated, for the reason that the indicator should be as transparent as possible. Only if the complex modeling of student mobility and effects of attended primary schools have important effects on the estimated value added of secondary schools, these models should be applied in educational accountability systems. Otherwise, if these complex models do not alter the estimated value added significantly, more simple models are preferable.

In the current study, we found very small but significant long-term effects of primary schools on the performance of students on their final examination in secondary education. Based on previous literature, these small effects are not surprising. Even though there appeared to be very small lasting primary school effects, the inclusion of the multiple classification of primary schools with secondary schools did not alter the estimated value added of secondary schools much. These results are in contrast to the findings of Leckie (2009) that showed that including long-term effects of primary schools in the analysis of value added of secondary schools did change the estimated secondary school effects.

Allowing students to be a member of multiple secondary schools, however, did appear to have an effect on the estimated value added of secondary schools. A strong, positive correlation was found between a traditional value-added model and a multiple membership model. However, more than 50% of the schools change more than 10 places in the rank order. Differences between the estimated value added of the traditional model and the multiple membership model imply that student mobility should be included in the analysis. Similar conclusions were drawn based on a study in British secondary education (Leckie, 2009). In this current study, the inclusion of multiple membership in the model changes the estimated value added especially for secondary schools in the middle range. Estimated value added for the most and least effective schools seemed rather stable over the models. Most educational accountability systems are designed to identify potential underperforming schools. The relative stable effects of value-added estimates for the weakest schools over different models imply that traditional value-added models seem sufficient in identifying underperformance.

A number of limitations of the data and the models applied in this study should be considered when interpreting the results. In the first place, only data from one of the tracks in a differentiated educational system is used in this study for the analyses of long-term primary school effects and student mobility. This can be regarded as a relatively homogeneous population. The small differences between the various value-added models might partly be due to this relatively homogeneous character of the sample. Furthermore, results from one track cannot easily be generalized to other tracks, as tracks differ in length, content, level, and possibilities for between-track mobility. The effects of long-term primary school effects and student mobility on the estimated value added of secondary schools might depend heavily on these track characteristics.

Second, only student mobility within the school track was estimated in this study because of data requirements for estimating multiple membership models. Examination results had to be available for both the delivering and receiving school. Especially in strongly differentiated educational systems, such as Dutch secondary education, where not all schools provide education in all tracks, there can be considerable mobility between tracks. In such a differentiated educational system, the multiple membership models only partly resolve the student mobility problem on the estimation of secondary school effects because of the data requirements and the subsequent underestimation of student mobility in differentiated educational systems.

Furthermore, in multiple membership models lower level units are allowed to be a member of multiple units at the higher level, in this case a student can be a member of multiple secondary schools. The multiple membership models however cannot account for the order in which the students attended the secondary schools, which might cause bias in the estimated value added. The effects of a secondary school, in the case of student mobility, might be passed on to the subsequent school. Alternative weighting options in multiple membership models can be explored to assess the impact of ordering of schools on student performance, a combination of time spent in school and order of schools might be considered. In a previous study on British secondary education, the time spent in schools as weighting for secondary schools showed the best fit with the data (Leckie, 2009). Furthermore, the data used in this study did not allow for a very precise determination of the weights for the multiple membership models. Which school the student attended was only registered once every year. Therefore, we might miss some schools if students attended them very briefly, and we might misestimate the time spent in schools by students, because we have only one measurement per year. Despite these imprecise measurements of time spent in schools, the study nevertheless shed some light on the effects of student mobility on the estimation of value-added indicators for secondary schools.

For future research, we suggest to assess the effects of student mobility and long-term effects of primary schools on the estimation of secondary school effects on larger data sets, for example, national student databases. Furthermore, the effects of primary schools might also be investigated by including the average final achievement per primary school as a predictor in the analyses of value added for secondary schools since this is an indicator of observed quality.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Aitkin

Longford

(1986). Statistical modelling issues in school effectiveness studies. Royal Statistical Society, 149, 1-43.

Bressoux

Bianco

(2004). Long-term teacher effects on pupils’ learning gains. Oxford Review of Education, 30, 327-345.

Browne

W. J.

(2009). MCMC estimation in MLwiN Version 2.13. Bristol, England: Centre for Multilevel Modelling, University of Bristol.

Browne

W. J.

Goldstein

Rasbash

(2001). Multiple membership multiple classification (MMMC) models. Statistical Modelling, 1, 103-124.

Creemers

Kyriakides

Sammons

(2010). Methodological advances in educational effectiveness research: Quantitative methodology series. New York, NY: Routledge.

Engec

(2006). Relationship between mobility of students and student performance and behavior. Journal of Educational Research, 99, 167-178.

Fielding

Goldstein

(2006). Cross-classified and multiple membership structures in multilevel models: An introduction and review. Nottingham, England: Department for Education and Skills.

Goldstein

(1997). Methods in school effectiveness research. School Effectiveness and School Improvement, 8, 369-395.

Goldstein

(2001). Using pupil performance data for judging schools and teachers: Scope and limitations. British Educational Research Journal, 27, 433-442.

10.

Goldstein

Burgess

McConnell

(2007). Modelling the effect of pupil mobility on school differences in educational achievement. Journal of the Royal Statistical Society A, 170, 941-954.

11.

Goldstein

Sammons

(1997). The influence of secondary and junior schools on sixteen year examination performance: A cross-classified multilevel analysis. School Effectiveness and School Improvement, 8, 219-230.

12.

Graham

J. W.

(2009). Missing data analysis: Making it in the real world. Annual Review of Psychology, 60, 549-576.

13.

Hattie

J. A. C.

(2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.

14.

Hill

P. W.

Goldstein

(1998). Multilevel modeling of educational data with cross-classification and missing identification for units. Journal of Educational and Behavioral Statistics, 23, 117-128.

15.

Hill

P. W.

Rowe

K. J.

(1996). Multilevel modelling in school effectiveness research. School Effectiveness and School Improvement, 7, 1-34.

16.

Hill

P. W.

Rowe

K. J.

(1998). Modelling student progress in studies of educational effectiveness. School Effectiveness and School Improvement, 9, 310-333.

17.

Keeves

J. P.

Hungi

Afrassa

(2005). Measuring value added effect across schools: Should schools be compared in performance? Studies in Educational Evaluation, 31, 247-266.

18.

Kuyper

Van der Werf

M. P. C.

(2003a). VOCL’99-1: Technisch rapport [VOCL’99-1: Technical report]. Groningen, Netherlands: GION.

19.

Kuyper

Van der Werf

M. P. C.

(2003b). VOCL’99: de resultaten van het eerste leerjaar [VOCL’99: Results of the first grade]. Groningen, Netherlands: GION.

20.

Kyriakides

Creemers

(2008). A longitudinal study on the stability over time of school and teacher effects on student outcomes. Oxford Review of Education, 34, 521-545.

21.

Leckie

(2009). The complexity of school and neighbourhood effects and movements of pupils on school differences in models of educational achievement. Journal of the Royal Statistical Society A, 172, 537-554.

22.

Luyten

(1998). School effectiveness and student achievement consistent across subjects? Evidence from Dutch elementary and secondary education. Educational Research and Evaluation, 4, 281-306.

23.

Mehana

Reynolds

A. J.

(2004). School mobility and achievement: A meta-analysis. Children and Youth Services Review, 26, 93-119.

24.

Pustjens

Van

Gaer

Van Damme

Onghena

Van Landeghem

(2007). The short-term and the long-term effect of primary schools and classes on mathematics and language scores. British Educational Research Journal, 33, 419-440.

25.

Rasbash

Steele

Browne

W. J.

Goldstein

(2009). A user’s guide to MLwiN. Bristol, England: Centre for Multilevel Modelling, University of Bristol.

26.

Raudenbush

S. W.

Bryk

A. S.

(1986). A hierarchical model for studying school effects. Sociology of Education, 59, 1-17.

27.

Raudenbush

S. W.

Willms

J. D.

(1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307-335.

28.

Roeleveld

(2003). Herkomstkenmerken en begintoets: Secundaire analyses op het PRIMA-cohortonderzoek [Background characteristics and prior achievement: Secondary analysis on the PRIMA-cohort studies]. Amsterdam, Netherlands: SCO-Kohnstamm Instituut.

29.

Rubin

D. R.

(1987). Multiple imputation for nonresponse in surveys. New York: John Wiley and Sons.

30.

Sammons

Nuttall

Cuttance

Thomas

(1995). Continuity of school effects: A longitudinal analysis of primary and secondary school effects on GCSE performance. School Effectiveness and School Improvement, 6, 285-307.

31.

Sammons

Thomas

Mortimore

(1997). Forging links: Effective schools and effective departments. London, England: Paul Chapman.

32.

Snijders

T. A. B.

Bosker

R. J.

(1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling (1st ed.). London, England: Sage.

33.

Snijders

T. A. B.

Bosker

R. J.

(2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). Thousand Oaks, CA: Sage.

34.

Strand

Demie

(2007). Pupil mobility, attainment and progress in secondary school. Educational Studies, 33, 313-331.

35.

Temple

J. A.

Reynolds

A. J.

(1999). School mobility and achievement: longitudinal findings from an urban cohort. Journal of School Psychology, 37, 355-377.

36.

Tymms

(1995). The long-term impact of schooling. Evaluation and Research in Education, 9, 99-108.

37.

Tymms

Merrell

Henderson

(2000). Baseline assessment and progress during the first three years at school. Educational Research and Evaluation, 6, 105-129.

38.

Van Buuren

(2011). Multiple imputation of multilevel data. In Hox

J. J.

Roberts

J. K.

(Eds.), Handbook of advanced multilevel analysis (pp. 173-196). New York, NY: Routledge.

39.

Veenstra

(1999). Leerlingen-klassen-scholen: Prestaties en vorderingen van leerlingen in het voortgezet onderwijs [Students–classes–schools: Achievement and progress of students in secondary education]. Groningen, Netherlands: ICS (Interuniversity Centre for Social Science Theory and Methodology).

40.

Willms

J. D.

Raudenbush

S. W.

(1989). A longitudinal hierarchical linear model for estimating school effects and their stability. Journal of Educational Measurement, 26, 209-232.