Abstract
This article is a summary of a report prepared for the University Council for Educational Administration Program Improvement Project for the Wallace Foundation. This explores the research base for educational leadership preparation programs, specifically examining literature on program features. The review covers context, candidates, faculty, curriculum, design, delivery, pedagogy, internships, student assessment, mentoring and coaching, comprehensive leadership development, and program evaluation. In addition to summarizing the major findings in these program feature areas, the article provides a critical evaluation of the substantive and methodological gaps and future research directions.
Keywords
Introduction
This article is a summary of the report Program Features of Educational Leadership Preparation: An Update of What We Know (Crow & Whiteman, 2013), prepared for the UCEA Program Improvement Project for the Wallace Foundation. 1 That report is an exploration of the research base for educational leadership preparation programs, and itself is an addendum to the Handbook of Research on the Education of School Leaders (M. D. Young, Crow, Murphy, & Ogawa, 2009). In this article, we review the research base on features of leadership preparation programs.
A review of the research base can help the field reach a consensus about preparing school leaders for the current policy context (Hallinger, 2013). As Darling-Hammond, Meyerson, LaPointe, and Orr (2010) noted, there is a great deal of variation in preparation programs. That variation is not inherently problematic; the literature demonstrates that tailoring preparation programs to meet the local needs of school districts may be critical to those districts’ continued success. However, variation without intention or variation generated within the gaps of what we do not yet know about effective leadership preparation may be problematic. Indeed, as will be shown, there is still a great deal we do not know about the design and effectiveness of preparation programs.
Hallinger (2013) recommended a conceptual framework for conducting systematic literature reviews. The questions forming this framework are the following:
What are the central topics of interest, guiding questions, and goals?
What conceptual perspective guides the review’s selection, evaluation, and interpretation of the studies?
What are the sources and types of data employed for the review?
How are data evaluated, analyzed and synthesized in the review?
What are the major results, limitations, and implications of the review? (Hallinger, 2013, p. 130)
We will address this framework throughout the review, starting with the review’s structure and how it relates to the topics of interest, guiding questions, and goals.
Our purpose in this review was to examine the research base on preparation program features, particularly in relation to the individual program feature reviews contained in the 2009 Handbook. We were interested in knowing the extent to which the research base on program features has developed since 2009, including ways in which the literature has or has not addressed questions or gaps identified in the Handbook. Although other approaches to organizing the literature review are possible, we chose to use the Handbook organization for two reasons. First, the Wallace Foundation was motivated by questions of how the research base for program features has developed since the Handbook was published. Second, the Handbook is the most thorough examination of the literature on preparation program features. Therefore, this review’s structure is responding to the funding organization’s questions and previous comprehensive reviews. It begins with a brief overview of the current context in which leadership preparation occurs, then covers the research on candidates in educational leadership programs; educational leadership faculty; program curriculum, design, and delivery; pedagogy; internships; student assessment; and mentoring and coaching. There is also a substantial section on evaluation of leadership preparation programs.
Method
Here we address several questions in Hallinger’s (2013) framework, including literature sources, selection, interpretation, synthesis, and evaluation. The literature reviewed here was culled from the ERIC database and “core journals” (Hallinger, 2013, p. 130) focused on educational leadership and leadership preparation, such as Educational Administration Quarterly, Journal of Research on Leadership Education, Journal of School Leadership, and Journal of Educational Administration. The literature also draws from studies focusing on the U.S. context. Although excellent work is being conducted outside the United States, we felt that with the influence of the national and state contexts on program features, we should narrow our search to studies of U.S. preparation programs. We have also focused primarily on studies of university-based preparation programs. We understand that other entities are involved in the development and implementation of educational leadership preparation, but the large majority of educational leaders are still trained in university programs and by far the largest focus of research is on these programs.
Selection criteria for the review were fairly simple. The 2009 Handbook reviewed literature up to 2006-2007. This article then covers literature from 2006 to 2007 not included in the 2009 Handbook up to August 2013. To be included, a publication must be an empirical study, generally published in peer-reviewed journals, though some reports of studies commissioned or produced by foundations and agencies are also included. Publications describing program features without also including a means of empirically investigating those features were not included. All modes of inquiry (qualitative, quantitative, mixed methods) were included, as methodological diversity is important to understanding social phenomena (Habermas, 1988; Whiteman, 2015). As described above, studies must be investigations in the U.S. context. Furthermore, studies must be focused on building-level school leaders (i.e., principals), and university-based preparation programs.
To locate literature, we used the EBSCOhost to search the ERIC database. The search strategy included using broad subject search terms (e.g., “Administrator Education”) and filtering for year and the U.S. context. Articles were then sorted according to topic. A draft of the review was presented at the 2013 UCEA Convention, where feedback helped to identify some additional studies that met the inclusion criteria. Included studies were summarized and synthesized by the authors. In all, 212 publications were included in the review.
It is important to note here that this is primarily a review and synthesis of empirical findings and the implications of those findings on leadership preparation programs, particularly relative to the knowledge base described in the 2009 Handbook (Young et al., 2009). This is not a review of methodology in the field, nor is it a review of the scholarly rigor of various publication outlets in the field, although we acknowledge methods used in the literature. Thus, studies are “evaluated, analyzed, and synthesized” (Hallinger, 2013, p. 130) based on what those studies tell us (or do not tell us) about specific program features.
Program Elements
In this section, we identify the specific program elements and discuss the major highlights from the literature since the Handbook was published. Rather than providing an extensive discussion of each study on specific elements, we will identify the major findings and the gaps we see that characterize research on particular program elements.
Context
The literature on leadership preparation program elements does not exist in a vacuum but in a context that has both internal and external forces. These forces influence the types of studies and the attention given to different program elements. Crow, Arnold, Reed, and Shoho (2012) identified four internal elements that contribute to the complexity of change of university leadership programs. These include faculty reward systems, faculty governance, institutional resources, and university responses to state-level influences. What educational leadership faculty members choose to focus their research attention on as well as their attempts to leverage program change is influenced by these internal factors. In addition, external forces influence the research on these program features. In the 2009 Handbook, Lamagdeleine, Maxcy, Pounder, and Reed (2009) identified four factors: the critique of public education, the rise of New Public Management, the altered operating environment, and the growing strength of state influence. Goldring and Schuermann (2009) argued that these kinds of forces have increased in intensity and complexity. In particular, growing state influence has provided a strong regulatory but frequently inconsistent message to university preparation programs that influences the research on program elements (Cibulka, 2009; Louis, Thomas, Gordon, & Febey, 2008; Roach, Smith, & Boutin, 2011).
In addition, several contextual changes influence the research we will discuss (Louis et al., 2008; Roach et al., 2011): greater focus on outcomes and assessments, greater competition from outside university settings, intensified critique of university-based programs, increasing but varied role of states in regulating programs, and continuing and growing expectations of faculty roles especially regarding direct work with schools. In addition, Cibulka (2009) identified three more internal features that weaken leadership programs, including value disagreements, inability to monitor quality, and weak research on program effectiveness.
Roach and colleagues (2011) used the notion from institutional theory of isomorphism to describe the external policy context of leadership preparation. The effect of this isomorphism is the development of both uniformity and flexibility and the emphasis on continued leadership development, which includes preparation, induction, and mentoring. A variety of organizations, including the National Council for Accreditation of Teacher Education, the Council of Chief State School Officers (CCSSO), the Southern Regional Education Board, the National Policy Board in Educational Administration, the National Association of State Boards of Education, the two national principals’ associations, and foundations, such as Wallace “all act as both suppliers and agents of the consumers of a new policy paradigm, namely a continuum of leadership development.” Roach and colleagues (2011) suggested several trends in the “organizational field” of leadership preparation, including a movement toward continuous leadership development, the uniformity of state standards while allowing greater flexibility of providers, the impact of Interstate School Leadership Licensure Consortium (ISLLC) standards, tiered licensure and renewal, and the continual changing of state policy environments. These changes, according to the authors, are increasing in pace, as state codes around preparation are more explicit, measures of effectiveness of preparation program are more performance based, and more emphasis is placed on career-long leadership development. In addition, the authors argued that a kind of normative isomorphism has resulted in preparation becoming more modeled around “best practices” “rather than utilizing the academy to generate new knowledge in administrator preparation (English & Papa, 2010).
Candidates of Educational Leadership Programs
In the earlier Handbook, Browne-Ferrigno and Muth (2009) identified the lack of rigorous, empirical research on characteristics of current candidates. In a more recent literature review, Browne-Ferrigno and Muth (2012) acknowledged that this gap has not been filled. Recent work, however, is beginning to provide some information on the characteristics of educational leadership program students. For example, Orr (2011) found a picture of graduates in her survey study of 17 leadership preparation programs in 13 institutions. Although we cannot assume that these characteristics are the same for candidates, the Orr study provides some evidence of program constituent characteristics. Overall, 64.7% were female and 11.0% were Black, Latino/a, American Indian, or from other underrepresented groups. The graduates’ mean years of teaching was 11.6 and the mean number of prior leadership experiences was 3.1. Orr and Orphanos (2011, following Darling-Hammond et al., 2010) compared principals from four exemplary preparation programs with principals in a national, conventionally trained sample. Principals in the two groups were similar in regard to age, but they varied on other characteristics, for example, the principals from the exemplary programs were more likely to be female and from underrepresented groups.
Recruitment of candidates
One of the most cited recent studies of exemplary principal preparation is Darling-Hammond and colleagues (2010). In their study, rigorous recruitment was a key factor in the success of these exemplary programs. The authors found that universities worked actively with districts to identify excellent teachers with leadership potential and to expand the racial and ethnic, cultural, and gender diversity of candidates. These programs also looked for individuals with experience as teaching coaches and those who demonstrated commitment to service in high-need areas.
Program admission processes
Very little new research has been conducted on program admission procedures and outcomes. A few exceptions include I. P. Young’s (2008) study using discriminant function analysis that found that Graduate Record Examination scores and Grade Point Averages were not particularly accurate predictors of program acceptance or completion. In addition, Griffin, Taylor, Varner, and White (2012) described the recruitment and selection processes of a redesigned cohort preparation program, which included selection criteria related to the ability to (a) serve as a school leader of learning, (b) provide effective leadership in challenging schools, and (c) be successful in curriculum and instruction offered by the program. Finally, Karanxha, Agosto, and Bellara (2014) in their qualitative case study pointed to an admission issue evidenced in the lower numbers of minority candidates and graduates in most leadership preparation programs. These authors found that disproportionate rejection of underrepresented applicants acts as a hidden curriculum in the preparation program.
The scant research on candidates and the recruitment/selection processes reinforces the call (Crow et al., 2012) for a national database on educational leadership program candidates that would enable the field to understand and, hopefully, focus recruitment and selection processes in ways to improve preparation programs.
Faculty of Educational Leadership Programs
Research on faculty in educational leadership preparation programs has been consistent over time. Thanks to a series of national surveys beginning in 1973 (Campbell & Newell, 1973) and continuing primarily under the leadership of Martha McCarthy (Hackmann & McCarthy, 2011a; McCarthy & Kuh, 1997; McCarthy, Kuh, Newell, & Iacona, 1988), we have a valuable portrait of faculty characteristics and trends over time.
These studies provide an extensive look at faculty, and we will mention only four major highlights from the most recent survey by Hackmann and McCarthy (2011a). First, the characteristics of faculty have remained fairly constant with change in some areas. Faculty members in leadership preparation programs still tend to be White males in their mid-50s; however, the number of females is increasing. Persons of color show a slow but steady increase over time (also found in the survey of Texas faculty by Haliburton, 2010).
Second, there are increasing numbers of part-time clinical and adjunct faculty in preparation programs and a decreasing number of full-time clinical. These clinical faculty members’ roles primarily are teaching and advising, along with some service. In other studies, these authors (Hackmann, 2007; Hackmann & McCarthy, 2011b) found that clinical faculty tended not to have job descriptions.
Third, an increasing number of faculty members have prior administrative experience. Although this would be expected due to the increasing number of clinical faculty, prior experience in school and district-level administration also characterizes tenure-track faculty. According to Hackmann and McCarthy (2011a), there has been a significant increase in faculty’s administrative experience since 1994: one third compared with two thirds in 2008.
Fourth, faculty in preparation programs are mostly satisfied with their jobs, and this satisfaction has increased over time. The top three factors that influence individuals to enter the professoriate in this field include interest in students and teaching, desire to improve education and/or society, and interest in research/extension of knowledge.
Faculty professional development is a significant gap in the program faculty research. Given the rapid change in educational organizations and reform trends, professional development of faculty seems critical. Research that describes what professional development is available, and the effectiveness of this resource seems an important area for future research.
Educational Leadership Preparation Curriculum
Since 2007, preparation programs have continued to vary in curriculum, and we know somewhat more about how curriculum matters in preparing educational leaders.
Recommended curriculum
Osterman and Hafner (2009) concluded that there was a lack of curriculum cohesiveness in preparation programs. In our review, we were not able to determine whether this cohesiveness has improved or not. However, we found a growing body of evidence that cohesiveness matters in exemplary preparation programs. Darling-Hammond and colleagues (2010) found that along with research-based content, “curricular coherence linking goals, learning activities, and assessments around a set of shared values, beliefs, and knowledge about effective organizational practice” (p. 42) was evident in exemplary programs.
We found more specificity and uniformity in the literature regarding curriculum content in leadership programs, for example, the focus on instruction, organizational development, and change management (e.g., the description of the Delta State University program, Griffin et al., 2012). More specificity is occurring in some areas of the curriculum, for example, equity, diversity, and social justice, perhaps as a result of national initiatives around learning for all students (Boske, 2012; Herrity & Glasman, 2010; Jennings, 2012; Marshall & Hernandez, 2013). Except for the Jennings’s (2012) cross-program survey analysis of 55 programs, the literature (cited here) involves qualitative case studies or content analyses.
Some literature has critiqued program curriculum. Gordon (2012) in a descriptive study offers an alternative framework that includes the components of awareness, care, critique, expertise, relationship, community, and accountability. Other authors also critique leadership curriculum as overly technocratic, focusing on skills and competencies and ignoring the development of values, beliefs, and identities (Lumby & English, 2009; Scribner & Crow, 2012).
Taught curriculum
In addition to the areas of instructional leadership, educational change, and managerial competence described in the earlier Handbook, we found more recent literature emphasizing ethical leadership, inquiry skills, conflict management, and especially social justice in the taught curriculum. Most of the studies are clearly descriptive rather than evaluative, so we lack research on the effectiveness of how these areas are actually taught. In the area of inquiry skills, Bustamante and Combs (2011), in a qualitative, cross-program review, found (a) variation among programs in the research course requirements, course titles, and course descriptions; (b) minimal emphasis on research skills in the program; and (c) a lack of consensus regarding views of the importance of this area.
A growing body of literature has examined how equity, diversity, and social justice are taught. For example, multiple authors have identified critical skills necessary for building social justice knowledge and skills, including experience in self-reflection and critical consciousness (Bruner, 2008; Christman, 2010; Diem & Carpenter, 2012), connecting theory with actions (Dentith & Peterlin, 2011), integrating social justice throughout the program rather than isolating it to one course (Diem & Carpenter, 2013; Everson & Bussey, 2007; Rodríguez, Chambers, González, & Scheurich, 2010), creating cognitive dissonance and concern (Guerra, Nelson, Jacobs, & Yamamura, 2013), assistance in identifying appropriate entry points for making change (Guerra et al., 2013), international practicum experiences (Richardson, Imig, & Ndoye, 2013), and deliberative dialogue (Mutchler, 2011). Although most of these studies are descriptive or normative, the Richardson et al. (2013) study used a mixed methods design with a diversity scale survey and interviews.
Several scholars have critiqued this area of the curriculum. Among the concerns raised are the lack of research connecting issues of diversity and race with leadership preparation curriculum (Boske, 2012; Diem & Carpenter, 2012, 2013; Hernandez & McKenzie, 2010) and the lack of research on what an entire program oriented toward social justice would look like (Hernandez & McKenzie, 2010).
Hackmann and McCarthy (2011a) found that program administrators perceived that the recommended and taught curriculum had changed over the last several years, with more attention on school improvement skills and learning opportunities for diverse students, and less attention on school management, finance and budgeting, school facilities, curriculum, and organization theory. What seems to be missing from the literature on curriculum is evidence of cohesiveness and the outcomes of the changes in the taught curriculum.
Design and Delivery of Leadership Preparation
According to Orr (2011) and Hackmann and McCarthy (2011), faculty have paid greater attention to redesigning preparation programs than in the past. Several elements have contributed to this emphasis (Orr, 2011), including conceptual, policy, and market elements (Brazer & Bauer, 2013; Buskey & Polizzi, 2012; Goldring & Schuermann, 2009; Hallinger & Snidvongs, 2008; Machado, 2012). The major characteristics of effective leadership preparation program design have been identified. The Darling-Hammond et al. (2010) study is frequently cited and includes rigorous recruitment and selection, research-based content, curricular coherence, field-based internships, problem-based learning strategies, cohort structures, mentoring or coaching, and collaboration between universities and school districts. Other studies have adapted these elements, for example, Orr (2011), which included challenging program, leading learning content, active learning instruction, knowledgeable faculty, cohorts, positive student relationships, supportive organizational structures, and internships.
Preparation programs designed with the elements found in the Darling-Hammond and colleagues’ (2010) study yield generally positive graduate perceptions of learning (Ballenger, Alford, McCune, & McCune, 2009, using a within-program comparison of follow-up survey responses; Orr, 2011, based on a survey study of 17 programs), graduate satisfaction with program (Orr, 2011), career intentions (Orr, 2011), leadership practices (Darling-Hammond et al., 2010; Orr & Orphanos, 2011), and indirectly student achievement and school effectiveness factors (Orr & Orphanos, 2011; Fuller, Young, & Baker, 2011 in a quantitative analysis of Texas Education Agency data of programs). Three particular types of design features have gained attention in the literature: cohort models, on-line/distance education delivery, and partnerships.
Cohort models
Research on the variety of cohort mechanisms is sparse (Mullen & Tuten, 2010). There is some research that identifies strengths—peer group development and learning values, for example (Preis, Grogan, Sherman, & Beaty, 2007, in their literature review). These authors also identified several concerns with cohorts including flexibility for students, limited academic freedom, and the lack of research on whether cohorts are in fact effective. Several studies have investigated the outcomes of cohorts and suggested this design feature increases the likelihood of program completion (Nimer, 2009, in a self-reflective, descriptive study), strengthens the development of professional skills—if a sense of trust and community exist (Brown, 2011, in a mixed methods study), and increases participants’ perception of being prepared to take on leadership roles (Huang et al., 2012). The Huang et al. (2012) study used a mixed method design including candidates’ self-reported preparedness and National Asssociation of Secondary School Principals’ (NASSP) Assessment Center data.
On-line/distance education delivery
Hackmann and McCarthy (2011) found that 73% of preparation programs used some form of distance learning. Fostering less traditional leadership styles, increasing the opportunities to improve teaching and learning, expanding geographic reach, and possibly equalizing race, gender, and disability advantages in the preparation programs are all benefits of on-line delivery (Preis et al., 2007).
Since the 2009 Handbook, more research on the outcomes of on-line/distance education has emerged, especially around perceptions of candidate hiring satisfaction, sense of community, and learning. However, much of the research is contradictory. For example, surveys of human resource directors indicated they were reluctant to hire principals with on-line credentials (Richardson, McLeod, & Dikkers, 2011a, 2011b). However, students had similar views in evaluating on-line versus face-to-face delivery modes in three survey studies (Chapman, Diaz, Moore, & Deering, 2009; Ritter, Polnick, Fink, & Oescher, 2010; Sherman, Crum, & Beaty, 2010). Ritter et al.’s (2010) study, using the Classroom Community Scale, found that students perceived face-to-face and blended as producing more sense of community, but there was no difference in the evaluation of learning outcomes.
We found only one study that examined learning outcomes (Korach & Agans, 2011). In a mixed method study of students’ perceptions, these authors found that blended versus classroom delivery had similar outcomes, including critical and reflective thinking, knowledge of systems, and capacity to analyze data and diagnose organizations. In a similar fashion to Preis et al. (2007), these authors found that on-line delivery modes decreased the capacity for individual voices to have power and influence, thus equalizing the teaching and learning environment.
Partnerships
Several researchers using primarily descriptive methodologies have found that university–district partnerships have important advantages, including bridging theory and practice, creating more delivery options, and emphasizing collaborative leadership (Darling-Hammond et al., 2010; Preis et al., 2007). Other research identifies challenges, including obtaining consistent participation and ownership from all stakeholders, maintaining a non-favoritism approach, dealing with leader turnover in districts, placing interns (Brooks, Havard, Tatum, & Patrick, 2010), and addressing the constraints of institutional impediments and economic conditions (Browne-Ferrigno, 2011). Other descriptive studies on partnership designs have also identified elements of effective partnerships, including keeping an overarching focus on a common goal, defining responsibilities, communicating authentically, aligning common interest, maintaining creative dynamic receptiveness to ongoing evaluative feedback (Kamail, Barber, Schulman, & Reed, 2012, p. 902), developing a mandate from the state for all programs, acknowledging the importance of the dean for developing and maintaining trust with other stakeholders, and developing a significant number of active district partners who are willing and able to participate (Brooks et al., 2010).
An emerging literature examines the outcomes of effective partnerships, mostly from student and graduate perceptions. Students perceive opportunities to connect theory and practice (Borden, Preskill, & DeMoss, 2012, using quantitative and qualitative data from end of class and end of course questionnaires and evaluations), gains in leadership knowledge and skills, opportunities for advancement (e.g., Orr & Barber, 2007, comparative study), greater program quality, stronger role for districts (Orr, 2011; Orr & Orphanos, 2011; and the case study design of Korach, 2011), and self-understanding and readiness for role change (Simmons et al., 2007, using an action research approach).
Research on program design has begun to address many of the issues raised previously, for example, identifying the elements of effective design and outcomes. As with many outcomes studies, however, we need to examine more complex perspectives, for example, how contextual factors influence the outcome of particular design elements.
Pedagogy
Pedagogical theories are rarely empirically explored in research on preparation program features. Instead, principles of various pedagogical theories (e.g., adult learning theory) are often backgrounded in the rationale for various program features. Davis, Leon, and Fultz (2013) recommended preparation programs built on adult learning theories advanced by Knowles, Holton, and Swanson (2005) and Mezirow (1997), in which programs are contextually driven; include problem-based and on-the-job learning activities; supply candidates with coaching, mentoring, and peer networking opportunities; gradually release candidates to more practice-based responsibilities and to rely more on their own inquiry and problem-solving abilities. Arguably, these principles form the rationale for program features found throughout this review but may not always be explicitly articulated in the original publications.
Beyond Davis et al.’s (2013) exploration of adult learning theory and contextually relevant workplace situations, there is sparse empirical research focused explicitly on the foundations of pedagogy with two exceptions. First, research primarily through the Carnegie Project on the Education Doctorate has examined signature pedagogy and other instructional strategies primarily as part of the final degree stage (Olson & Clark, 2009; Perry & Imig, 2008; Shulman, 2005). Since then, little has been published regarding signature pedagogy of entry level preparation. Second, a piece by Black and Murtadha (2007) emphasized leadership as an ethical and moral craft that drew on pedagogies such as complex case studies, inquiry-centered internships, collaborative and interdisciplinary leadership institutes, and continuous assessments. According to these authors, these instructional practices lead to the development of public intellectuals and transformative leaders. Some literature has also identified influences on pedagogies, including educational reform movements, national standards, and the impact of neoliberalism, globalization, and social justice (Jacobson & Cypres, 2012).
The literature on specific pedagogies is not extensive and tends to be descriptive and normative rather than evaluative. The major pedagogies examined in the literature are multimedia methods, cases, simulations, technology, action research, and reflection. Several descriptive studies examined how the use of multimedia tools can enhance social justice (Boske, 2011), public speaking and media communication skills (Friend, Adams, & Curry, 2011), and leadership identity (Guajardo et al., 2011).
Case pedagogy has existed for many years, and the number of cases available for leadership preparation has increased with the publication of the UCEA Journal of Cases in Educational Leadership. However, the increase in the number and use of cases has not seen a corresponding increase in research on the outcomes of case use. Some research, again mostly descriptive, however, has demonstrated that case use enhances the development of skills aligned with preparation standards (Bass, Garn, & Monroe, 2011), advances reflection (H.-D. Meyer & Shannon, 2010), and builds a community of practice (H.-D. Meyer & Shannon, 2010). Tucker and Dexter (2011), in their surveys of students and faculty, found the use of on-line cases was related to developing students’ decision-making skills and enhancing more generalized self efficacy, confidence, and certainty about decision-making processes.
Simulations of leadership situations have also been popular in preparation programs. Friend et al. (2011) described their use in developing persuasive speech skills and in dealing effectively with the media. Other descriptive research has demonstrated simulations used to develop decision-making skills (Mann et al., 2011).
Some literature on the use of technology was discussed under the Design section of this article; however, other literature has dealt with two specific types of technology as a pedagogical strategy: e-portfolios and on-line discussion. Hyland and Kranzow (2012), in a mixed method study involving both students and faculty, found that e-portfolios increased students’ critical thinking and self-directed learning. In regard to on-line discussions, Nash (2011) identified from his own experience elements for critiquing its success, based on the scaffolded learning framework. He emphasized autonomous learning rather than transmissive approaches to learning.
Action research is receiving more attention both as an approach to develop leaders as researchers and as a doctoral program capstone (Barnett & Muth, 2008; Osterman, Furman, & Sernak, 2014; Zambo & Isai, 2013). A small amount of research is focused on the outcomes of the use of action research. Batagiannis (2011) found that developing action research skills had an impact on leadership identity, on transformative leadership, and on the type of reflection necessary for leading professional learning communities. Her study design involved a reflective case study, an analysis of self-evaluations of students, and a content analysis of the action research topics.
Several mostly case studies went beyond description to focus on outcomes of reflection for leadership development, identity as a change agent, problem-solving skills, and collaborative skills (Bowers & Murakami-Ramalho, 2010; Brody, Vissa, & Weathers, 2010; Mutchler, 2011). Boske (2011) found that reflection, based in multimedia approaches that created safe environments, transformed the conversations and views of social justice.
Although pedagogical improvements seem to have gained attention (Hackmann & McCarthy, 2011a), there has been sparse research on the conceptual foundations that ground the choice of pedagogies. Moreover, most of the literature on specific pedagogies is descriptive and normative rather than evaluative.
Internships in Preparing School Leaders
Various studies have reinforced the importance of quality internships (Christian, 2011; Darling-Hammond et al., 2010; Duncan, Range, & Scherz, 2011; Orr, 2011). In their literature review, Barnett, Shoho, and Copland (2010) identified three types of internships: (a) full-time embedded internships which are rare but growing in appeal; (b) detached internships, in which students document completion of activities in journals or portfolios; and (c) course-embedded internships, in which the activities are distributed across courses.
There is apparent agreement on the elements necessary for successful internships, including support by site-based and university-based mentors (Christian, 2011, in a survey study), clear expectations (Clayton & Myran, 2013, using student logs and surveys), provision of real-world practical leadership responsibility (Havard, Morgan, & Patrick, 2010), and high-quality mentoring and coaching (Duncan et al., 2011; Shoho, Barnett, & Martinez, 2012). In their literature review, Sherman and Crum (2009) also identified the following actions necessary for successful internships: (a) involve the district in the internship collaboration, (b) design experiences around ISLLC standards, district priorities, and site-based needs; (c) provide prolonged internship engagement not scheduled at the end of the program, (d) carefully pair mentors and interns, (e) provide time for interns to reflect and share their experiences, and (f) provide university support for the student, mentor, and district leadership.
An emerging group of studies is attempting to fill the gap on the effects of exemplary internship experiences (Barnett et al., 2010). The positive findings from these authors’ literature review included positive effects on leader knowledge, skills, and dispositions; change in role conceptions and career decision making; increase in knowledge and skills related to building operations, problem solving, interpersonal skills, time management, and reflection; and the reduction in isolation and the increase in confidence. However, these authors also found that internships can maintain the status quo.
Several quantitative or mixed method studies have examined more specific effects of internships, for example, involving acquisition of skills related to standards (Barton & Cox, 2012; Ringler, Rouse, & St. Clair, 2012; Stevenson & Cooner, 2011). For example, Ringler and colleagues (2012), using multiple student assessments and surveys of supervising principals and university faculty, found that students enacted instructional leadership and managerial leadership the most during internships. Candidates’ changing career intentions are another outcome of internships (Orr, 2011). Perez, Uline, Johnson, James-Ward, and Basom (2011) found that over time, interns came to see the work as complex; lead others toward a clear, shared vision of excellence; mobilized a collaborative effort; built individual and organizational capacity to enact a vision; and used data to inform change. The authors also found that interns were more ready to lead after the internship experience. Christian (2011) discovered in a quantitative, survey study that program completers agreed that internship experiences and mentor support facilitated their application of the standards.
Although research on internships is maturing and more attention is devoted to their effects, the research tends to be descriptive or limited in the number of cases. Cross-institutional studies of internships would contribute to our understanding of contextual influences on internship effects.
Student Assessment
Student assessment is receiving more attention recently due in part to the policy debates around standards. For example, Cheney and Davis (2011) critiqued the ISLLC student assessment as measuring inputs rather than outputs. In her critique of the Cheney/Davis perspective, Orr (2012) pointed out that these authors ignore many existing practices in candidate assessment that utilize these output measures.
Four types of student assessments are reflected in the literature: portfolios, journal mapping, simulations, and formal multirater assessment instruments. Bengtson (2012) in a descriptive study of the Arkansas Leadership Academy found that reflective practice was related to higher quality candidate portfolio assessments for a portfolio aligned with the ISLLC standards. Knoeppel and Logan (2011) analyzed 5 years of data from student portfolios to measure continuous student progress from program entry to exit, evaluation of student understanding and application of ISLLC standards, and linkage of theoretical knowledge to field-based projects. On the whole, the researchers found a general positive trend.
Journal mapping emphasizes assessing candidate skills in reflection. Stevenson and Cooner (2011) conducted a longitudinal empirical study of journal mapping and found interns’ perceived knowledge of the standards and integration of the standards grew throughout the internship.
Simulations are used both as instructional tools and assessment techniques. Literature on the assessment side tends to be descriptive rather than evaluating the effectiveness for assessment purposes. Several studies describing simulations as assessments can be found in the Pedagogy section above.
Multirater assessments are also being developed to assess student skills. The most discussed is the Vanderbilt Assessment of Leadership in Education (VAL-ED) (Cravens et al., 2013) instrument. This rating scale assesses the effectiveness of principals’ behaviors known to influence teachers’ performance and in turn students’ learning.
Korach and Agans (2011) argued for using a variety of assessments at all stages (formative, quarterly, and summative) and the usefulness of technology to make these assessments more systematic. Buskey and Karvonen (2012) described the development of an assessment process using end-of-course reflections, outcomes of action research projects, and 360-degree feedback to measure student growth. This last point demonstrates the need for more research using multiple assessments to determine adequate measures to assess student outcomes regarding leadership knowledge and skills.
Program Evaluation of Leadership Preparation
Evaluating leadership preparation programs has become an important responsibility of university faculty charged with preparing educational leaders. This responsibility has intensified and expanded in terms of the stakes within national, state, district, foundations, and private sector contexts and influences. Two recent events at the national level have intensified the importance of program evaluation: the creation of the Council for the Accreditation of Educator Preparation (CAEP) and the current consideration by the CCSSO of the revision of the ISLLC standards (Canole & Young, 2013; CAEP, 2013; M. D. Young & Mawhinney, 2012).
As noted in the Context section, the state’s influence on program approval and improvement has intensified in the last several years and in many instances has emphasized outcome measures (Cheney & Davis, 2011; Phillips, 2013a, 2013b). Murphy, Moorman, and McCarthy (2008), based on their use of primary and secondary sources related to preparation program reforms in six states, found a great deal of unevenness of the results of these statewide reforms. Roach and her colleagues (2011) demonstrated mimetic isomorphism in the process of copying reforms across state boundaries.
In addition to national and state emphasis on program evaluation, other units have contributed to the raised stakes for program evaluation. These include districts developing their own leadership preparation programs (Mitgang, 2012), foundations (especially Wallace Foundation, 2010), and the private sector. Not everyone believes these high stakes uses of program evaluations are valuable. This literature questions the assumptions behind some of the critiques of university-based preparation programs (Bogatch, 2011; Breault, 2010; English, Papa, Mullen, & Creighton, 2012; Reitzug, 2010).
Program evaluation efforts by level
The majority of program evaluation efforts are conducted at the state, rather than national level. We found no examples of nationwide program evaluations. However, there are some studies of multistate evaluations, most prominently policy briefs conducted by the Regional Education Laboratory (REL)–Midwest (Behrstock, Bhatt, Cushing, & Wraight, 2010; Cushing, Bhatt, Wraight, Behrstock, & Meyer, 2010; C. Meyer, Bhatt, Wraight, Behrstock, & Cushing, 2010), the SREB (Schmidt-Davis, 2009, 2010), and the Murphy et al. (2008) cross-state comparisons.
Several examinations of specific states have been conducted using a variety of methods, including interviews with professors, principals, and other key stakeholders (Black, 2011; Bost, 2009; Browne-Ferrigno, 2011, 2013; Doolittle, 2013; Militello, Gajda, & Bowers, 2009). Murphy et al. (2008) examined the extent of reform of preparation programs in six states. These authors found (a) most program developers are following an unproductive paradigm, (b) the results of reform are uneven and overall fall short of the mark, (c) lack of actionable theory is a major shortcoming, (d) reforms are hampered by weak curriculum focus and content, and (e) neglect of the technical and adaptive dimensions of change makes serious reform unlikely (Murphy et al., 2008, p. 2173). These studies, overall, seem to find mixed results in state-level preparation program reform.
Findings regarding program assessment practices
Program assessment practices appear to be moving toward more variety and measurement expertise. Need/gap analysis studies, using surveys, interviews, pre and post self-assessments and other methodologies, have identified specific areas of principal need not being met by programs, including conflict management skills (Anderson, 2007), special education and gifted skills (McHatton, Boyer, Shaunessy, & Terry, 2010), practicum experiences (Tubbs & Holliday, 2009), or skills related to the various ISLLC standards (Williams & Szal, 2011). This type of analysis was also used to assess a more general array of knowledge and skills, for example, human relations, personnel, special education, student services, as identified by program graduates (Petzko, 2008, using a survey of first-, second-, and third-year principals and assistant principals).
Implementation studies were also found in the program evaluation literature. This literature conducted as descriptive studies or using surveys, interviews, documents, and observations focused on the implementation of university–district partnerships (Browne-Ferrigno, 2011; Korach, 2011; Simmons et al., 2007), program design (Browne-Ferrigno, 2013; Kochan & Reames, 2013; Reed & Llanes, 2010), and pilot programs (Schmidt-Davis, 2009).
By far the largest number of program assessments focused on outcomes, not surprising given the current policy influences that emphasize candidate performance. The majority of studies used perceptual methods. These varied from surveys of graduates for a within-program comparison (Ballenger et al., 2009) to case studies (Browne-Ferrigno, 2013) to appreciative inquiry (Calabrese et al., 2007) to a few quantitative studies (Fuller et al., 2011). The Ballenger (Ballenger et al., 2009) study is part of a larger UCEA and American Educational Research Association (AERA) Teaching for Educational Leadership Special Interest Group (SIG) collaborative research effort to rigorously examine program effectiveness and impact (Orr & Pounder, 2006).
The foci of these outcome studies also vary from participant satisfaction (Edmonds, Waddle, Murphy, Ozturgut, & Caruthers, 2007); effects on certain types of knowledge, skills, identities (Fuller et al., 2011; Guerra et al., 2013; Keiser, 2009; McHatton et al., 2010); gender of graduates (Collay & Cooper, 2008); student and teacher outcomes (Fuller et al., 2011; Martorell, Heaton, Gates, & Hamilton, 2010; New Leaders for New Schools, 2011; Nunnery, Yen, & Ross, 2011; Orphanos & Orr, 2013; Orr & Orphanos, 2007); types of programs and program designs (Christian, 2011); and more complicated foci, for example, the effect of the district in influencing school leadership preparation program (Orr, King, & LaPointe, 2010). Examples of findings from two of the more rigorous studies provide an overview of what these program evaluations tell us about preparation programs. Ballenger et al. (2009) used a comparison of follow-up survey responses from two sets of program graduates. The authors found that graduates of a redesigned program involving an intensive internship, revised on-line modules, and a newly Educational Leadership Constituencies Council (ELCC) aligned curriculum rated program outcomes higher in a wide array of factors. Orr’s (2011) study included faculty surveys of graduates of 17 preparation programs between 2004 and 2007 to identify the characteristics and outcomes of the programs. She found that these programs had many of the characteristics of exemplary programs found in the Darling-Hammond et al. (2010) study, and these elements were positively related to outcomes including graduates’ leadership learning, satisfaction with the program, and beliefs about the principalship.
Other researchers are beginning to assess how principal preparation programs relate to school improvement and student learning outcomes, for example, teacher team quality (Fuller et al., 2011). Orr and Orphanos (2011; Orphanos & Orr, 2013) also found a positive but mediated relationship between preparation program quality and school improvement progress and school effectiveness climate.
Comparative analysis of programs is a valuable trend in the program evaluation research. This research includes comparisons within the same state (Black, 2011, in Indiana), across several states (Darling-Hammond et al., 2010; Murphy et al., 2008), schools with similar characteristics (Doolittle & Brown, 2011), different state political climates (Louis et al., 2008), and different foci (e.g., social justice, Rodríguez et al., 2010; turnaround principals, Schmidt-Davis, 2012).
Development of assessment instruments is a new trend in program assessment. The UCEA/Teaching in Educational Administration SIG taskforce has been instrumental in designing assessment frameworks and in encouraging the development of appropriate instruments (Orr & Pounder, 2006; Pounder, 2012). Other researchers have also taken on this important initiative (e.g., Darling-Hammond et al., 2010; Melton, Tysinger, Mallory, & Green, 2011; Ross, 2010).
Research examining program evaluation practices is increasing, no doubt because of the high stakes nature. However, more rigorous studies with student outcome and school improvement variables are needed. More complex studies (e.g., Orr et al., 2010) are needed to consider variables such as district effects.
Leadership Mentoring and Coaching
Research on mentoring focuses on types of settings and elements of effective delivery in these settings. Duncan and Stock (2010) examined mentoring in rural school districts and found three critical topics perceived as needed by principals: professional and organizational socialization, use of data for informing decision making and instructional leadership, and work with difficult faculty. Clayton, Sanzo, and Myran (2013) in their qualitative study found that time and accountability pressures that inhibit interactions and poor mentor/protégé pairing affect mentoring success. They also found mentoring to be more successful if focused on a project and delivered through a structured tool to direct conversation. Other studies (Mullen & Tuten, 2010, using a narrative case study; Searby, 2010, based on student responses to an assignment involving seeking a formal mentor) emphasized the value of informal peer support and the co-construction of knowledge by both mentor and mentee.
The literature on effectiveness reflects a varied picture with some positive and some negative impacts on mentoring. For example, two studies of statewide mentoring programs (Alsbury & Hackmann, 2006; Dyer, 2010) found a mixed picture. The Alsbury and Hackmann (2006) study based on a formative assessment with open-ended surveys found four factors that increased the likelihood of mentoring success including sufficient time to meet, mentors initiating communication, pairings from the same district, and specific interpersonal skills, namely, listening rather than advising and asking questions rather than always giving answers. In the Dyer (2010) study, based on a mixed methods design, more than half the protégés were not assigned a mentor, and those who were assigned a mentor largely rated the program as ineffective. However, protégés were positive regarding opportunities to develop skills in a safe environment, enjoyed having a sounding board, and appreciated guidance and direction.
Three studies of district mentoring programs (Daresh, 2007; Mitgang, 2007; Woolsey, 2010) identified several elements of success: focusing on knowledge and skills regarding instructional leadership, management and operations, change and building school community; building protégé confidence; developing a culture of continued leadership development; and elements such as sufficient time, mentor training, and state funding. As an example, Daresh (2007) in his interviews with 20 mentor principals found that beginning confidence in administrative skills facilitated later development of instructional leadership competence. A case study of an early-career female principal emphasized the importance of non-hierarchical, mutual learning interactions (Peters, 2010).
Mentor selection
It is surprising that given the importance in previous literature (Cohn & Sweeney, 1992; Crow & Matthews, 1998; Walker & Stott, 1993) of mentor selection and training to the success of mentoring, there has not been more research on this topic. Graff and Pettengill (2010) in their policy analysis study found only 15 states had a policy that could be evaluated, and these policies had very little requirements to govern the process. The recommended selection criteria included minimum years of relevant experience, a sound record of success, and self-nomination. In terms of matching mentor and protégé, they also recommended geographic proximity and similar school background.
Coaching
In a dissertation study, Roberson (2011) explored six principal/coach cases in a California district and emphasized the following impacts: shaping reflective, instructionally focused practice; feeling supported during stressful times; having an authentic, trusting relationship; and enhancing relationship-building through technology. James-Ward (2011) in her descriptive study found that coaches and school leaders did not always have the same understandings of the district initiatives. However, the coaches felt having time for the coaches to meet together helped reduce role ambiguity and gave coaches time to identify common issues among the new principals. James-Ward and Salcedo-Potter (2011) studied new and veteran principals’ perceptions of coaching and found that the participants largely agreed that the program had a positive impact, but they differed in some of their concerns. The coaches indicated keeping principals inspired and preparing them to be instructional leaders had the most impact. The principals felt most impact from the coaches in terms of giving feedback to teachers and helping the principals to be more reflective.
Two studies of university-based internship programs with coaching found elements for success. Shoho et al. (2012), based on a description of an exemplary coaching program, argued for a full-time, job-embedded internship enhanced by transformational coaching. Coaches were in the school on a weekly basis and encouraged interns to be reflective, strategic, relational, and proactive. Silver, Lochmiller, Copland, and Tripps (2009) found new principals viewed the coaching program positively. Both the coaches and the principals pointed to personalized support as the most significant aspect. Other components that were viewed as critical were the coach–principal match, the coach’s 3-year commitment to the principal, and the focus of coaching conversations that began with the administrative basics before moving to instructional leadership.
Research on mentoring and coaching still lacks rigorous examinations of the effectiveness and outcomes of mentoring programs. Large-scale studies of mentoring and coaching effectiveness across multiple programs would move the field away from viewing these learning tools as panaceas to a more authentic understanding of their benefits and costs. In addition, we need empirical studies on mentor and coach selection that can inform this vital element of successful mentoring.
Conclusion
Hallinger’s (2013) last point in his conceptual framework for a literature review includes an indication of the major results, limitations, and implications of the review. The first major finding is that the literature on educational leadership preparation programs has definitely expanded since 2009 and, in many areas, has enriched our understanding of effective preparation that can inform the development and refinement of program standards. For example, since 2009, we have a clearer set of features that should be included in exemplary programs. We have research that can contribute to a stronger understanding of what should be included in preparation curriculum, what pedagogies have the potential for developing strong leaders, what a rigorous internship should include, and what ingredients make for a more effective partnership between universities and districts.
Like all literature reviews, ours has limitations. First, we intentionally left out literature from outside the United States. Although we did this for practical purposes, this decision prevented us from including some valuable research that could inform U.S. leadership preparation programs. A growing body of international literature on successful principals (e.g., Day & Gurr, 2014) and principal preparation (e.g., Slater & Nelson, 2013) and the growing involvement of UCEA in international collaborations suggest the valuable international resources available to inform preparation programs. Second, this review focuses on university-based leadership preparation. The Context section acknowledges that preparation is being conducted by probably a growing number and types of alternative providers. Although we suspect there is currently sparse research on these providers, our focus on university preparation programs ignores this important area for research.
In spite of these limitations, our review has provided a clear idea of where research needs to be focused, for example, who is participating in preparation programs, how to recruit and select the most likely innovative leaders, what kinds of student assessments are most useful, and what new kinds of program evaluations we need for reforming and refining our preparation programs. There are ample directions for future research that this article has identified in the various sections of the review. We highlight a few of these that help to fill methodological and practical gaps in research on leadership preparation.
In regard to methodological gaps, it should be clear from our review that most research on leadership preparation is descriptive. Although descriptive studies certainly have their merits, increased methodological diversity is required to understand the complexities of leadership preparation. Our review suggests that there are at least three other designs that could benefit the field. First, effect studies that examine the outcomes of different pedagogies, curricula, internships, and designs would strengthen the research knowledge base of preparation. Second, longitudinal studies would be helpful in understanding how candidates’ knowledge, skills, and dispositions develop over time from recruitment to placement. Third, we need cross-institutional comparisons that provide a better understanding of the complexity of preparation, for example, how various school and district contexts influence leadership learning.
In addition to opportunities for methodological diversity, our review has identified several important directions for future research related to the practice of preparation. First, although we know much about faculty, we know very little about the candidates. Recently, research has identified characteristics of graduates but not the candidates who enter our programs. More importantly, no national database is available to enable us to develop the big picture in terms of those individuals who are attracted to preparation programs. Second, we need to know the outcomes of our recruitment practices, especially in terms of underrepresented students, our use of culturally relevant pedagogies and curriculum, our field experiences, and our mentoring processes. Finally, although the focus of this literature review has been on the beginning stage of leadership development, that is, preparation, the policy environment is emphasizing continuous leadership development. Understanding how preservice informs induction and growth are critical for reforming the profession.
Standards and policies for leadership preparation programs are only effective if they are based on what we know about both what is happening and what should be occurring in order to prepare effective, innovative change agents for schools. Our hope is that this article helps to inform these considerations and decisions on how we prepare these leaders.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.