Enriching the Preparation of Education Researchers and Practitioner-Scholars

Abstract

This article analyzes a novel effort to strengthen the preparation of both practitioner-scholars and education researchers. It describes a university–district partnership that offers graduate students the opportunity to develop research understandings and skills through participation in a “real” research project and provides district leaders the formative evaluation information they need to guide decision making on major reform initiatives. The article suggests that partnerships that link school district research priorities with university-based, policy evaluation research courses can enrich the university’s approach to the teaching of research methods and can yield rigorous studies that both serve local districts and speak to broader academic and professional audiences.

Keywords

research apprenticeship school–university research partnerships authentic teaching model

Introduction

The preparation of education researchers (those who aspire to positions that focus primarily if not solely on research) and practitioner-scholars (individuals who seek to conduct and use research in their leadership positions in school systems) has been the object of intense criticism and recurrent debate (Eisenhart & DeHaan, 2005; Levine, 2007; Shulman, Golde, Bueschel, & Garabedian, 2006). Multiple factors, including claims that education research often fails to meet the variously defined quality, significance, and utility standards of scholarship, have prompted some colleges of education (or units of them) to revisit how they prepare their graduate students to conduct research. For example, some colleges have redefined the research requirements for master’s and doctoral study, drawn sharper distinctions between EdD and PhD degrees, and/or restructured particular courses (Eisenhart & DeHaan, 2005; Metz, 2001; Shulman et al., 2006). Amid ongoing disputes surrounding what counts as worthwhile research, who should be prepared to carry out education research, and how they should be equipped to do so, colleges of education continue to search for ways to inject rigor and relevance into the research methods components of their graduate education programs (Ball & Forzani, 2007; Gutierrez & Penuel, 2014).

This article analyzes a novel effort to strengthen the preparation of both practitioner-scholars and education researchers. It focuses on a university–district partnership that provides master’s and doctoral students rigorous research experiences and district leaders relevant data on major reforms they are undertaking.¹ This article is, to borrow Metz’s words, “a product of reflective practice rather than of formal research” (Metz, 2001, p. 12). It is based on my experience as the co-founder and the co-teacher of a year-long research apprenticeship course that equips students to develop a robust formative evaluation of a major reform initiative in a nearby school district. Over the past decade, through work on four distinct research apprenticeship projects, my colleagues and I have helped 35 full-time and part-time graduate students with quite different career aspirations learn how to conduct research that can inform local district decision making and speak to broader professional audiences through refereed conference papers and journal articles.² I use this case-in-point experience as a springboard for considering how carefully crafted research apprenticeships might operate as a powerful pedagogical strategy for developing graduate students’ research capabilities. Following a brief description of the research partnership and the research apprenticeship course, I discuss its promise as an approach for preparing individuals to carry out educational research.

The District–University Partnership

In May 2006, the Prince George’s County school district, located adjacent to the University of Maryland, College Park (UMD), initiated a research partnership with the Department of Education Policy Studies, lodged in the College of Education at UMD. Dr. Donna Muncey, a member of the incoming superintendent’s cabinet, contacted me to see whether I would be willing to spearhead the creation of a research team composed of graduate students whom she and I would train to carry out a formative evaluation of one of the interventions the new superintendent would be launching in a subset of the districts’ “struggling schools.”³ An experienced researcher and graduate instructor, Donna understood the importance of securing high-quality formative evaluations to guide district decision making as well as the contributions that authentic research experiences might make to a graduate student’s program of studies. She also understood that while the district had an established research and evaluation unit, this unit lacked the capacity to take on additional, labor-intensive responsibilities and the district lacked the funds required to acquire an evaluation from reputable but pricey private contractors. Simply put, in her view, the district needed the university’s help to carry out essential research. Contacting someone she knew in the College of Education was a logical place to begin exploring the prospects for forging a research partnership.

I readily agreed to assist for three reasons. First, I trust and respect Donna. We had worked together on several research projects that involved graduate students, so I knew she was an excellent scholar with impeccable integrity and strong mentoring skills. I also knew I could count on her to uphold her end of any bargain we might strike. Second, I was not satisfied with the policy/program evaluation course I had been teaching. I was convinced that graduate students interested in education policy and leadership needed an opportunity to learn about the substantive, political, and ethical issues embedded in evaluation research not only through readings about and critiques of this research but also through intensive involvement in all phases of an actual evaluation. While I wanted to have students engaged from the design of a policy/program evaluation to the dissemination of results, I simply had not figured out how to accomplish that within the confines of the semester course the department dedicated to evaluation research.⁴ Third, I was concerned that opportunities to be involved in faculty-directed research projects were generally limited to full-time students. Like policy and leadership departments in other universities (Anderson, 2002), my department was, at the time, enrolling a sizable number of part-time master’s and doctoral students who did not qualify for or have time for the research assistantships that the college afforded. Since I believe that all graduate students should have the opportunity to become producers and consumers of research, the unequal access to meaningful research experiences was a situation I was eager to rectify.

Donna, a doctoral student, and I developed a proposal for a year-long research apprenticeship course that might enrich the research method options available to graduate students and provide useful information on the implementation of an initiative the newly appointed superintendent intended to launch to improve “struggling schools.” With the support of the district and the university, the research apprenticeship moved from an interesting idea to a unique, “free-standing,” course offering structured as a two-semester, six-credit seminar with an option to earn additional independent study credits during the summer term. The course was not embedded in a particulate graduate degree program. Rather, it was open, initially, to all graduate students in the College of Education who, after talking with the course instructors, indicated they understood the course demands and were willing to take on the responsibilities associated with completing a research project for a public school system. In later years, the course accepted students from the School of Public Policy and the College of Education.

Team-taught by those who developed the proposal, the first research apprenticeship focused on how the new superintendent and his leadership team developed the Intensive Supports and Intervention Schools initiative and how that initiative played out in practice (Malen, Basinger, Gonzalez, & Nutter, 2007). Subsequent iterations of the research apprenticeship, team-taught with colleagues from my department, focused on the initial design and implementation of an educator incentive program (Malen et al., 2009); the ongoing implementation and early effects of this program (Malen et al., 2011); and the development and initial implementation of a student-based budgeting initiative (Malen et al., 2013). Each iteration was authorized in a meeting with the superintendent at the time (the district has experienced substantial turnover in this office) and developed in collaboration with the superintendent’s designee(s). In every instance, that designee was a member of the superintendent’s cabinet and the individual ultimately responsible for the reform initiative under study.⁵

Generally speaking, the partnership has been sustained, despite turnover in the superintendent’s office and cuts in the district budget, because it has provided the district useful information at a reasonable cost, the department a way to provide all graduate students an opportunity for hands-on research experience, and the broader university a means to merge its flagship and land-grant missions through research done with and on behalf of public schools. It also has been sustained by the rapport the research teams have established with district leaders and school-level personnel and the respect that these educators have for university faculty efforts to create “real-life” learning experiences for our graduate students. Although its future is uncertain, primarily because the department is short-handed and the research apprenticeship requires a heavy investment of faculty resources, the College of Education has recognized the research apprenticeship as a valuable form of outreach to districts and an engaging way to teach research methods.

The Research Apprenticeship Course

Each research apprenticeship course has taken on some unique features given the nature of the initiative under study and the scope of work the district officials and the course instructors have negotiated; yet, all four versions were learning laboratories characterized by common aims, components, and supports. All four versions also have unfolded through interrelated but identifiable phases, yielded detailed formative evaluations for the district (Malen et al., 2007; Malen et al., 2013; Malen et al., 2009; Malen et al., 2011), and provided empirical data for conference papers and publications in peer-reviewed journals (see, for example, Malen, Dayhoff, Egan, & Croninger, 2015; Malen, et al., 2015; Rice et al., 2012; Rice, Malen, Jackson, & Hoyer, 2016).

Aims

Each rendition of the research apprenticeship has sought to help students develop a well-grounded understanding of the research process and the criteria used to judge the quality of the end product, the complexities of education reform, and the challenges of conducting rigorous, relevant research on the implementation and impact of policies/programs that are in a constant state of flux.⁶ More specifically, each apprenticeship has focused on helping students develop the capacity to frame questions, link questions and methods, uncover sources of data, collect and analyze information, inspect inferences, develop lines of argument, assess the trade-offs embedded in evaluation research choices, communicate findings, and articulate the implications of those findings for research, theory, policy, and practice.

Components

My colleagues and I have pursued these goals through the strategic integration of formal instruction carried out in weekly seminars and field research. Generally speaking, the weekly seminar sessions have focused on preparing students to carry out a theory-driven, stakeholder-based formative evaluation; translating the research proposal into a workable research plan; developing the protocols we need to track the relevant literature and to collect data required to address our research questions; and incorporating procedures to check for bias and error in the collection, analysis and resporting of data.⁷ The sessions also have included opportunities to practice using the data collection instruments we develop, to clarify and coordinate field research assignments, to debrief the students’ field research experiences, and to develop analytic memos that capture what we are learning about the initiative of interest and that enable us to critically assess the emergent themes and patterns. The field research typically involves document reviews, observations of relevant training sessions, and management meetings as well as individual and focus group interviews with district officials and site-level educators.⁸ In some instances, students also have acquired experience with survey research and budget analyses.

While research methods courses often combine formal instruction with field-based projects, the assignments tend to be narrow in scope, carried out as an isolated exercise to practice particular data collection and analysis techniques, and submitted to the instructor as a course paper.⁹ The research apprenticeship is different. Students are full-fledged members of a research team that is contractually bound to deliver a detailed formative evaluation of a major reform to district officials and the broader public. They are involved in weekly seminar sessions and extensive field research for two semesters and in follow-up work during the summer months and beyond. Simply put, the work is prolonged, complicated, and intense; the stakes are high; and, the pressures are real.

Supports

Given its ambitious goals, complex components, and public products, the research apprenticeship requires a substantial investment of faculty resources and related supports. Virtually all of the research apprenticeship projects have been led by two faculty members who have the topical and methodological expertise needed to design and conduct a formative evaluation of the initiative of interest. These apprenticeships also have been supported by two full-time graduate assistants who meet weekly with the faculty to operationalize various phases of the research project and to plan the seminar sessions for students enrolled in the research apprenticeship course. This core group shares its plans for moving the research project forward and the thinking behind those plans with the full research team during the weekly seminar sessions. While this arrangement means that the graduate assistants see the planning process from the ground up while the graduate students enrolled in the research apprenticeship course join that process as critics of and contributors to this preliminary work, all members of the research team are extensively albeit differentially involved in this important aspect of research.

Major Phases

The faculty, along with the graduate assistants, work with students enrolled in the course to produce the formative evaluation. I describe the key phases of this production process below.

Framing the project

At the outset of the research apprenticeship, faculty take the responsibility for grounding the research project in the field of policy evaluation research, the broader policy context, the theory of action undergirding the initiative of interest, and theories of implementation. Such a multi-dimensional framing helps students recognize that, like other “models of evaluation,” the approach we take has strengths, limitations, and trade-offs that we must acknowledge and assess as we move through the research process. It also helps students understand some of the reasons why advocates of the particular reform we are studying view it as a promising approach to school improvement. In short, they get a sense of the initiative’s theory of action and the assumptions that must hold if this initiative is to operate as intended. Furthermore, they are encouraged to view policy implementation as a continuous, adaptive process (Cuban, 1998; Honig, 2006) that can be understood by taking both a top–down and a bottom–up perspective (Elmore, 1982; Sabatier, 1986; Scheirer, 1987). That is, we forecast that we will be studying what the district is trying to do and how their efforts are playing out at the site level.

This orientation to the project alerts students to the fact that they are not only studying a moving target but also embarking on an unpredictable journey. They are forewarned, if you will, that field research is a fluid, iterative, often unpredictable venture that requires researchers to make thoughtful judgments throughout the process; to recognize how those judgments affect the validity, integrity, and utility of the research product; and to adjust their work in light of what they are learning and in light of unanticipated developments in the field. They are alerted to what is for some an unsettling reality—The research apprenticeship course will evolve as the project unfolds. Put differently, the syllabus is not a cut and dried document that depicts specific assignments and due dates. It provides a general outline of how we will pace the work and the major tasks we will focus on each term; it defines, in more detail, the topics and tasks for the first month of the course, but it makes clear that the particulars of class sessions/field research assignments will be defined as we go and that coping with ambiguity and uncertainty is one of the many skills field researchers need to develop.

Framing the project also involves discussing our responsibilities as researchers. At the most basic level, faculty, graduate assistants, and students enrolled in the course are responsible for carrying out the scope of work the course instructors defined with the district. In addition, all members of the research team are bound by the Institutional Review Board code of ethics and the “group agreements” that govern our work. The group agreements relate to the confidentiality of information acquired through participation in the research apprenticeship and to the stance we take as researchers. Simply put, faculty underscore that we seek to learn from the people on the front lines of education reform; we seek to understand what district leaders hope to accomplish by launching the initiative we are studying, how they are supporting (or not supporting) its implementation, and how the initiative is playing out at the site level. Because school personnel are an indispensable source of information and insight but often overburdened, we need to think carefully about how we structure our efforts to capture their perceptions and perspectives.

Getting ready

Having provided a big picture perspective on the project, we turn to the tough task of translating a set of general research questions into a defensible research plan and equipping ourselves to carry out those tasks. I have found Getting the Facts, Jerome Murphy’s classic treatment of field research methods used in program evaluation research published in 1980, to be an invaluable resource for initial discussions of how we might draw upon documents, observations, and interviews to carry out our study and how we might check for bias and error throughout the data collection and data analysis process. In part because Murphy wrote the book when he was a novice researcher who had taken on the awesome responsibility of assessing prominent programs, this resource confronts the anxieties most students in the course are experiencing, provides strong guidance for tackling the complexities of field-based evaluation research, and underscores that one hones field research skills by working closely with more experienced scholars and by relying on preparation, experience, and reflection, precisely the approach taken in this course. Other resources (e.g., Merriam, 1998; Patton, 2002; Weiss, 1998; Yin, 1995) are also helpful as we work to refine the research questions, assess design options, finalize design choices, and develop data collection priorities, protocols, time tables, and access strategies. Since all research designs and data collection strategies have strengths and limitations, part of the challenge of these “getting ready” sessions is to help the team reach consensus on options that are well suited to the questions we are asking and the resources we have at our disposal and to do so in a timely manner.

Diving in

Because the research apprenticeship seeks to integrate formal instruction in research methods with actual experience using those methods and to complete a formative evaluation for public consumption, students dive into field research after the first class session.¹⁰ We typically begin with documents that provide information about the district context and the initiative’s design and with observations of initiative-related training sessions and management meetings. We also schedule, early on, interviews with district officials who are in a position to help us understand the purpose of the initiative, the process through which it was formulated, and the strategies employed to support its implementation. Doing so provides a real incentive to learn as much as we can as fast as we can from the documents, to work through (and become comfortable with) our interview protocols, and to clarify the district’s reasons for launching this initiative and its role in implementation. We carry out some of those interviews in class so that all students can meet key district officials, observe the interview process, and, upon completion, discuss how the interview was carried out, what we can learn from it, and what additional information we may need to secure. We conduct most of these early interviews in the field. Sometimes, an instructor carries out one or two of these initial interviews to model particular skills; more often, two students take responsibility for these interviews, practice with one of the course instructors, and conduct the interviews while the course instructors adopts a “fly on the wall” role. Whenever possible, students who are not responsible for conducting these interviews attend them to see, firsthand, how the interviews were handled and to participate in the debriefing/feedback sessions that follow each of these interviews. All members of the research team are responsible for listening to the tapes, reading the interview logs/transcripts, and contributing to the analysis of those interviews.

As data collection progresses, seminar sessions focus on making sense of the district-level data we have collected and identifying the district-level information we will continue to collect as the study progresses. For example, students develop and critique analytic memos that crystallize what we think we know about the initiative’s key features (purpose, provisions, underlying theory of action), the impetus for it, the process through which the district formulated it, the steps the district has taken to support its implementation, and the issues that are surfacing. We begin distilling key findings about the district’s role in the formulation and early implementation of the initiative of interest, arraying/assessing the confirming and disconfirming evidence, and jotting notes for the follow-up or “year-end” interviews we conduct to round out our understanding of the district’s take on implementation. In addition, we formulate plans for studying how the initiative of interest is playing out in schools. We work through site selection and site entry issues as well as preliminary plans for data collection at the school level.

Digging deep

In each of the research apprenticeships, we have developed in-depth case studies of implementation in multiple sites led by reputedly effective principals. That screen helps us keep the focus on the initiativerather than the competence of the principal, and increases the chances that we are giving the initiative a full and fair “test.” To carry out this phase of research, the instructors divide the class members into smaller student-led teams that focus on one of the case study sites so that students can concentrate on developing a deeper understanding of how the initiative is playing out in that particular setting. As one would expect, we begin our work in the schools by interviewing the principals to capture their perspectives on the initiative and to get their best advice for how we might set up interviews with faculty, staff, and, where appropriate, parents and community residents. At this point in the course, we dedicate the weekly seminar sessions to formalizing plans for collecting site-level data, debriefing the students’ field research experiences, developing/critiquing analytic memos that capture what we are learning, and creating a rubric to guide the development of case narratives that display the data and address, in explicit terms, our research questions. In some instances, we have augmented these data with surveys of school personnel and individual or focus group interviews with a broader sample of principals to gauge whether the themes and patterns we uncovered in our case study sites were evident in other settings. However, the site-level data collection is handled, we try to complete that process before testing begins in mid-March so we do not disrupt the schools during these tense times and so we can concentrate on analyzing our data and crystallizing our key findings.

Because these in-depth case studies increase the number and complexity of the research tasks to be accomplished, the apprenticeship experience shifts from formal instruction and a graduated, directly supervised exposure to various forms of field research to a more intense, indirectly supervised immersion in field work. That is, faculty accompany students to the sites of study, participate in some data collection activities, and check in with principals to gauge how school personnel are responding to the researchers’ presence at their sites. But faculty handle most of the “supervision” of field research by listening to the interview tapes, reviewing students’ field notes, and providing detailed feedback on their work. As this description implies, students may be “on their own” in the field. Instructors are always available by phone, but the students operate with less direct supervision, more independence, and greater responsibility for the field research tasks. At this stage of the study, we dedicate virtually all of the seminar sessions to analyzing the site-level data, developing school-specific case narratives and cross-case analyses of site-level implementation, and linking our work to ideas in the research methods materials we read early on.

Stepping back

Like more seasoned field researchers, graduate students often find collecting data more exciting than writing the research report. So a key aspect of the research apprenticeship involves helping students recognize the importance of stepping back from their field experiences so that they can interrogate the data and integrate the streams of work (district-level design and implementation efforts and site-level implementation patterns) into a coherent report that offers the district a clearer picture of how a particular initiative has unfolded and how that initiative might be strengthened. We all find it hard to distance ourselves from the invigorating human interactions that are such a defining feature of field research, so we talk through how to exit the study sites, express our appreciation for educators’ participation in the study, arrange for principals of these schools to review the penultimate draft of our research report, and get copies of the final report to the study sites.

Communicating results

The report we develop at the close of the two-semester course is often a good start but not a finished document. That is, the research team has worked through a detailed outline of the report; individuals and groups have drafted sections and received feedback on that work. But first cuts are never final cuts. Consequently, students who wish to be part of efforts to finalize the research report sign up for independent study credits and work with faculty to complete the report during the summer months. All students who enrolled in the two-semester research apprenticeship course review the penultimate research report that the research team submits to the district, so they can see the “finished” fruits of their labors and catch any errors we may have made. In addition, all students have the opportunity to be an integral part of presentations/discussions of the study at district meetings as well as co-authors of conference papers and published articles that flow from this research.

Composing the final report is arguably the most challenging and the most powerful activity in the research apprenticeship course because it requires that students draw on multiple sources of data to distill responses to a set of research questions, reconcile conflicting evidence, develop convincing lines of argument based on thorough but imperfect data sets, and write in an incisive, engaging style. The multiple spin-off opportunities to disseminate findings through presentations for local and national audiences are also valuable. For example, students are involved in efforts to share the preliminary findings with key district administrators and staff members through informal debriefing sessions held throughout the research process and to share the final results with district administrators, the board of education, and site-level personnel when the research project is completed. In addition, students often take the lead in presentations at national conferences. Through these opportunities, students begin to see how carefully crafted formative evaluations of actual reform efforts can be translated into information sessions with district officials as well as conference papers and publications in top-flight journals (e.g., Educational Evaluation and Policy Analysis, Educational Administration Quarterly, Educational Policy). Students often tell us that working shoulder to shoulder with faculty to develop manuscripts based on research that students have been meaningfully involved in producing helps them develop a clearer understanding of the demands of scholarly writing and the possibilities of doing work that is immediately useful to school systems and respected in the academy and beyond. District leaders also tell us that the research reports and debriefings have helped them get a clearer picture of “what was happening” and precipitated adjustments in both the design and the implementation of the initiatives we studied.

The Research Apprenticeship as Potentially Promising Pedagogy

Embedded in the description of the research apprenticeship course is a primitive but illustrative model of the underlying process through which this unique course seeks to develop the research skills and dispositions identified in the goals of this course. Summarized in Figure 1, this model suggests that the faculty’s theoretical orientations and methodological stance set the parameters within which the research team determines how the formative evaluation will be carried out. This model also suggests that the development of the desired research skills and dispositions occurs through continuous and ever-expanding cycles of preparation, execution, and reflection.

Figure 1.

The research apprenticeship as promising pedagogy.

In the case I have described, the course instructors focus on formative evaluation research because that is a form of research that students interested in policy and leadership tend to view as important and need to understand, especially if they seek to take on leadership roles in public schools (Shulman et al., 2006). Faculty ground the formative evaluation research in the initiative’s underlying theory of action and view implementation as an adaptive process shaped by the design of the initiative, the context in which it is launched, and the efforts of both district officials and site educators to fit an often complex initiative into the rhythms, routines, and regularities of schools. Faculty emphasize that our intent is to learn from those directly involved with and affected by the initiative of interest and to generate insights regarding how the initiative might be strengthened.¹¹ Others who elect to develop a research apprenticeship course might ground their work in different theoretical orientations and methodological traditions. The point here is simply that those perspectives set the parameters for decisions regarding how the research project gets carried out.

But how do students acquire the research skills and dispositions this research apprenticeship seeks to develop? The model posits that students learn through the multiple cycles of preparation, execution, and reflection that lie at the heart of the research apprenticeship experience (see also, Bowers, in pressl issue). Throughout the research apprenticeship, students prepare for an array of tasks related to the collection and analysis of data and the communication of key findings. Recall, they are repeatedly involved in data collection activities such as determining what types of data we need to address our research questions, developing data collection protocols, and piloting those instruments with members of the research team. Students repeatedly execute their data collection plans through intensive, interrelated field research activities (notably, document reviews, observations, and interviews). Having completed a particular data collection task, they reflect on what they are learning about the initiative and on what they are learning about the research process and themselves as field researchers. Those reflections inform the next round of preparation. Throughout the research apprenticeship, students also develop plans for analyzing and communicating their data. Moreover, they repeatedly execute those plans as they interrogate their data; array what Yin (1995) terms the “chains of evidence” that support and challenge emergent interpretations; develop analytic memos, case narratives, and the final report; practice for presentations of findings to particular audiences; and convert the research report into conference papers and journal articles. During these phases of the research process, they continue to reflect on what they are learning about the initiative, the research process, and themselves as researchers.

The model posits that the cumulative impact of the multiple, interrelated and ever-expanding cycles of preparation, execution, and reflection is that students become more thoughtful and more proficient researchers. It reflects the earlier noted assumption that aspiring scholars can learn a lot about research methods, particularly field research methods, by carrying out studies with experienced scholars (Murphy, 1980).

The model is generally consistent with theories of “authentic pedagogy” (Newman, Marls, & Gamoran, 1995) in that students produce as well as consume knowledge through active and sustained engagement in disciplined inquiry. They draw on prior knowledge and the extensive original research they carry out to develop a more fulsome understanding of an educational reform in its “real-world” context. That arguably thorough, iterative, and contextualized process yields work of value beyond the successful completion of a course requirement. The model is also consistent with broader theories of situated learning (Brown, Collins, & Duguid, 1989; Lave, 2016; Wenger, 2007) in that individuals with shared interests in a particular domain (in this case, education research and education reform) engage in salient tasks and learn through regular social interactions. In different words, students participate in a “community of practice” that helps them hone their craft (Lave & Wenger, 1991; Wenger, 2007). Over the course of the apprenticeship, they move from more “legitimately peripheral” members of that community to more central players capable of carrying out more complex work (Lave & Wenger, 1991).

Whether this primitive model fully accounts for the student learning we observe is an open, empirical question. My colleagues and I have not systematically examined how the various components of the research apprenticeship might combine and interact with other arguably relevant factors (e.g., the time students have to invest in the work, students’ prior knowledge and research experience) to affect the development of students’ research skills and dispositions. Our assessments are more informal. That is, we have watched anxious and inexperienced students enhance their understanding of the logic and limits of formative evaluation research and carry out key aspects of that research with competence and care. To be sure, some students acquire stronger sets of skills than others, but the growth students demonstrate over the course of the study is evident not only to those of us who teach the course but also to study participants (notably, district administrators and building principals) who interact with members of the research team over the course of the study and to the students themselves. Faculty and study participants’ observations, student self-assessments, and faculty reviews of students’ contributions to the research project and related presentations and publications suggest that the research apprenticeship is a potentially powerful pedagogical strategy for enhancing the preparation of individuals with diverse career aspirations but genuine interest in education research as a means of enhancing the capacity of school systems to improve.

Whether this model fosters district learning is also an open, empirical question. My colleagues and I have not systematically tracked how and why the district uses (or decides not to use) the research findings internally or with broader publics. Here too, our assessments are informal. That is, in private conversations, public meetings, and formal letters, district officials have offered comments that suggest the research projects provided “invaluable” information, generated “insights into issues” that inevitably accompany the implementation of education reforms, “led to local changes,” and “advanced core initiatives” in the district. Although these broad statements suggest the research apprenticeship may foster organizational learning, whether and how that might be happening, given all the issues districts face and all the factors that shape what districts may and must do, is a matter that warrants more careful attention than it has received.¹²

A Cautiously Optimistic Endorsement

While those of us who have been part of the research apprenticeship course and its spin-offs view it as a promising approach to the preparation of both practitioner-scholars and education researchers, we also recognize that it poses real challenges. First, the apprenticeship has a lot of moving parts. In general terms, it involves (a) cultivating relationships with district officials and site educators, (b) conceptualizing a research project, (c) teaching graduate students with varied backgrounds and career interests about multiple facets of the research process, (d) developing a team that can carry out research responsibly, (e) ensuring that each member of that team gains rich experiences with multiple aspects of field research, (f) creating opportunities to acquire data in a relatively efficient yet unobtrusive manner, (g) analyzing mounds of data, and (h) completing a study that passes muster in a tight time frame. Because orchestrating, assessing, and integrating all those pieces can overwhelm faculty and students, all members of the research team need to be ready to invest heavily in all dimensions of this unique course offering. Second, given the demands of this approach, it may be hard to institutionalize. While my colleagues and I have been able to offer four rounds of the research apprenticeship course, faculty resources are finite; competing demands are plentiful. Offering the research apprenticeship even on an every-other-year rotation may not be feasible. Third, the approach is not well suited to the abbreviated “executive degree” weekend workshops and online formats that are gaining currency in the field of education leadership. The research apprenticeship requires students to invest a considerable amount of time over a calendar year in face-to-face interactions with study participants and research team members and to carve out concentrated periods of time to think hard about what the data mean and how the data might be communicated. Put differently, the research apprenticeship is not easily condensed or readily broken up into modules; its power resides in the extensive and intensive work associated with systematic efforts to capture the complexities of education reforms as they unfold in school systems. Finally, formative evaluations are risky forms of research. They often expose problems that can be embarrassing and destabilizing for school systems and unsettling for researchers and the institutions they represent.

On the more positive side, district officials have conveyed their support of the research apprenticeship, and student interest remains high. Part-time and full-time students in the policy and leadership graduate programs report that they appreciate the opportunity to be immersed in research that focuses on real issues in real school systems and engaged in ambitious projects that enable them to develop their ability to carry out a salient form of research. Equally important, calls for universities in general and colleges of education in particular to take greater responsibility for the function and fate of school systems are persistent and pervasive. Various organizations and commissions summon higher education systems to become publicly “engaged universities” that redesign their “teaching, research, and extension and service functions to become even more sympathetically and productively involved in their communities” (Kellogg Commission on the Future of State and Land-Grant Universities, 1999:9, cited in McCroskey, 2003, p. 125; see also Benson & Harkavy, 2003; Tate, 2007). The research apprenticeship represents a step in that direction. It pivots on a partnership of mutual respect and mutual benefit that situates a university-based research course in the work of schools as well as in our knowledge of research methods. Through this integration, the research apprenticeship enriches the preparation of both educational researchers and practitioner-scholars.

Footnotes

Author’s Note

I am indebted to Kristin Sinclair, Chad Lochmiller, and three anonymous reviewers for their thoughtful comments on earlier versions of this manuscript.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biography

Betty Malen, a former school administrator, is a professor in the Division of Education Policy and Leadership and a Distinguished Scholar Teacher at the University of Maryland. She is broadly recognized for her research on the political determinants and the substantive effects of prominent education reforms (e.g., decentralization, school reconstitution, high-stakes accountability policies, pay-for-performance initiatives) and on the micro-politics of schools. She teaches courses on policy, politics, and research methods (e.g., case study designs, policy/program evaluation) and has received multiple awards for her scholarship, teaching, and mentoring.

References

Anderson

G. L.

(2002). Reflecting on research for doctoral students in education. Educational Researcher, 31(7), 22-25.

Ball

D. L.

Forzani

F. M.

(2007). What makes education research “educational”? Educational Researcher, 36, 529-540.

Benson

Harkavy

(2003). The role of the American research university in advancing system-wide education reform, democratic schooling, and democracy. In Brabek

M. M.

Walsh

M. L.

Latta

R. E.

(Eds.), Meeting at the hyphen: Schools-universities-communities-professions in collaboration for student achievement and well being (pp. 94-114). Chicago, IL: University of Chicago Press.

Bowers

A. J.

(in press). Quantitative research methods training in education leadership and administration preparation programs as disciplined inquiry for building school improvement capacity. Journal of Research on Leadership Education.

Brabeck

M. M.

Walsh

M. E.

Latta

R. E.

(2003). Meeting at the hyphen: Schools-universities-communities-professions in collaboration for student achievement and well being. Chicago, IL: University of Chicago Press.

Brown

J. S.

Collins

Duguid

(1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32-41.

Coburn

C. E.

(2010). The partnership for district change: Challenges of evidence use in a major urban district. In Coburn

C. E.

Stein

M. K.

(Eds.), Research and practice in education: Building alliances, bridging the divide (pp. 167-182). Lanham, MD: Rowman & Littlefield.

Coburn

C. E.

Stein

M. K.

(2010). Research and practice in education: Building alliances, bridging the divide. Lanham, MD: Rowman & Littlefield.

Cuban

(1998). How schools change reforms: Redefining reform success and failure. Teachers College Record, 99, 453-477.

10.

Eisenhart

DeHaan

R. L.

(2005). Doctoral preparation of scientifically based education researchers. Educational Researcher, 34(4), 3-13.

11.

Elmore

R. F.

(1982). Backward mapping: Implementation research and policy decisions. In Williams

(Ed.), Methodological and administrative issues (pp. 36-72). Chatham, NJ: Chatham House.

12.

Fowler

F. J.

(1993). Applied Social Research Methods Series: Vol. 1. Survey research methods (2nd ed.). Newbury Park, CA: Sage.

13.

Goldring

Sims

(2005). Modeling creative and courageous school leadership through district-community-university partnership. Educational Policy, 19, 223-249.

14.

Gutierrez

K. D.

Penuel

W. R.

(2014). Relevance to practice as a criterion for rigor. Educational Researcher, 43, 19-23.

15.

Harkavy

Puckett

J. L.

(1991). Toward effective university-public school partnerships: An analysis of a contemporary model. Teachers College Record, 92, 556-581.

16.

Honig

(2006). Complexity and policy implementation: Challenges and opportunities for the field. In Honig

M. I.

(Ed.), New directions in education policy implementation: Confronting complexity (pp. 1-23). Albany: State University of New York Press.

17.

Hubbard

(2010). Research to practice: A case study of Boston Public Schools, Boston Plan for Excellence and Education Matters. In Coburn

C. E.

Stein

M. K.

(Eds.), Research and practice in education: Building alliances, bridging the divide (pp. 183-200). Lanham, MD: Rowman & Littlefield.

18.

Lave

(2016). Situated learning theory. Retrieved from http://www.learning-theories.com/situated-learning-theory-lave.htlm

19.

Lave

Wenger

(1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: University of Cambridge Press.

20.

Levine

(2007). Educating researchers. Washington, DC: The Education Schools Project.

21.

Malen

Basinger

Gonzalez

Nutter

with assisstance from Muncey

D.E.

Hildreth

Seelke

Susskind

Vincent

(2007). Urgent work in progress: The first year of the Intensive Supports and Intervention Schools Initiative. College Park: University of Maryland.

22.

Malen

Curtis

Sinclair

Croninger

R. G.

Garcia

Egan

(2013). In pursuit of equity, autonomy and improvement: A study of the student-based budgeting initiative in the Prince George’s County Public Schools. College Park: University of Maryland.

23.

Malen

Dayhoff

Egan

Croninger

R. C.

(2015). The challenges of advancing fiscal equity in a resource-strained context. Education Policy. Advance online publication. doi:10.1177/0895904815613445

24.

Malen

Rice

J. K.

Bauman

Beaner

Chen

Dougherty

. . .Robinson

(2009). Building the plane in flight: A formative evaluation of Financial Incentive Rewards for Supervisors and Teachers (FIRST). College Park: University of Maryland.

25.

Malen

Rice

J. K.

Bowsher

Bivonia

Hoyer

K. M.

Hyde

(2015). Developing organizational capacity for implementing complex education reform initiatives: Insights from a multi-year study of a Teacher Incentive Fund (TIF) program. Educational Administration Quarterly, 15, 133-176.

26.

Malen

Rice

J. K.

Jackson

Hoyer

K. H.

Hyde

Bivona

… Sutter

(2011). Implementation, payouts, and perceived effects: A formative analysis of Financial Incentive Rewards for Supervisors and Teachers (FIRST). A research monograph for the Prince George’s County Public School System, Prince George’s County, MD.

27.

McCroskey

(2003). Challenges and opportunities for higher education. In Brabek

M. M.

Walsh

M. L.

Latta

R. E.

(Eds.), Meeting at the hyphen: Schools-universities-communities-professions in collaboration for student achievement and well being (pp. 117-135). Chicago, IL: University of Chicago Press.

28.

Merriam

S. B.

(1998). Qualitative research and case study applications in education. San Francisco, CA: Jossey-Bass.

29.

Metz

M. H.

(2001). Intellectual border crossing in graduate education: A report from the field. Educational Researcher, 30(5), 12-18.

30.

Murphy

J. T.

(1980). Getting the facts: A fieldwork guide for evaluators and policy analysts. Santa Monica, CA: Goodyear.

31.

Newman

F. M.

Marks

H. M.

Gamoran

(1995, April). Authentic pedagogy and student performance. Paper presented at the annual conference of the American Education Research Association, San Francisco, CA.

32.

Patton

M. Q.

(1990). Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.

33.

Patton

M. Q.

(2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

34.

Penuel

Fishman

B. H. C.

Sabelli

(2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40, 331-337.

35.

Rice

J. K.

Malen

Baumann

Chen

Dougherty

Hyde

. . . McKithen

(2012). The persistent problems and confounding challenges of educator incentives: The case of TIF in Prince George’s County, Maryland. Educational Policy, 26, 892-933.

36.

Rice

J. K.

Malen

Jackson

Hoyer

K. M.

(2015). Time to pay up: Analyzing the motivational potential of financial awards in a TIF program. Educational Evaluation and Policy Analysis, 37, 29-49.

37.

Rice

J. K.

Malen

Jackson

Hoyer

K. M.

(2016). Administrator responses to financial incentives: Insights from a TIF program. Leadership and Policy in Schools. doi:10.1080/15700763.2016.1205201

38.

Sabatier

P. A.

(1986). Top-down and bottom-up approaches to implementation research: A critical analysis and suggested synthesis. Journal of Public Policy, 6, 21-48.

39.

Scheirer

M. A.

(1987). Program theory and implementation theory: Implications for evaluators. New Directions for Evaluation, 33, 59-76.

40.

Schensul

J. J.

LeCompte

Nastasi

B. K.

Borgatti

S. P.

(1999). Enhanced ethnographic methods: Audiovisual techniques, focused group interviews, and elicitation. Walnut Creek, CA: AltaMira Press.

41.

Shavelson

R. J.

Towne

(Eds.). (2002). Scientific research in education (Report of the National Research Council’s Committee on Scientific Principles for Education Research). Washington, DC: National Academies Press.

42.

Shulman

L. S.

(1988). Disciplines of inquiry in education: An overview. In Jaeger

R. M.

(Ed.), Complementary methods for research in education (pp. 3-19). Washington, DC: American Educational Research Association.

43.

Shulman

L. S.

Golde

C. M.

Bueschel

A. C.

Garabedian

K. J.

(2006). Reclaiming education’s doctorates: A critique and a proposal. Educational Researcher, 35(3), 25-32.

44.

Stipek

(2007, November 14). Bridging the research-practice gap. Education Week, p. 19.

45.

Stufflebeam

D. L.

(2000). Professional standards and principles for evaluations. In Stufflebeam

D. L.

Madaus

G. F.

Kellaghan

(Eds.), Evaluation models: Viewpoints on educational and human services evaluation (pp. 439-455). Boston, MA: Kluwer Academic Press.

46.

Tate

W. F.

(2007). Linking education research and civic responsibility: Why now? Educational Researcher, 36, 408-409.

47.

Weiss

C. H.

(1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.

48.

Wenger

(2007). Communities of practice: A brief introduction. Retrieved from http://www.ewenger.com/theory/