Abstract
This issue highlights problems with the neoliberal research regime in education policy—one that marginalizes, homogenizes, and dismisses communities, critical voices, and methodologies that could improve educational practice. We argue for greater methodological diversity in policy studies when engaging in research conceptualization, applying theoretical frameworks, collecting data, undertaking data analysis, and critiquing the researcher’s epistemological standpoint and positionality in order to redress these deficits in understanding and representation. Indeed, we suggest that qualitative research plays an important role in also contesting the positivist regime in that it empowers those affected by policy, particularly voices often placed in the margins of policy, and contests a methodological hegemony that privileges a neoliberal paradigm in education. Each article in this issue situates a methodological approach within the praxis of education research, describes an essential departure from the quantitative “gold standard” in policy analysis and research, and provides pedagogical evidence of its application to a specific policy. The introduction of this special issue underscores the need for diverse methodological approaches to understanding the complexity of education policy, provides overviews of the articles, and ends by considering the significance of the issue to researchers, policymakers, and practitioners.
Increasingly, policy researchers are calling for new approaches to support the analysis of complex educational policies. Although quantitative research dominates public policy, it is not without its critics and limitations. Concurrently, the legitimacy of qualitative research has grown but it is not yet accepted within public policy discourse. Thus, the dominance of quantitative research in public policy is both a political issue and a methodological issue. One of the challenges of public policy research is the question of what constitutes rigorous and valuable research that addresses the persistent issues in education. The prevailing answer is experimental and quasi-experimental designs that support causal inference, generalizability, and replication. Innovative quantitative and mixed methods approaches are favored while rigorous and creative qualitative methods have been overlooked, despite arguments to the contrary. In writing this introduction, we build on the existing literature that explores the epistemological origins and applications of positivist claims and extend the growing literature that critiques the hegemonic focus of much policy studies, with a focus on education research in particular.
This special issue will explore the problems associated with this quantitative regime in education policy research and analysis and offers policy researchers alternative approaches to the study of education policy while providing meaningful methodological details on how to engage with these approaches in order to be practically useful to both novice and experienced researchers. The issue is broadly broken down into the core inquiry processes of data collection and data analysis, as well as a broader examination of theoretical frameworks, paradigmatic issues, and collaborative approaches relevant to qualitative public policy inquiry. Each article argues for the methods it describes and includes one to two cases to which the methods are applied in order to provide the practical instructional details often missing in empirical research. Collectively, the articles herein represent creative and rigorous qualitative approaches to policy inquiry while also representing the voices and perspectives of new scholars, women, women of color, and marginalized communities.
Setting the Stage
Public policymaking refers to the actions taken by the government to solve public problems; it is a political process that requires conflict, negotiation, the use of power, bargaining, and compromise (Anderson, 2011, p. 5). Researchers argue that public policy needs qualitative research in order to understand the “complex behaviours, needs, systems and cultures” of policy settings (Ritchie & Spencer, 2002, p. 305). Furthermore, they position qualitative research as the solution to methodological and practical problems including the critique that research is inaccessible and unconnected to practice (Sallee & Flood, 2012). Qualitative research can provide context, understanding, depth, comparison, and voice (Tierney & Clemens, 2011) and can thereby lay the foundation for arguments for new methodological approaches (Donmoyer, 2012a, 2012b; Donmoyer & Galloway, 2010; Maxwell, 2012).
“Rigorous” Research
Research in the social sciences has suffered under increasing threats to the independence of researchers and to the varying ways in which they engage with social research (Atkinson & Delamont, 2006). Funding agencies, politicians, and the academy itself all account, to varying degrees, for the dismissal of diverse approaches as rigorous forms of inquiry. Morse (2006) suggested that this stems from a 1972 model to evaluate research rigor developed by Archie Cochrane in the health sciences, which relegated research that was not a randomized trial to a category of studies that should not inform practice. Furthermore, this assertion prompted a political reaction that resulted in the exclusion of qualitative research from the resources (i.e., funding) provided to research. “Quantitative researchers now largely are able to lay claim to the policy and practice arena and make use of hundreds of millions of dollars in governmental and foundation funding” (Tierney & Clemens, 2011, p. 58). But are policymakers really able to make sense of the research that emerges? Although most (but perhaps not all) politicians can count, there is little evidence that they understand the output and limitations of, for example a logistic regression. Nonetheless, this looks like “science,” can be taken to be proof and serves as magic to underpin stories of progress on the basis of policy initiatives. (Byrne, Olsen, & Duggan, 2009, p. 520)
Despite this, in 2002, the National Research Council acknowledged qualitative inquiry as a form of scientific research, but underscored that the gold standard of research was replication and generalizability, née quantitative and experimental. This has established a methodological conservatism in education and a hostile political climate to qualitative research (Denzin, 2009), often silencing researchers on the margins—queer, feminist, disabled, and/or indigenous (Cacciattolo, Vicars, & McKenna, 2015).
The Neoliberal Influence
Questions of rigor and funding regimes in education research suggest that evidence is both socially and politically constructed. “This political economy of evidence . . . is not a question of evidence or no evidence, but who controls the definition of evidence and which kind is acceptable to whom” (Larner, 2004, p. 20). The performative, operational definition of evidence may be attributed to the neoliberal orthodoxy now governing public education in most countries, both developed and developing (Ball, 1998). Harvey (2005) defined neoliberalism as a political-economic practice that favors free markets and limited government engagement in social provisions; where markets do not exist (i.e., education), they must be created in order to ensure “the quality and integrity of money” (p. 2) and promote a consumerist turn within this marketplace.
The Bush administration and NCLB (No Child Left Behind) firmly instituted neoliberalism into public sector practice (Hursh, 2007). This has informed even the way we understand and engage in educational practice, proliferating quantitative measures into educational institutions: “Under neoliberal reform, schools are mandated to increase the number of assessments they administer and are penalized or rewarded according to student performance,” affecting the funding such institutions receive (Brathwaite, 2017, p. 430). Neoliberal political discourse frames minoritized communities as broken and that favored education policies will fix these communities (Baldridge, 2014; Brathwaite, 2017, among many others). “Considering the current broader political and educational climate, it is imperative that researchers challenge the persistent narrative and trend toward ‘damage-centered’ and deficient framing to include more humanizing methods . . .” (Baldridge, 2014 p. 467). However, research that contests the deficit models inherent to such discourses will no doubt suffer under its funding regimes. In his article, “Show Me the Money! Neoliberalism at Work in Education,” Ball (2012) asserted that we can thank “funding for advocacy, ‘research’ and ‘influence’ activities in making neoliberalism thinkable, possible, obvious” (p. 26) in education, from the government, philanthropic organizations, and policy entrepreneurs who play key roles in “dollarizing” education. And there have been unintended consequences. Funding, as a mechanism of neoliberal control in education, has forced academic researchers to dismiss methodological limitations of social science research (Donmoyer & Galloway, 2010) and overestimate the impact of their research in order to obtain highly competitive, and scarce, research money (Chubb & Watermeyer, 2017); increasingly, a replication controversy is brewing in published research, in line with what Lather (2004, 2012) referred to as the “legitimation crisis of neoliberalism.”
Methodological Weakness
These political and discursive interventions in education, however, are not solely to blame for the dismissal of qualitative inquiry. Some of this blame must be apportioned to researchers themselves.
There are four weaknesses that have affected American qualitative research in education since 1971 . . . a narrow definition of education; ethnocentrism; a neglect of the long history of qualitative methods; and the use of “quick fix,” “data to go” or “smash-and-grab” methods of data collection instead of intensive fieldwork. (Atkinson & Delamont, 2006, p. 749)
Education research is often perceived to be of low quality, burdened by weak theory, opacity, bias, and “anecdote masquerading as evidence” (Feuer, Towne, & Shavelson, 2002, p. 5). Suggesting education research be more like medical and engineering research, Burkhardt and Schoenfeld (2003) stated that educational research had limited utility because it could not prove impact: “The point is that most such studies, while rigorous enough to be published and while providing some form of insight, tend not to provide adequate information to allow for replication and extension” (p. 8). The authors make some conciliatory gestures toward qualitative research by advising “that a consensus on findings need not come at the expense of methodological pluralism” (p. 10). Yet, despite this suggestion, qualitative educational research is increasingly colonized by positivist assumptions regarding appropriate sample sizes, rigorous methods, and impact, despite the methodological turn in some health research toward qualitative, descriptive studies (see Green & Thorogood, 2018).
Today’s policy discourse prioritizes quantitative data as evidence—the only kind of evidence—and in so doing, perverts other human sciences. Research, then, is no longer rooted in methodological and disciplinary traditions, but rather becomes an atheoretical, aphilosophical, and amethodological process of data collection and reporting, absent of any interpretation or context (Asen, Gurke, Conners, Solomon, & Gumm, 2013). For example, research in education may engage with data produced from a local schooling context, but this engagement is often troubled by messages about evidence and data within the broader context of the technologization of education policy (Ball, 1998). Without sufficient guidance, researchers must manage the tension between evidence and context by essentializing phenomena in pursuit of the unnecessary requirement of generalizability as a standard of validity, ultimately weakening the research and its implications for understanding policy and its outcomes.
New Directions
This special issue contends with these challenges to conducting qualitative inquiry within the neoliberal regime currently dictating the terms of much education policy research and analysis. Each article provides a substantive critique of this regime and offers evidence of when a particular qualitative approach improved our understanding of a specific education policy. By attending, comprehensively, to the descriptions of how these approaches are applied to one or two cases per article, this issue will serve as a unique resource to researchers seeking new approaches to meet the complex needs of public policy research. We argue that although quantitative research now dominates public policy, there is a need not only for qualitative approaches but also a need to rigorously detail such approaches for broader use. The intention is not to replace quantitative methods; rather, our intention is to expand the toolbox of available techniques.
Due to the heterogeneity (and confusion) with which people discuss research in public policy and policymaking, this issues adopts Fischer’s (2003) perspective on the various ways of labeling public policy studies: “The term ‘policy studies’ is used here to designate the general field of inquiry. And ‘policy research’ is employed in this work to refer to the primary activity of scholars engaged in policy studies” (p. 1). We also employ the term policy inquiry to capture the interpretive turn in policy studies that includes context in order to explore the social meaning of policy (Fischer, 2003). This approach centers the researcher in the research process. This is important because of the aforementioned conservatism that has silenced policy researchers on the margins. To address this, in this issue, we feature women, women of color, pre-tenure researchers, critical theoretical perspectives, and research on education policy that affects marginalized communities. This discussion in this special issue is representative of a turn away from methodological purism; in so doing, we challenge the “evidence-based” coda proliferating much social science research.
The first article in the issue, authored by Kerrigan and Johnson, demonstrates the value of qualitative time-series analysis to public policy inquiry and presents two case studies that employ this analysis approach in order to provide guidance to other researchers interested in studying policy over time. By focusing on two disparate but complementary cases, the authors illustrate the use of qualitative time series with data displays to establish a type of causality in the policy-making process (at both the state and institutional level and the interaction between these levels) by uncovering direct associations between events, actors, and policies.
The second article, authored by Natow, focuses on the use of online data sources and research methods to explore policymaking, with a particular focus on regulatory policy due to its influence on everyday life. Natow assists the reader in conceptualizing online data sources, differentiating between passive and interactive uses of the web, understanding the different types of policy inquiry and how online data sources can assist the researcher, and establishing protocols for trustworthy, credible online data usage in policy studies. She deftly applies all of these notions to one regulatory policy in particular, the Borrower Defense Rule in the United States, and exhorts readers to expand their toolboxes, critically and carefully, when researching policy.
In the third article of this issue, Sam argues the need to explore local narratives to understand the public’s perception of important policies in education through Foucauldian discourse analysis, particularly as these narratives evolve over social media. She suggests that this approach enables us to better understand how the recipients of policy make sense of, support, and/or contest it as they engage with policy stories online. Using the Common Core State Standards, she carefully lays out her research procedures and provides compelling examples of Common Core narratives on Twitter and their influence on the success of the policy, using Foucauldian discourse analysis.
The fourth article by Kimball explores the use of the extended case method to understand policy outcomes related to federal financial aid, specifically through the analysis of a college preparatory program targeting low-income and racially minoritized youth. Kimball asserts that the extended case method, as a qualitative approach, is capable of providing verifiable casual connections between policy interventions and outcomes through the use of social theory and the (re)creation of a logic model that explains the goal of policy. Kimball skillfully demonstrates how the focus on one college preparatory initiative, the Conscious of College program, enables a complex, localized exploration of federal financial aid policy in contrast to the homogenization of experiences that often emerges from related quantitative policy work.
In her article, Martinez takes up the question: How can higher education scholars employ NPA to examine the complexities of public policy issues when singular narratives dominate the public or policy discourse even though rational counterarguments exist that policy actors ignore or delegitimize? In doing so, she demonstrates her approach to applying narrative policy analysis (NPA) to Nevada higher education policy as a way to capture the implicit narratives that influence the policy process. In addition to discussing the methods and tools for NPA, she shares her own insights for novice policy researchers that will prove helpful to anyone undertaking this approach. Her thoughtful elaboration on the NPA process, including a layered analysis that enabled a nuanced understanding of how policy actors use symbolic language, objects, and acts in crafting policy and its meaning makes explicit the underlying narratives.
The sixth article by Golan et al. details the challenging processes of recruiting participants, maintaining ethical standards, and selecting appropriate technology in service of video ethnography for the New Jersey Families Study. While these logistics might otherwise dissuade researchers, this article addresses the practical considerations that such a project entails. The authors present video ethnography as a method that captures the contents of daily life while deftly addressing common critiques of validity and generalizability. Particularly noteworthy for any researcher today is their discussion of how they addressed ethics for institutional review board requirements, and specifically how to maintain confidentiality and privacy and also secure informed consent.
Jones, the seventh and final author in the special issue, paints a compelling portrait of a new researcher struggling with the implicit, postpositivist paradigms at a play in a large-scale qualitative educational policy project she participated in as a student, postdoc, and new faculty member. She highlights lessons from the project that illuminated her own emergent personal paradigmatic assumptions, related to critical perspectives, and how these created tensions for her in the creation of the conceptual framework, the sampling of participants, the collection of data, and the analysis of that data. She ends by providing a series of questions to assist neophyte and seasoned researchers to consider as they engage in policy research and encouraging graduate programs to train students in diverse methodological approaches.
By attending to the research process—including data collection, data analysis, and the paradigms that inform the specific methodological techniques—and immersing the reader in the details of using the techniques in documented examples, this special issue encompasses both breadth and depth of qualitative approaches to policy research. Education scholars, in addition to researchers in broader social science fields, will find this issue thought-provoking and functionally useful as a complement to existing approaches to policy research. The methods discussed in this special issue will serve as expedient tools while promoting discussion about how to do policy inquiry in ways that eschew the dominant neoliberal discourse.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.