What’s the Problem Represented to Be? Problem Definition Critique as a Tool for Evaluative Thinking

Abstract

Problem definition in program planning and evaluation is rarely problematized. In this article, I discuss why the lack of problem problematization is itself problematic—in other words, why treating problems as self-evident can pose a risk for evaluation practice. Then, to help avoid such risks, I suggest Carol Bacchi’s “What’s the Problem Represented to Be?” approach as a useful tool to focus evaluative thinking on the frequently tacit step of problem definition. Bacchi’s approach, informed by feminist and poststructuralist epistemologies, is designed to facilitate critical interrogation of policies and programs, focusing on the social and value-laden ways in which problems are initially defined. I propose that evaluators can fruitfully use Bacchi’s tool to promote evaluative thinking about the assumptions that inhere problem representation and thus can help promote better evaluation.

Keywords

problem definition policy analysis feminist evaluation evaluative thinking

Roughly 120 years ago, Du Bois (1898) published an article in The Annals of the American Academy of Political and Social Science entitled, “The Study of the Negro Problems.” That article presciently rings true today, in light of the ongoing struggles for racial justice in the United States and elsewhere. Du Bois’s article also contains an important lesson for program evaluators and others involved in social inquiry. Describing the discourse around “the Negro problem” at the turn of the 20th century, he wrote, “If a Negro discusses the question, he is apt to discuss simply the problem of race prejudice; if a Southern white man writes on the subject he is apt to discuss problems of ignorance, crime and social degradation; and yet each calls the problem he discusses the Negro problem” (Du Bois, 1898, p. 9). Way ahead of his time, Du Bois foreshadowed the notion encapsulated in the title of Robert Chambers’s (1997) book, Whose Reality Counts, or what in Science and Technology Studies is called “ontological politics” (Mol, 1999).

A more recent manifestation of conflicts in the problem definition process from the field of evaluation is presented by Kouévi, Van Mierlo, Leeuwis, and Vodouhê (2013) in their discussion of a contextualized responsive evaluation framework for fishery management in Benin:

The fishery stakeholders share general problem definitions relating to the impairment of fishing communities’ livelihoods and fish stock and habitat depletion. Income-source diversification is also viewed by intervention stakeholders as an alternative for livelihood improvement. However, the stakeholders present different perceptions about the causes of the fishery problems. Whereas fishing communities attribute the causes of the problems to the absence of socioeconomic infrastructures, to the hydroelectric dam constructed upstream, and to the presumed indifference of interventionists, policymakers, and powerful community members, interventionists see the problems as being caused by the non-respect of sustainable fishery resource management rules by fishing communities themselves and the limited intervention resources available. Thus, interventionists and fishing communities differ on what solutions to prioritize. (p. 20, emphasis added)

In short, both Du Bois and these contemporary scholars imply that the ways in which people perceive and interact with the world (both social and material)—and more specifically with the problem-solution nexus—are not inert, neutral, or benign, but instead have stark political consequences that shape people’s lives. Problems do not simply present themselves to us, fully formed, even if rhetorically it often seems as though they do. To the contrary, as Rossi, Lipsey, and Freeman (2004) put it, “Social problems are not themselves objective phenomena. Rather, they are social constructions involving assertions that certain conditions constitute problems that require public attention and ameliorative programs” (p. 107). Yet problem definition in program planning and evaluation is rarely problematized. More broadly, the issue of problem definition or problem forming is “one of the widely neglected aspects of scientific method” (Mitroff & Bonoma, 1978, p. 248). It introduces what Mitroff and colleagues call Type III error: the probability of solving the “wrong” problem when one should have solved the “right” problem (Mitroff & Bonoma, 1978; Mitroff & Featheringham, 1974). The problem here is one of “defining what the problem ‘is’ or what will be considered as such” (Mitroff & Bonoma, 1978, p. 248).

In this article, I discuss why this lack of problem problematization is itself problematic—in other words, why treating problems as self-evident introduces risks into evaluation processes. Then, to help avoid such risks, I suggest Carol Bacchi’s “What’s the Problem Represented to Be?” (WPR) approach as a useful tool to focus evaluative thinking on the frequently tacit step of problem definition. To begin, I first review how problem definition is treated in the program evaluation literature.

Problem Definition in the Evaluation Literature

At first glance, it may not seem entirely evident that problem definition is an integral part of program evaluation. No definition of “program evaluation” includes or even implies an element of problem definition as being within the bounds of the act of evaluating. Yet a search of evaluation textbooks and journals reveals that, while it is by no means a ubiquitous theme, there are a handful of salient discussions of this issue in the evaluation literature, which I present here. For example, as stated by Shadish, Cook, and Leviton (1991), “Evaluation is part of a problem-solving sequence that includes problem definition, solution generation, solution implementation, solution evaluation and solution dissemination” (p. 264). Evaluators are clearly central actors in the solution evaluation step, yet they can also play an important role in the other steps, beginning with problem definition.

There are many ways in which evaluators might become aware of or intervene in the problem definition step. Often, any critical intervention in this step might seem off-limits or infeasible. For instance, requests for proposals from funding agencies—be they for new interventions to address social problems or for evaluations of those interventions—tend to come with the problem already represented and defined, often giving evaluators little leeway to rethink the underlying problem at hand. In similar fashion, a philanthropic organization or community-based agency founded on addressing one problem, one way, will not be apt to question their fundamental approaches to defining and addressing their focal problems (though with an increased emphasis on adaptive management, learning organizations, and nimble approaches, such paradigmatic questioning may be more likely now than in previous periods).

Other times, evaluators are invited to address the problem definition step head on, like when they are involved in creating logic models for programs. According to Newcomer, Hatry, and Wholey (2015), the second step in building a logic model is “clearly defining the problem the program will solve and its context” (pp. 72–73). As such, the evaluator is directly implicated in problem definition. Related to this discussion of logic models, problem definition is also an important aspect of theory-driven evaluation (Chen, 1990). A program theory is a combination of descriptive, explanatory, and prescriptive practice-oriented theories, whereby “the descriptive component of a program theory defines the presenting problem for which a program is given” (Sidani & Sechrest, 1999, p. 228). Problem definition is one of the six elements of program theory identified by Lipsey (1993): “Problem definition…specifies what condition is treatable, for which populations, and under what circumstances, that is, a statement of boundaries that distinguish relevant from irrelevant situations” for the particular program in question (p. 11). In the well-known context, input, process, and product model (Stufflebeam, 1966; Stufflebeam & Shinkfield, 1985), “input” incorporates variables that reflect the problem definition, though I posit that many aspects of problem definition are more appropriately considered part of “context” (e.g., Is the problem defined or framed in a context characterized by institutionalized racism, White supremacy, and patriarchy?)

In other cases, evaluators are one step removed from the problem definition process; they are hired to evaluate the extent to which a program attained its goals, with the tacit assumption that those goals are appropriate way stations on the path toward addressing whatever the underlying problem may be. The logical disparity between simply assessing whether a program met its goals and the fundamental evaluative question of whether those were the “right” goals, aimed at addressing the “right problem,” is one reason why Scriven (2007) lambastes the “cartoonlike effort of the goal-achievement model” (p. 8) in evaluation:

There are half a dozen flaws in the “goal-achievement model” of evaluation. Getting to the goals may only show they were set too low, not that the program was worthwhile. Not getting to them may only show they were set too high, not that the program wasn’t marvelous. The goals may be immoral, or they may be irrelevant to the needs of those served. Getting to goals that are perfectly matched to needs may still represent a trivial achievement; or it may be important, but overshadowed by side-effects, whose existence is not mentioned in this model for doing evaluation. Doing wonderful things is boring from the practical point of view if it costs far too much; or if the costs were moderate, if it could have been done another way for less. Doing even a little good may be worthwhile if it cost very little, And so on; and on. (p. 7; see also Scriven, 1993)

Taken further, in values-driven and social justice–oriented evaluation, evaluators can outweigh the evaluation client in the judgment of the “goodness” of the program’s results (Mertens & Wilson, 2012), predicated on the notion that the evaluator might have a more informed, more nuanced, or even more principled reading of the social problem being addressed (though of course the negotiation of criteria for making such claims requires a great deal of critical reflection, reflexivity, and humility on the part of the evaluator). For example, a governmental program designed to improve the economic position of single mothers may actually operationalize neoliberal, racist, patriarchal paradigmatic assumptions that do symbolic and physical harm to the participants, even if their incomes improve; the social justice evaluator, “reading” the problem and the solution differently than the governmental program administrators, is apt to problematize the assumptions at the root of the intervention.

This is why problem definition is a crucial consideration in culturally responsive and decolonizing approaches to evaluation. Kirkhart (2010), in her discussion of multicultural validity, reminds us to “Pay particular attention to how social problems are defined and by whom. Problem definition also relates to how power is distributed” (p. 405). Similarly, SenGupta, Hopson, and Thompson-Robinson (2004) wrote:

One key site for the cultural expression of values is in how social problems and the programs intended to address them are conceptualized. It is also in this context that one needs to be especially aware of the dangers of imposition, misinterpretation, and misrepresentation of values, of viewing a particular context with a different cultural lens in program design. (p. 8)

As they point out, “Problem definition, a core activity that drives ameliorative program development, is often a dominant culture’s interpretation of reality that perpetuates the myth of the deficit model,” including program theories framed by such terms as underprivileged, at-risk, chronically unemployed, and chronically homeless (SenGupta, Hopson, & Thompson-Robinson, 2004, p. 8). One specific example of a culturally responsive evaluation approach that engages stakeholders in problem definition in a culturally appropriate way is He Taniko, a feminist Maori evaluation approach, based on the world of weaving, using deductive public participation to weave together multiple local perspectives during the problem definition step (Cavino, 2013).

As the literature reviewed above suggests, problem definition “is not as straightforward as it may seem. Indeed, the question of what defines a social problem has occupied spiritual leaders, philosophers, and social scientists for centuries” (Rossi, Lipsey, & Freeman, 2004, p. 106). As Rossi and colleagues (2004) state,

the key point is that social problems are not themselves objective phenomena. Rather, they are social constructions involving assertions that certain conditions constitute problems that require public attention and ameliorative programs. In this sense, community members, together with the stakeholders involved in a particular issue, literally create the social reality that constitutes a recognized social problem. (p. 107)

What sounds like radical constructivism is here presented quite matter-of-factly as a key issue at the heart of all evaluation practice. This point is echoed by DePoy and Gilson (2008) in their book, Evaluation Practice: How to Do Good Evaluation Research in Work Settings, where they unequivocally state, “All problems are statements of value. What is a problem to some is not a problem to others, or how the same problem is conceptualized may differ among groups and individuals” (p. 16).

Elsewhere in the evaluation literature, Shadish et al. (1991) lament the often inadequate theory of social problems which introduces risks to good evaluation practice: “Solving a social problem is aided by a sound understanding of how social problems are defined” (p. 104). Yet a good theoretical and experiential understanding of a problem can also prove inadequate due to social interests and power differentials at play: “Social problem definition is more complicated. Social interests often define needs for reform efforts, furthering some stakeholders’ interests over those of others” (Shadish, Cook, & Leviton, 1991, p. 157); the interests of the economically and politically powerful will be dominant. According to Robinson (2016), the debate over problem definition has,

tremendous importance when the operation of power in a political system is considered. Ignoring or deemphasizing the substantive nature of the expanding, shrinking, or competing dimensions of problem definition risks concealing how these processes serve as a significant instrument of power. (p. 505)

The power dynamics concerning the representation of the problem at hand in turn shapes the evaluation by influencing the selection of evaluative criteria. How are we to analyze the inclusion, exclusion, and manipulation of criteria in defining a problem and evaluating efforts to address that problem?

One response to that query is provided by Schwandt (2018). Drawing from the domain of critical systems heuristics, he offers an analysis of boundary judgments in evaluation. Boundary judgments are “about what aspects of a situation are and ought to be part of the picture we create of what is being studied and evaluated as well as what other aspects are and ought to be left out” (p. 131). Schwandt (2018) continues,

Boundary judgment making is a participatory process of relating and questioning the boundary judgments (multiple perspectives) that different parties to the evaluation make. Making boundary judgments is thus not an expert-driven, objective assessment of supposedly “given” conditions (context) comprising the situation or circumstances wherein a particular evaluation is being done. Boundaries are neither set nor directly and permanently linked to some identifiable features of a situation. (p. 131)

In other words, we must examine “boundaries themselves as entry points for inquiry into the relations between science [or, evaluation] and power, to ask how they come about, and what functions they serve in channeling both knowledge and politics” (Jasanoff, 2003, p. 394). Shining an analytical light on the social process of boundary making is thus one way to try to understand problem definition and its relation to the establishment of evaluative criteria.

Boundary making is an inherently social process. This highlights the important role of stakeholder involvement in problem definition—“The quality of an evaluation largely depends on the quality of the underlying problem definition and the quality of the problem definition often improves as stakeholder involvement increases” (Moret-Hartman, Reuzel, Grin, Kramers, & Van der Wilt, 2011, p. 37). Moret-Hartman and colleagues developed and implemented an interactive “fourth-generation” approach to problem structuring through which interviewees were asked for their own views and then asked to comment on the responses of others and were then themselves reinterviewed, all to achieve a more interactive and iterative version of problem structuring. Many other discussions of problem definition in evaluation relate to collaborative, participatory, empowerment, democratic, and responsive approaches.

Two additional evaluation texts that provide helpful insights and guidance on how to address the problem definition step in program evaluation are Newcomer et al. (2015) and Rossi et al. (2004). Both of these texts call for the type of problem definition analysis that I discuss in this article, anticipating the potential utility of a tool like Bacchi’s “WPR” approach. As summarized by Rossi and colleagues (2004), “Defining a social problem and specifying the goals of intervention are thus ultimately political processes that do not follow automatically from the inherent characteristics of the situation” (p. 107). What follows from this realization is the enjoinder for evaluators to conduct or to facilitate problem definition critique: “An important role evaluators may play at this stage is to provide policymakers and program managers with a critique of the problem definition inherent in their policies and programs and propose alternative definitions that may be more serviceable” (p. 108).

Newcomer and her colleagues (2015) offer a set of “Tips on Defining the Problem Addressed by the Program.” Their tips include the following points, quoted here in their entirety:

Look for what drives the need for the program. Some evaluators put client and customer needs as the first point in the model.

Define all the major factors that “cause” the problem.

Define the factors that the program addresses. Factors that “cause” the problem but that aren’t addressed by the program are part of the context within which the program operates.

Determine whether the program can be modified to address or take advantage of the contextual factors identified.

Identify possible performance partnerships with other programs or organizations whose results affect those of the program.

If necessary, reflect legislative language, perhaps by adding an additional layer of outcomes (Newcomer, Hatry, & Wholey, 2015, p. 73).

These helpful tips go at least part of the way toward the necessary unpacking of the assumptions at the heart of the problem definitnon step, yet additional approaches from the field of policy analysis, such as Bacchi’s approach described below, go even further. There is a need to go further, to get to the root of problem defininton and its impacts on program evaluation practice.

Interestingly, one such policy analysis perspective was reviewed some years ago in the American Journal of Evaluation. The book in question is Problem Definition in Policy Analysis, by David Dery, reviewed by Jung (1986). As Jung (1986) describes, Dery’s view is well-aligned with many of the viewpoints presented above: “His starting point is that policy problems are not objective realities. Rather, they are, in Dery’s words, ‘the product of imposing certain frames of reference on reality’” (p. 51). This is reminiscent of Heisenberg’s (1958) claim in his treatise on physics and philosophy, “What we observe is not nature in itself, but nature exposed to our method of questioning” (p. 57). Dery makes the “rather pessimistic assessment that evaluations of particular social programs, especially those with entrenched interests and long-standing bureaucracies, rarely ‘offer prospects for both relevance and fresh ideas’” (Jung, 1986, p. 52).

Seen in this light, the problem of problem definition is risky not only because it might lead us toward addressing the wrong problem but also because it limits the generative and creative potential of experimenting with new potential solutions, a point not lost on Campbell (1991) in his exhortations for an “experimenting society.” And though they don’t cite Campbell directly, Regeer, Hoes, van Amstel-van Saane, Caron-Flinterman, and Bunders (2009) also call for more experimentation and double-loop learning (“Mode 2 strategies”) as an antidote to flawed problem definition and solution design processes: “Thus, a linear process of problem formulation, project design, and implementation is in Mode 2 replaced by an experimental approach with a number of alternative frameworks that induce variation and offer diverse experiences to learn from and evaluate” (Regeer, Hoes, van Amstel-van Saane, Caron-Flinterman, & Bunders, 2009, p. 518). In their discussion of Mode 2 strategies for sustainable development and other complex issues, “it is suggested that the actual problem articulation becomes an important objective of the program itself, rather than settling down with an incomplete or inadequate set of goals and measuring the outcomes of the program against them” (Regeer et al., 2009, p. 521). This point is reiterated by van Mierlo, Arkesteijn, and Leeuwis (2010), who critique conventional “modernist” ways of addressing complex problems characterized by uncertainty, ambiguity, risks, and unintended consequences: “Thus, our problem-solving procedures are in many ways part of the problem” (p. 144).

Returning to Jung’s review of Dery, we find a call for analysts (i.e., evaluators) to get off the sidelines of the problem definition game: “Dery emphasizes that analysts should become more actively involved in the value-laden business of searching for and creating solutions as they critically assess existing approaches” (Jung, 1986, p. 52). I propose that Carol Bacchi’s WPR approach is one good way to do so. Below, I present her approach, including some literature-based case examples of applications of the approach from beyond the field of evaluation and then conclude by suggesting how this approach can be a fruitful tool for fostering and canalizing evaluative thinking.

The “WPR” Approach

Carol Bacchi, Professor Emerita of Politics at the University of Adelaide in Australia, is a feminist and poststructuralist scholar who researches and writes in the fields of feminist political theory, policy theory, mobility studies, and citizenship. In recent years, she has written extensively about an approach to critically assessing problem representation and misrepresentation in policy analysis (Bacchi, 1999, 2009, 2012a, 2012b, 2015, 2016; Bacchi & Goodwin, 2016). To introduce the approach, I first share an extended quotation from one of Bacchi’s (1991) earliest texts on the subject:

Have you ever read a newspaper article about a controversial topic and thought you would have approached the issue from a completely different angle? Have you ever compared the two perspectives, yours and that of the columnist or reported speaker, and noted that the contrast in views had all sorts of consequences, including how to deal with the issue? If so, you have already been applying the approach I call…“what’s the problem represented to be?” At its most basic, the insight is commonsensical—how we perceive or think about something will affect what we think ought to be done about it. In the words of the psychologists Don Bannister and Fay Fransella (1997, p. 57), “[T]he way we look at things determines what we do about measuring or changing those things; be it the problem child at school, racial prejudice, disturbed behavior in the individual…” The flip-side of this, and the guiding premise of a What’s the Problem? approach, is that every policy proposal contains within it an explicit or implicit diagnosis of the “problem,” which I call its problem representation. A necessary part of policy analysis hence includes identification and assessment of problem representations, the ways in which “problems” get represented in policy proposals. (p. 1)

While Bacchi is focused primarily on policies, I suggest that her approach is equally appropriate and apt for use on programs and other evaluands as well as policies, since any ameliorative effort (whether program, policy, or even product) necessarily implies a response to some problem situation, which in turn can logically present a candidate for problem definition critique.

Another clear and concise introduction to Bacchi’s (2012a) approach, written more recently, is presented here:

The “WPR” approach is a resource, or tool, intended to facilitate critical interrogation of public policies. It starts from the premise that what one proposes to do about something reveals what one thinks is problematic (needs to change). Following this thinking, policies and policy proposals contain implicit representations of what is considered to be the “problem” (“problem representations”). For example, if forms of training are recommended to improve women’s status and promotion opportunities, the implication is that their lack of training is the “problem,” responsible for “holding them back.” The task in a “WPR” analysis is to read policies with an eye to discerning how the “problem” is represented within them and to subject this problem representation to critical scrutiny. (p. 21)

The approach has been refined into a series of six guiding questions, which can and ought to be applied to any ameliorative effort aimed at effecting social change:

What’s the “problem” (for example, of “problem gamblers,” “drug use/abuse,” “gender inequality,” “domestic violence,” “global warming,” “sexual harassment,” etc.) represented to be in a specific policy or policy proposal?

What presuppositions or assumptions underpin this representation of the “problem?”

How has this representation of the “problem” come about?

What is left unproblematic in this problem representation? Where are the silences? Can the “problem” be thought about differently?

What effects are produced by this representation of the “problem?”

How/where has this representation of the “problem” been produced, disseminated, and defended? How has it been (or could it be) questioned, disrupted, and replaced? (Bacchi, 2012a, p. 21)

Note that these steps should be applied in an iterative way, so that any alternative problem representations identified in Step 4 may also be subjected, in turn, to all six steps.

Viewed through the WPR lens, the public is not governed by policies and programs but by problematizations, by how problems are constituted. There is a risk here in reading this claim as simple perspectival constructivist epistemology, grounded in differing perceptions of life. That is not Bacchi’s intent. She is focused on addressing the material, tangible, and concrete material effects of problem representation on people’s lives. On this point, Bacchi (2012a) writes,

To be clear, this claim does not ignore the host of troubling conditions in people’s (and peoples’) lives; nor does it suggest that we are simply talking about competing interpretations of those conditions. To the contrary the proposition is that lives are lived in specific ways due to the shaping impact of proposals that create particular understandings of “problems.” (p. 22)

Such an analysis thus counters the relativist assumption that any one “truth” is as good as any other.

For example, if one is to design or evaluate a program or policy aimed at addressing racism, one must first examine how the problem of racism is posed. A White supremacist may perceive that the truth about the problem of racism is that political correctness, affirmative action, and liberal immigration policies are oppressing White males by taking away their economic opportunities and putting their cultural heritage at risk, whereas a person of color may perceive the truth of the problem of racism as tied to a history of slavery, Jim Crow, lynching, redlining, police brutality, unbalanced sentencing patterns in the criminal justice system, implicit bias in teacher interaction with students and in hiring practices, microaggression in everyday encounters, and so on. Which truth is more true in how the problem of racism is represented?

Avoiding epistemic relativism, Bacchi’s analysis helps us compare these competing problem representations—leveraging theoretically informed assumption critique and the preponderance of amassed empirical evidence—to elucidate that the White supremacist and the person of color actually live racism, materially speaking, in very different ways. The White supremacist perpetrates racism, and the person of color suffers from it. This does not deny that the White supremacist may be unemployed or economically distressed and that elements of his cultural heritage (e.g., segregation) may actually be at risk. But neither of those conditions are the result of “racism.” Rather, they are likely (or at least plausibly) the result, in economic terms, of shifting modes of industrial production in the United States, and, in sociological and jurisprudential terms, of progressively evolving social norms and legal codes (i.e., unlike a few decades ago, the idea that segregation of racial groups in the United States is a good thing is no longer widely held and is now unconstitutional). Relativistically reversing narratives about what is true in the representation of a problem (e.g., choosing to believe that one lost one’s job because of immigrants from Latin America instead of because of increased mechanization in industry) does not actually make the untrue to be true.

To understand the WPR approach, it is helpful to examine its philosophical underpinnings and how they are operationalized in praxis, looking specifically at WPR’s feminist and poststructuralist epistemological roots. WPR is feminist much in the same way Smith’s (2005, 2006) “institutional ethnography” is feminist. Both are ways to do inquiry for which feminism is foundational but not always the focal object of inquiry. Both focus on power, knowledge, categorization, and problematization of seemingly everyday relations. Both began by focusing on problems and issues facing women. In the case of WPR, some of Bacchi’s foundational scholarship through which she developed and refined the approach was focused on the theme of “women, policy, and politics” (Bacchi, 1999). For example, Bacchi (2012) notes that there are many “plans of action that offer women training in order to increase their likelihood of acquiring positions of status or higher paying jobs. Such proposals make women’s lack of training the ‘problem’” (p. 5) rather than discriminatory compensation practices related to the gender pay gap (Blau & Kahn, 2000). The WPR approach can “shed new light on the assumptions embedded in gender policies and in social policies defined more broadly” (Goodwin, 2009, p. 30), such as how dominant or marginal representations of gender and gender inequality appear in key policy documents and programs.

Yet beyond applicability and pertinence for “women’s problems,” when we “understand feminist epistemology as practice rather than content” (Longino, 1994, p. 483), WPR and other feminist approaches are ubiquitously appropriate and needed because of their ability to focus on privilege, positionality, experience, rationality, objectivity, knowledge hierarchies, complexity of relationships, contextualization, antifoundationalism, and other theoretically and politically essential concepts. In other words, as explained by Anderson (2015), “The central concept of feminist epistemology is that of a situated knower, and hence of situated knowledge: knowledge that reflects the particular perspectives of the subject” (para. 2). Feminist epistemology can thus help in “defining the proper roles of social and political values in inquiry, evaluating ideals of objectivity and rationality, and reforming structures of epistemic authority” (Anderson, 2015, para. 2)—all tasks which apply not just to critiquing gender stereotypes and gendered ways of knowing but also to avoiding the production of “knowledge (science and technology) that is not useful for people in subordinate positions, or that reinforces gender and other social hierarchies” (Anderson, 2015, para. 1).

Many of these same concepts are also foregrounded by WPR’s poststructural approach, manifested especially through Bacchi’s application of Foucauldian discourse analysis—specifically adapting his approach to archaeology (Foucault, 1972) and genealogy (Foucault, 1977). As stated by Bacchi (1999), “The focus on interpretations or representations means a focus on discourse, defined here as the language, concepts and categories employed to frame an issue” (p. 2). Along these lines, Foucault (1980) was interested in questions about the rules of the production of true and false, about “what governs statements, and the way in which they govern each other so as to constitute a set of propositions which are scientifically acceptable, and hence capable of being verified or falsified by scientific procedures” (p. 112). The discursive truth-making through problem representation that occurs in seemingly routine program planning and evaluation processes must be problematized.

For example, when the U.S. Secretary for Housing and Urban Development (HUD) states that “poverty to a large extent is also a state of mind” (DelReal, 2017) he is engaged—from his privileged and powerful position of authority—in making the problem of poverty in the United States be one of poor people’s attitudes and perceptions rather than one of systematic structural inequality. He is pushing one truth over other truths, but they are not equal; the choice between these alleged truths is not benign. This particular example evokes Du Bois’s powerful statement from 120 years ago, quoted in this article’s introduction. An evaluator of a program intended to alleviate poverty in a low-income housing development must be aware and critical of the assumptions about the very nature of poverty manifest in that programmatic context before she sets out to evaluate the processes and outcomes of any such program. Applying Bacchi’s six questions to the situation described in this real-life vignette, we find the following responses:

The problem is “poverty,” which could be defined as an individual or collective economic status characterized by a lack of income required to secure basic needs such as housing, food, transportation, education, and so on. Yet more specifically, for the Secretary of HUD, the problem is a cognitive or affective one, about having the right “mind-set” and not having a “defeatist attitude.”

In this representation of the problem, there are assumptions that an important dimension of poverty is individual, mental, and affective, depending on the internal thinking and attitude of the poor person.

In this influential representation of the problem, the problem is being defined by a political appointee who has had a successful career as a neurosurgeon. The statements framing the problem this way were part of an interview the Secretary of HUD gave to an online radio program hosted by a friend of his (DelReal, 2017). The Secretary cites his personal experience growing up in poverty as one justification for his problem framing. Beyond that, this way of framing poverty is a standard, dominant, neoliberal way to frame poverty (Iyengar, 1990; Lawson, 2007).

What this problem representation leaves unproblematic is that the opposite might be true—that poverty is not an individual, mental shortcoming but rather might be a societal, systematic, institutionalized effect of neoliberal and capitalist governance structures. As such, “solutions” based on the Secretary’s framing of the problem (e.g., cutting programs that provide services to people and families) could actually exacerbate the problem. Here, it is important to not fall into a trap of establishing simple false dichotomies about how a problem ought to be represented; there are potentially countless plausible framings of a problem, and the task for the evaluator using the WPR approach is to systematically and transparently identify and weigh the probative value of the most plausible and likely problem framings among the myriad competing problem definitions.

The effects of this potential misrepresentation are twofold: (a) It could cause emotional stress to people living in poverty, since they could either feel victimized or alternatively could internalize the blame and question their self-worth, and (b) it could change the material conditions of people living in poverty, forcing them further into poverty.

As mentioned above in response to Question 3, this problem representation is a common political rhetorical trope of neoliberalism, disseminated and defended by some conservative politicians, pundits, and media outlets. One would hope that, if indeed this is a misrepresentation of the problem (i.e., poor people’s wrong mind-sets and mental defeatism, if they in fact exist, are actually not a primary reason for their poverty), that empirical evidence, theoretical critique, and responsive program planning and evaluation could help promote more accurate and valid representations of the causes and effects of poverty.

As an evaluator evaluating a poverty reduction program, applying Bacchi’s six steps as demonstrated above can improve the evaluation process in a number of ways. If the evaluation has a goal-attainment focus, and the goals of the program are based on a fallible definition of poverty that risks causing material or emotional harm to the program participants, the evaluator can use the results of Bacchi’s analysis to avoid erroneously evaluating a program as “good” which is actually causing harm to participants. If the evaluation is theory based, using a logic model or theory of change diagram as a guiding concept, the evaluator can use Bacchi’s approach as a way to analyze that theory from multiple perspectives and to see the extent to which the implicit problem representation is consistently reflected across all aspects of the program theory. Finally, regardless of evaluation approach or model, Bacchi’s method can help the evaluator write key evaluation questions which focus the evaluative inquiry on the heart of the matter—ensuring that the evaluation really addresses those learning and accountability questions that best reflect and respond to the underlying issues at the problem-solution nexus.

Conclusion: Problem Representation Critique as a Tool for Evaluative Thinking

Given that problem definition is an important yet not widely discussed aspect of program evaluation, and given the utility of Bacchi’s WPR approach as a way of critiquing the assumptions that undergird problem definition, I posit that evaluators can fruitfully use the approach as a tool to promote evaluative thinking. More specifically, WPR can help evaluators apply a critical lens in their work, one that is both culturally responsive and more generally sensitive to the power/knowledge nexus in problem definition and solution evaluation. The increased attention paid to the construct of evaluative thinking in recent years has highlighted the central and ubiquitous role of assumptions in program evaluation.

For instance, Buckley, Archibald, Hargraves, and Trochim (2015) position identifying (and rethinking) assumptions as a key element of their approach to evaluative thinking. For them, evaluative thinking is:

critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action. (p. 378; emphasis added)

Elsewhere, Nkwake and colleagues (Nkwake, 2013, 2015; Nkwake & Morrow, 2016a, 2016b) have provided a thorough analysis of the importance of assumptions in evaluative thinking and evaluation practice. Summed up well in a quotation provided by Nkwake:

Every human society rests on assumptions that, most of the time, are not only unchallenged but not even reflected upon. In other words, in every society there are patterns of thought that most people accept without question as being of the very nature of things. (Trachman & Bluestone, 2005, p. 131, quoted in Nkwake, 2013)

From outside of the field of evaluation, Brookfield (2012) has also provided helpful insights on how to practice critical thinking on the tacit and explicit assumptions that shape our lives. Bacchi’s WPR approach adds a potentially very useful tool to the evaluator’s toolbox to help critically work with assumptions to improve evaluation practice.

Such application can happen in two ways. On one hand, her six questions provide a heuristic device that can readily be applied in any phase of an evaluative process—whether in conducting a needs assessment or evaluability assessment, or in helping to articulate program theory or a program logic model, or in collaboratively setting the boundary conditions around the criteria of success for a given program. This application of the WPR approach is highly practical. And on the other hand, from a more conceptual and theoretical perspective, Bacchi’s approach helps imbue feminist and poststructuralist (i.e., Foucauldian) epistemologies into the program evaluation process. Granted, there are already many other traditions of doing just that in the field of evaluation. Yet Bacchi’s approach provides an innovative and creative way to do so. Taken together, these practical and theoretical ways of using Bacchi’s tool can likely promote evaluative thinking about the assumptions that inhere problem representation and thus can help promote better evaluation. Having proposed Bacchi’s approach, I invite the field to (1) try it out in practice to see how it works and (2) compare it (conceptually and practically) to the other approaches and traditions of problem representation, all in an effort to produce more valid, credible, and useful evaluations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Thomas Archibald

References

Anderson

(2015). Feminist epistemology and philosophy of science. In Zalta

E. N.

(Ed.), The Stanford encyclopedia of philosophy. Retrieved from https://plato.stanford.edu/archives/spr2017/entries/feminism-epistemology/

Bacchi

(1999). Women, policy and politics: The construction of policy problems. London, England: Sage.

Bacchi

(2009). Analysing policy: What’s the problem represented to be? Frenchs Forest, Australia: Pearson Education.

Bacchi

(2012a). Introducing the “What’s the problem represented to be?” approach. In Bletsas

Beasley

(Eds.), Engaging with Carol Bacchi: Strategic interventions & exchanges (pp. 21–24). Adelaide, Australia: University of Adelaide Press.

Bacchi

(2012b). Why study problematizations? Making politics visible. Open Journal of Political Science, 2, 1–8.

Bacchi

(2015). Problematizations in alcohol policy: WHO’s “alcohol problems.” Contemporary Drug Problems, 42, 130–147.

Bacchi

(2016). Problematizations in health policy: questioning how “problems” are constituted in policies. SAGE Open, 6, 1–16.

Bacchi

Goodwin

(2016). Poststructural policy analysis: A guide to practice. New York, NY: Palgrave.

Blau

F. D.

Kahn

L. M.

(2000). Gender differences in pay. The Journal of Economic Perspectives, 14, 75–99.

10.

Brookfield

(2012). Teaching for critical thinking: Tools and techniques to help students question their assumptions. San Francisco, CA: Jossey-Bass.

11.

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36, 375–388.

12.

Campbell

D. T.

(1991). Methods for the experimenting society. Evaluation Practice, 12, 223–260.

13.

Cavino

H. M.

(2013). Across the colonial divide: Conversations about evaluation in Indigenous contexts. American Journal of Evaluation, 34, 339–355.

14.

Chambers

(1997). Whose reality counts. London, England: Intermediate Technology.

15.

Chen

H. T.

(1990). Theory-driven evaluations. Newbury Park, CA: Sage.

16.

DelReal

J. A.

(2017, May 24). Ben Carson calls poverty “a state of mind” during interview. The Washington Post. Retrieved from https://www.washingtonpost.com/news/post-politics/wp/2017/05/24/ben-carson-calls-poverty-a-state-of-mind-during-interview/

17.

DePoy

Gilson

G. F.

(2008). Evaluation practice: How to do good evaluation research in work settings. New York, NY: Taylor and Francis.

18.

Du Bois

W. E. B.

(1898). The study of the Negro problems. The Annals of the American Academy of Political and Social Science, 11, 1–23.

19.

Foucault

(1972). The archaeology of knowledge, and the discourse on language ( Sheridan Smith

A. M.

, Trans.). New York, NY: Pantheon Books.

20.

Foucault

(1977). Discipline and punish: The birth of the prison. New York, NY: Pantheon Books.

21.

Foucault

(1977). Nietzsche, genealogy, history. In Bouchard

D. F.

(Ed.), Language, counter-memory, practice: Selected essays and interviews (pp. 139–164). Ithaca, NY: Cornell University Press. (Original work published 1971)

22.

Foucault

(1980). Power/knowledge. Brighton, England: Harvester.

23.

Goodwin

(2009). Women, policy and politics: Recasting policy studies. In Bletsas

Beasley

(Eds.), Engaging with Carol Bacchi: Strategic interventions & exchanges (pp. 25–36). Adelaide, Australia: University of Adelaide Press.

24.

Heisenberg

(1958). Physics and philosophy: The revolution in modern science. London, England: George Allen & Unwin.

25.

Iyengar

(1990). Framing responsibility for political issues: The case of poverty. Political Behavior, 12, 19–40.

26.

Jasanoff

(2003). Breaking the waves in science studies: Comment on H. M. Collins and Robert Evans, “The Third Wave of Science Studies.” Social Studies of Science, 33, 389–400.

27.

Jung

R. K.

(1986). Review of problem definition in policy analysis, by David Dery. Evaluation Practice, 7, 51–53.

28.

Kirkhart

K. E.

(2010). Eyes on the prize: Multicultural validity and evaluation theory. American Journal of Evaluation, 31, 400–413.

29.

Kouévi

A. T.

Van Mierlo

Leeuwis

Vodouhê

S. D.

(2013). The design of a contextualized responsive evaluation framework for fishery management in Benin. Evaluation and Program Planning, 36, 15–28.

30.

Lawson

(2007). Geographies of care and responsibility. Annals of the Association of American Geographers, 97, 1–11.

31.

Lipsey

M. W.

(1993). Theory as method: Small theories of treatments. New Directions for Program Evaluation, 57, 5–38.

32.

Longino

H. E.

(1994). In search of feminist epistemology. The Monist, 77, 472–485.

33.

Mertens

D. M.

Wilson

A. T.

(2012). Program evaluation theory and practice: A comprehensive guide. New York, NY: Guilford Press.

34.

Mitroff

Bonoma

T. V.

(1978). Psychological assumptions, experimentation, and real world problems: A critique and an alternate approach to evaluation. Evaluation Quarterly, 2, 235–260.

35.

Mitroff

Featheringham

T. R.

(1974). On systemic problem solving and the error of the third kind. Systems Research and Behavioral Science, 19, 383–393.

36.

Mol

(1999). Ontological politics. A word and some questions. The Sociological Review, 47, 74–89.

37.

Moret-Hartman

Reuzel

Grin

Kramers

Van der Wilt

G. J.

(2011). Strengthening evaluation through interactive problem structuring: A case study of hospital care after attempted suicide. Evaluation, 17, 37–52.

38.

Newcomer

K. E.

Hatry

H. P.

Wholey

J. S.

(2015). Handbook of practical program evaluation. Hoboken, NJ: John Wiley.

39.

Nkwake

A. M.

(2013). Working with assumptions in international development program evaluation. New York, NY: Springer.

40.

Nkwake

A. M.

(2015). Credibility, validity, and assumptions in program evaluation methodology. New York, NY: Springer.

41.

Nkwake

A. M.

Morrow

(2016a). Clarifying concepts and categories of assumptions for use in evaluation. Evaluation and Program Planning, 59, 97–101.

42.

Nkwake

A. M.

Morrow

(2016b). Volume overview: Working with assumptions. Existing and emerging approaches for improved program design, monitoring and evaluation. Evaluation and Program Planning, 59, 94–96.

43.

Regeer

B. J.

Hoes

A. C.

van Amstel-van Saane

Caron-Flinterman

F. F.

Bunders

J. F.

(2009). Six guiding principles for evaluating mode-2 strategies for sustainable development. American Journal of Evaluation, 30, 515–537.

44.

Robinson

(2016). A discussion of Frank R. Baumgartner and Bryan D. Jones’ the politics of information: Problem definition and the course of public policy in America. Perspectives on Politics, 14, 505–506.

45.

Rossi

P. H.

Lipsey

M. W.

Freeman

H. E.

(2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.

46.

Schwandt

T. A.

(2018). Evaluative thinking as a collaborative social practice: The case of boundary judgment making. New Directions for Evaluation, 158, 125–137.

47.

Scriven

(1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation, 58, 1–107.

48.

Scriven

(2007). The logic of evaluation. Ontario Society for the Study of Argumentation (OSSA) Conference Archive, 138. Retrieved from http://scholar.uwindsor.ca/ossaarchive/OSSA7/papersandcommentaries/138

49.

SenGupta

Hopson

Thompson-Robinson

(2004). Cultural competence in evaluation: An overview. New Directions for Evaluation, 2004, 5–19.

50.

Shadish

W. R.

Cook

T. D.

Leviton

L. C.

(1991). Foundations of program evaluation: Theories of practice. London, England: Sage.

51.

Sidani

Sechrest

(1999). Putting program theory into operation. American Journal of Evaluation, 20, 227–238.

52.

Smith

D. E.

(2005). Institutional ethnography: A sociology for people. New York, NY: Altamira.

53.

Smith

D. E.

(Ed.). (2006). Institutional ethnography as practice. New York, NY: Rowman & Littlefield.

54.

Stufflebeam

D. L.

(1966). A depth study of the evaluation requirement. Theory into Practice, 5, 121–133.

55.

Stufflebeam

D. L.

Shinkfield

A. J.

(1985). Systematic evaluation: A self-instructional guide to theory and practice. Boston, MA: Kluwer-Nijhoff.

56.

Trachman

Bluestone

(2005). What's love got to do with it? Rethinking common sense assumptions. College Teaching, 53, 131–136.

57.

van Mierlo

Arkesteijn

Leeuwis

(2010). Enhancing the reflexivity of system innovation projects with system analyses. American Journal of Evaluation, 31, 143–161.