What Might International Development Assistance Be Able to Tell Us About Contemporary “Policy Government” in Developed Countries?

Abstract

The article examines international development assistance—aid. Donors assert that experts possess predictive knowledge and project belief in such knowledge into organizational form—the Logical Framework Approach. While such beliefs lack predictive power, as aid operates under multiple sovereignty conditions, no single authority determines truth. Donors ease pressure on experts by accepting variation in intervention logics, yet assert the validity of “single truth” knowledge; knowledge production practices have not basically changed. Belief that what is believed is true, revealed in aid work, illuminates the nature of policy in rich countries and helps explain low confidence in government.

Keywords

comparative governance evaluation change management aid underdevelopment development studies

Introduction

This article is about what we may learn about “policy government” in developed countries from looking at international development assistance. My view is that policy government is one among many causes of the evident lack of popular confidence in politics common in liberal democracies, and I link this to the implausibility of beliefs that it entails. By policy government, I mean the belief that government both should and can be based on deployment of “policy,” where policy is to be believed to articulate some predictively known and evidence-based set of cause–effect relationships. Thus, policy is to be trusted to knowably set out to do something by intentionally “doing X” so as to “attain Y.” My sense of “knowability” here is unabashedly predictive. I argue that there is, in terms of empirics—the internal facts that accompany such beliefs—very limited empirical support for such beliefs, and this clarifies the evident failure of the policy science project (Fforde, 2010; Pressman & Wildavsky, 1973).¹ This project clearly entailed, at a simple level of consistency, reliable predictability, so when that predictability is lacking, cognitive tensions are inevitable. My stress here is on such cognitive tensions, underpinned by my own belief that exploration of such issues offers more explanatory potential than common alternatives, such as reference to structures of power that require governance doctrines to be such as to support, for example, “central power,” in part as I believe that power asymmetries would be needed to support any agreed set of beliefs about the basics of government, including beliefs that were skeptical, asserting a lack of knowledge in the predictive sense used here. Indeed, hierarchical structures that can exert considerable power do exist that are based on precisely such skepticism.²

My arguments are obviously contentious. My entry point to them comes from a discussion of mainstream international development assistance. This is because such beliefs, clearly identifiable as beliefs in policy science, remain central to most development practice. It is striking that aid workers, if professionally trained, confront through their studies accounts of what to do that stress how “what to do” varies over time and place, in stark contrast to what is taught to students of successfully predictive practices such as a medicine and engineering. I argue that this is because the lack of predictive power means that international development agencies are somewhat free to choose their cause–effect beliefs unconstrained by robust empirical regularities, and so students then choose how to align their own (perhaps only outwardly professed) beliefs with those of the agency they choose to work for.

For reasons I discuss, this may be changing. One reason is the way experts, required to articulate the predictive knowability of cause–effect relations embedded in development interventions, become overloaded, in the sense that they are unable to play their allotted functional role comfortably within the structures of mainstream international development. The origins of that lack of comfort are also contested; here, I stress the cognitive tensions created for people when their expertise is meant, like that say of a doctor or engineer, to confer predictive powers, but apparently does not. What makes this of relevance is that international development practice, although applied in developing countries, is, as taught and practiced, founded on normative organizational and governance assumptions and knowledge production practices in developed countries, so that studying the former tells us about the latter. My conclusions about the former then act as an illustrative caricature, so that issues that are nuanced or muted in developed countries may be brought into clearer relief.

The article is reflective and attempts to synthesize experience, literature, and reflection; it is far from conclusive. This is a call in particular for ethnographic research.³

The article has four interrelated strands: First, the idea that, while in the relevant areas there is a clear absence of clear and knowable predictive relations between causes and effects, this is contrary to what is implied by international development practices. Such practices assume a single truth, when evidently there is a multitude of beliefs. I will argue that the key issue here is the heterogeneity of the stuff that aid must deal with: The data suggest that terms such as GDP, rural development, and so on have insufficient ontological stability—as sampling is not from a single population, statistical analyses generate spurious results.

Second, the observation that in international aid work there is no single sovereign power to enforce an authoritative statement of what the (singular) truth of a matter is.

Third, the puzzle as to how this situation may be explained: two competing alternatives here are cognitive (i.e., “this happens to be what people believe, and this can change and probably is changing”) and related to power asymmetries (i.e., “the situation benefits the powerful”).

Fourth, the observation that the key area to examine in assessing this situation is the position of experts, whose expertise is used by international development agencies to document, justify, and provide authoritative accounts of predictive cause–effect relations.

Because most international development assistance is guided by deep beliefs about knowledge and change in contemporary developed countries, what happens when organizations and experts rely on these beliefs to organize international development interventions is an indirect indicator of the nature of developed countries’ important ideas about how change should be done. The puzzles of aid, therefore, tell us quite a lot about developed country governance structures and knowledge production practices.

To manage these four strands, the article is organized as follows. I start with a discussion of international development assistance—aid—and how it is organized. I then look at the belief in predictive knowability that underpins and accompanies aid, which includes a discussion of empirics, and I follow this with a section that examines the issue of multiple sovereignty. Before concluding, sections discuss the “overloaded expert” and then possible patterns in the reinvention of development—current change processes.⁴

International Development Assistance—Aid

Introduction

A service industry, aid has various organizational forms. On the supply side, rich country government agencies answer to their governments and legal systems. These so-called bilaterals, such as the United States’ USAID (United States Agency for International Development), Australia’s AusAID (Australian Agency for International Development)⁵ and the United Kingdom’s DFID (Department for International Development), also channel funds through other bodies. Two important organizational forms here are the nongovernment organizations (NGOs), with large international NGOs (INGOs) playing a major role, and so-called “multilateral” agencies such as the World Bank, UN agencies, and regional organizations such as the Asian Development Bank. Globally, aid from Organisation for Economic Co-Operation and Development (OECD; 2012) governments was US$133.5 billion in 2011, excluding humanitarian aid, debt relief, and some donor costs. Aid amounted to about 0.3% of OECD members’ gross national income (GNI). Rich country governments channel about a quarter of such official aid through NGOs and INGOs and there is a significant and rising volume of private sector support (Kharas & Rogerson, 2012, p. 10). Clearly, aid involves significant resources and, largely derived from tax revenue, is responsible to rich country governments and legal systems.

Aid and development are associated with a large body of research and practical knowledge. The vast amounts spent since World War II (WWII) offer a substantial resource for developing our understanding of governance issues. Development studies are, however, in many academic contexts often rather distant from policy studies. This distance may seem surprising, given apparent shared interests in change management and the use of resources to secure change. One striking result of research has been a valuable negative—globally there are almost no robust relations between policies and outcomes (Fforde, 2005; Levine & Zervos, 1993)—see next section.

Aspects of aid are dealt with in tertiary courses, usually under the heading of development studies. Despite its superficially practical orientation, prospective aid workers study, not a single known “best practice,” but a range of alternatives. Thus, a good textbook (Willis, 2005), is touted as “an ideal introduction to development theories and practices” (Willis, 2005, frontispiece). The stance is that different theories link to different practices: beliefs explain behavior. Rist (2008), another widely used textbook, also stresses the importance of belief, treating development conceptually, as an “element in the religion of modernity” (pp. 21-24). Students are exposed to horror stories that usually show how “the road to hell is paved with good intentions.” Very widely cited is the study of aid in Lesotho by Ferguson (1997) from a widely used collection on “post-development” (Rahnema & Bawtree, 1997).⁶ Ferguson argues that official aid, founded on erroneous beliefs about development in Lesotho, had highly destructive effects. These and other studies suggest to students that aid is not only highly questionable, but that accountability is unreliable, despite aid agencies frequently presenting aid as having altruistic motivations.⁷ Students are aware that aid is managed by agencies of rich country governments, located within standard accountability structures, such as audit-for-value agencies and responsibility to legislatures. Such horror stories then pose tough questions. Examination of such textbooks, however, also shows the tangles that arise when strongly globalizing languages, using terms such as GDP, gender, participation, life expectancy, development theory . . ., sit beside a multiplicity of theories and practices each asserting their correctness.

Aid Logics in Official Development Interventions

Official aid organizes on the belief that donors or their experts⁸ know change processes predictively in advance, as is shown by normatively required use of the Logical Framework Approach (LFA) or some equivalent to manage resource allocation and evaluation of its use (DAC, 2015).

An organizational tool, the LFA rests on a predictive metaphor articulated by an expert or experts linking inputs, through a matrix, to outputs in turn leading to certain development outcomes. An example would be how training courses lead to better educated poor female farmers who then see as a result their livelihoods improve. The LFA entails two beliefs: First, donors and their experts can reasonably expect the inputs to lead to the outputs (“known in advance”); second, they can reasonably expect these outputs to lead to certain developmental outcomes and then successfully evaluate the intervention in these terms (“known after the event”). Known predictability is thus conceptually manifest in the LFA. It exposits the intervention logic and guides the work of officials and others.

Consider the following from AusAID (AusAID, 2005, p. 1)⁹:

The Logical Framework Approach (LFA) is a long established activity design methodology used by a range of major multilateral and bilateral donors, including Australia. It is based on a systematic analysis of the development situation, particularly key development problems, and of the options for addressing those problems. . . . The LFA is an analytical, presentational and management tool which can help planners and managers

analyse the existing situation during activity preparation

establish a logical hierarchy of means by which objectives will be reached.

Donors, according to AusAID, use the LFA for “activity design.” An LFA includes an “activity description” that “clearly specifies what the proposed activity is to do and how” (AusAID, 2005, p. 2). The activity description defines the intervention’s “activity components,” and crucially the “activity rationale” that

outlines the nature of the development situation, particularly the causes and effects of the key development problems which the activity is designed to improve

outlines the cause/effect logic of the proposed activity design, and the expected results of implementing the activity, and

justifies the use of Australian and partner resources in terms of the expected benefits of activity implementation (that is, those expected results of implementation which are benefits relative to the policies and priorities of both Australia and the development partners with whom we work, particularly other partner governments).

In this way the activity design makes explicit the means by which the desired ends of the activity are to be attained. That is, it outlines the means-end relationship between what the activity actually does and the attainment of its objectives, and between the attainment of the lower level objectives of the activity and its higher level (or ultimate) objectives. (For example, as explained later, the relationship between its immediate Purpose, and its ultimate Goal). (pp. 2-3)

The AusAID Guidelines (2005) go on to say that

One standard analytical product of the LFA is the Logical Framework Matrix (LFM).¹⁰ It consists of a matrix with four columns and a number of rows, which summarise selected aspect of an activity design, namely

what the activity will do, and what it will produce (Activity Description)

the activity’s hierarchy of objectives and planned results (also Activity Description)

the key assumptions that are being made (Assumptions), and

how the activity’s achievements will be measured, monitored and evaluated (Indicators and Means of Verification). (p. 3)

The LFA thus treats development interventions as elements of predictively knowable processes, knowable in advance and after the event, that can be both monitored in implementation and evaluated afterwards. It is striking that this is for a given intervention a singular truth: each intervention has conceptually only one rationale. Multiple truths here therefore mean a possibility of different interventions having different “activity rationales,” but in each intervention there is only one normative truth. The LFA is an official requirement of activities that donors will support. It values highly the expertise that provides the knowledge on which the logical framework is constructed.

The LFA conceptual framework appears to sit easily within a results-based management discourse, but faces two major problems in aid work. The first is the absence of a single authority to state, if disputes or disagreements arise, what the truth of the matter is, because there is multiple sovereignty (I discuss this later). The second is that the evidence from application of relevant statistical methodologies to the available data sets is that there are almost no robust relations between our actions and their effects, yet most published research says that there is. Evidence suggests that evaluation produces comparable problems. To quote Tilley (2000),

The mixed findings from . . . evaluation studies are typical. Where several evaluation studies are found the most usual finding is that results vary. (p. 4)

Repeated evaluations of the same intervention, basic to assessing evaluation methods, are hard to find. There is no mention of them, despite considerable discussion of the assessment of evaluations, in Hageboeck (2009). This is telling.

Other bilaterals such as the United Kingdom’s DFID treat predictive knowability as fundamental. For example, a DFID study suggests,

. . . in order to develop the proposed M&E framework, it is important that there is a good understanding of how the various inputs, which could be expenditure or staff time, translate into outputs, outcomes and impact, that is the assumed “results chain.” Such a model or “theory of change” will allow the identification of indicators at different points in the results chain to trace through what activities are being effective in what ways. (Thomson, Barnett, & Yaron, 2009, p. 4)

As we shall see, “theories of change” (ToCs) techniques may mark a pattern in the reinvention of development. Yet they defend expertise by accepting differences between individual projects’ change rationalities but still require a singular rationality for a given project. Thus, AusAID Guidelines (2005) state that documents must contain an activity rationale. This

. . . outlines the cause/effect logic of the proposed activity design, and the expected results of implementing the activity, and justifies the use of Australian and partner resources in terms of the expected benefits of activity implementation . . . the activity design makes explicit the means by which the desired ends of the activity are to be attained. That is, it outlines the means-end relationship between what the activity actually does and the attainment of its objectives . . . (pp. 2-3)

LFAs are not challenged by ideas that there can be multiple truths. To quote a DFID study,

in order to develop the proposed M&E framework, it is important that there is a good understanding of how the various inputs, which could be expenditure or staff time, translate into outputs, outcomes and impact, that is the assumed “results chain.” Such a model or “theory of change” will allow the identification of indicators at different points in the results chain to trace through what activities are being effective in what ways. This has the benefit of allowing DFID Research staff to trace any bottlenecks in achieving results more efficiently. (Thomson et al., 2009, p. 4)

A situation where action assumes predictive knowledge that is evidently absent is full of tensions. It may appear to senior management in the donor capital city that their and their experts’ words have stable referents, so it makes sense to talk about participation, GDP, and industrialization being about the same things across time and space. Such essentialist thinking can help preserve the structural position of experts and the LFA, through such conceptual devices as the idea that although there is some essential nature to GDP, in particular places and times, it may be different—reminiscent of Plato’s fire and shadows on the wall. But what is fundamental is whether this makes sense empirically, for if it does not, then “X” will not lead to “Y,” researchers add it to their list of horror stories and officials’ careers may be at risk.

Belief in Knowability

Managing Intervention Logics—the Unknown, the Unknowable

Just what we mean by knowable varies. Scholars like Nisbet (1969) argue that social science criteria for the acceptability of accounts are historically deep-rooted and exclude a criterion requiring comparison of accounts. By contrast, predictive power was argued to have been added to Aristotelian norms many centuries ago (Crombie, 1953), a criterion within method requiring comparison of accounts.

A rapid literature search reveals a range of ways to understand meanings of knowability and ignorance. Thus, for example, agnotology uses the term ignorance to mean false belief (Proctor & Schiebinger, 2008). By contrast, the identification problem in econometrics refers to the idea that certain unknown aspects of reality arise from the empirical circumstances within which a model has to exist, where the researcher is unsure that “there is no other structure . . . also capable of generating the data” (Hsiao, 1987, p. 95). Thus, the validity of a model may be unknown empirically, for “without a priori restrictions imposed by economic theory it would be almost impossible to estimate economic relationships” (Hsiao, 1987, p. 99). A third option is to treat predictive ignorance as reflecting lack of regularity in what we observe. These three contrast ignorance as false belief, the unknown, and the unknowable.

More generally, perhaps reflecting Nisbet’s argument about theories of social change in the West long being multiple and essentialist, when it comes to action that requires predictive power, one might expect a skeptical attitude. Yet we find a pattern of belief in belief: habits that tend, rather than withholding belief in the predictive power of some offered explanation or theory, to choose to believe in it and so underplay possible negative consequences. It is very common to find explanations that first command beliefs, and then data that confound them (Fforde, 2009, Chapter 7; Gundlach, 1999). And this is what we find in aid practice.

The Rejection of Skepticism—Citations of Levine and Zervos (1993)

A valuable indicator of knowledge production practices is the impact of Levine and Zervos (1993), which reported almost no robust relations between economic performance and policy viewed globally. I investigated the impact of this study through citations data (Fforde, 2005). Most citations did not consider that the result had great significance. Only a minority reflected on what it could imply for the validity of prevalent beliefs in the existence of known and robust relationships between policy and outcomes. This situation suggests that belief has remained central to understanding the deployment of knowledge and the role of expertise. The apparent lack of robustness sits beside a wide range of contradictory published studies that present evidence that X does indeed lead to Y. These tend to contradict each other, which is likely why Levine and Zervos (and Levine and Renelt) carried out their research.

The robustness studies cited above are now some two decades old. They have not really been challenged.¹¹ Even the revisionist study by Hoover and Perez (2004) only showed that some 40% of economic success was, in terms of policy, knowable. Recent citations show pretty much the same pattern as reported in Fforde (2005).¹²

What this is showing, therefore, is that published research from these knowledge production practices that proclaims the existence of regularities, in any given sampled population (whether local or global), is suspect. This is a characteristic of the research practice, not of what it studies. The practice produces understanding, not reliable predictive power. The issue is not whether regularities do or do not really exist, but of their predictive knowability in terms of social epistemology—the social production of knowledge. The empirics suggest that robust predictability is absent, yet, as we have seen, official aid organizes as though it were present, and is supported in doing so by a range of published research from which expertise can be selected.

Levine and Renelt (1992) reviewed the empirical growth literature for the World Bank. The majority of studies examined by them were from the 1980s and show the suggestive pattern: Each individual report was publishable in its own terms but, viewed as a research program, there was disagreement. This suggests that assumptions untestable at the level of the individual paper were awry. One candidate was that sampling was not from a single population; put in more technical terms, the assumption of ontological universality was awry (Kenny & Williams, 2001). If this were true, then the empirical results of each individual paper would be spurious—not wrong, simply empty—because an assumption needed to give meaning to the statistical inference was false. It is plausible that this was obvious to the better econometricians far earlier than the early 1990s, when Levine and his collaborators published, for the data were saying that robust cause–effect relations in cross-country growth analysis were in effect unknowable. To repeat, a good candidate for the critical assumption was that sampling was from a single population—in simple terms, that the stuff that makes up our world is far more diverse than our discourses imply. To cite perceptive remarks from within the evaluation literature that parallels this history, “like was not being compared with like” (Tilley, 2000, p. 4).

From the perspective of practices such as medicine or engineering, this situation is extremely striking. Research that showed, to think up examples, that antibiotics had in fact no robust impact, and that steel reinforcements to concrete did not either, would surely have rapidly led to “something big.” Levine and Zervos did not. One may conclude that the relationship between empirics and belief is, for discussions of policy and economic growth, by the majority still accepted as “vague,” to put it politely (Fforde, 2013). The criteria applied to accounts of social change for them to be considered as potentially valuable, as Nisbet suggests, do not include a criterion for comparing them. By contrast, predictive power is included in acceptability criteria presented by Crombie and requires comparison (Crombie, 1953; Nisbet, 1969). The cross-country growth regressions literature shows a world of multiple truths that obey acceptance rules that lack a criterion that they be compared.

This situation poses interesting problems for experts. A leading member of the profession and contributor to economists’ discussions, Lant Pritchett, coauthored a paper in 2002 that argued that from around the start of the new millennium economists were increasingly unable to agree on broad statements about what works and why at a global level (Lindauer & Pritchett, 2002). Evidence for shifts away from belief in known global results can also be found at high level in the aid industry. By the early noughties, the International Monetary Fund (IMF) was overtly taking a skeptical stance regarding important policy issues. Thus, in a formal doctrinal statement about the positive or negative value to growth of an open capital account of the balance of payments, two senior officials of the IMF published the following authoritative statement in the Financial Times:

Economic theory leaves a number of complex and crucial questions unanswered. For instance, in order to control the risks associated with opening up to capital inflows, it seems necessary for countries to have strong institutions. On the other hand, inflows of capital, especially foreign direct investment, may bring technological know-how and knowledge of best practices in other countries that can improve domestic institutions. So should a country postpone opening its capital markets until it has good institutions? Or should it use financial integration as a tool to improve its institutions? Unfortunately, there are no definitive answers to these issues, which are best approached by each country depending upon its circumstances. (Prasad & Rogoff, 2003, quoted in Fforde, 2009, pp. 119-120, emphasis added)

The tensions implied by this situation suggest that experts in these areas who confront informed audiences would be overloaded: They are structurally required to articulate beliefs in “what will work,” yet the empirics support, to the contrary, skepticism, and the expression of predictive ignorance. Their offered understandings are not advisedly treated as predictive knowledges. This is not because of lack of theoretical effort, but because the truth of the matter—what the data available say—is that there are no robust regularities.

The Issue of Multiple Sovereignty

Here I consider implications of the core aspect of the multiple sovereignty issue, which is the lack of a single authority to state the truth of the matter. As we live in a world formally structured in terms of states that respect each other’s sovereignty (Spruyt, 1994), it is conceptually true that aid, using the resources of one country to implement interventions in another, is subjected to multiple sovereignty. Bilateral aid is usually based on formal Agreements, but there is no international Court or similar body to impose solutions if there is disagreement. Comparison with official agency activities within their own (developed) countries is useful, for then, like within a well-structured corporation, there should be ways of establishing “truth”—whether a task has been done, what “profitable” means, and so forth.

An illustrative example of multiple sovereignty is a Memorandum of Understanding (MoU) signed in April 2013 between Australia and China stating the framework to govern an aid program for the Chinese Ministry of Commerce (Government of Australia and Government of China, 2013). Signed by the two foreign ministers, it states (in the section headed “Settlement of Differences”),

This MoU serves only as a record of the participants’ intentions and does not constitute or create (and is not intended to create) rights or obligations under domestic or international law and will not give rise to any legal process . . . Any dispute, controversy, or claim which arises out of the interpretation or application of this MoU will not be subject to adjudication or arbitration, but will instead be dealt with through amicable consultations and negotiations as the only method of achieving the peaceful settlement of that dispute, controversy or claim. (Clause 12)

This boiler plate clause expresses the two sovereign parties’ agreement to deny participation by any higher authority than the two governments in resolving related disputes between them: multiple sovereignty. The Clause refers to “interpretation” of the MoU, and so to any disagreement about what causes what and what caused what. It is self-evident that the two governments have very different ideas about the basics of governance: one is a federal liberal democracy and the other is a constitutionally one-party state ruled by a Communist Party.¹³

The question of multiple sovereignty has been examined by students of development, most interestingly by considering how the question of what correct development has been answered. An important book in the mid-1990s argued for a radical reinterpretation of the “problem of development” that links directly to the issue of authoritative truth. Cowen and Shenton (1996; see also Fforde, 2009) argued that, for relevant groups, development conceptually both happens and is made to happen, leading to important conceptual tangles. For them, as outcomes are predictable, they argued that one of two things has historically happened to cope with the question of what constitutes correct development and so—crucially for aid work—defines correct development policy and practice. Clearly, if development is done, is made to happen, then this requires that development be believed intentional, and intentionality has then to be sited somewhere. In their first option, the development process itself then incorporates the subjectivity of that intentionality. This they see as a Marxian position where policy simply reflects the logic of capitalism. In their second option, recourse to authority answers questions as to what constitutes correct development and development policy. This second position is persuasive and, in their study, supported by a wide range of historical evidence. It fits with the lack, according to Nisbet, of a criterion within the list required for accounts of social change to be acceptable, of a criterion requiring comparison of alternative accounts (as a predictive criterion does). To quote the study for USAID already cited,

Unlike applied sciences, such as medicine, where theory, policy, and practice tend to move sequentially, trends in evaluation theory, policy, and practice exist in a common space and influence each other, but directionality varies. (Hageboeck, 2009, p. 3)

In aid practice, parallel to its organizational logic, donor authority states truth, and this is articulated by and through experts. However, without regularity in the data to ground authoritative statements, we must expect, and there is much evidence that we find, experts overloaded by the need to negotiate with a range of authorities and variation over time in what a particular authority states the truth of the matter to be. Rather than working in ways comparable to, say, engineers or doctors, aid workers develop distinct skill sets and career paths (Fechter & Hindman, 2011). Their expertise is not predictive, but must appear so.

From the point of view of aid workers, experts are subsidiary to donor intentions; with their views ungrounded in robust empirics, as we have seen above, training in development studies includes exposure to radical critiques, such as “post-development” (Ferguson, 1997) that paint the tensions of donor development doctrine in stark relief. Thus, unlike doctors and engineers, instead of having their authority based in an appreciation of stable and robust empirical regularities, experts are chosen, and their expertise funded, by donors, and donors vary and donors’ beliefs vary over time (Fforde, 2009). Donors’ agencies are responsible to their own sovereign governments, yet operate in a world of multiple sovereignty. They may legally require compliance from contracted consultants and aid workers but cannot exert equivalent authority over other actors on the recipient side. Such structural issues are glued to the question of sovereignty. Note Hinsley (1986),

If we wish to explain why men have thought of power in terms of sovereignty we have but to explain why they have assumed that there was a final and absolute authority in their society—and why they have not always done so . . .

The concept has been formulated when conditions have been emphasizing the interdependence between the political society and the more precise phenomenon of its government. It has been the source of greatest preoccupation and contention when conditions have been producing rapid changes in the scope of government or in the nature of society or in both. It has been resisted or reviled—it could not be overlooked—when conditions, by producing a close integration between society and government or else by producing a gap between society and government, have inclined men to assume that government and community are identical or else to insist that they ought to be. In a word, the origin and history of the concept of sovereignty are closely linked with the nature, the origin and the history of the state. (pp. 1-2)

Neither of the elements of aid work just discussed is entirely unfamiliar to experienced students of governance in developed countries, but there they are not quite so structural. There may well be a writ to run in the provinces of a rich country, and whether it actually does or not is quite another matter, but under normal circumstances there are ways to sort out these conflicts within that political community. This not only does not necessarily threaten the idea of sovereignty as deployed within such political communities, it will usually rely on it.

Aid workers, thus, usually operate in situations of multiple sovereignty, where conceptually there is no sovereignty—there is no single source of official authority over what they do (at a minimum there are at least two) and where “multiple truths” and “diverse beliefs” are an innate part of the social environment within which they work. The embassy where donor officials sit says one thing, the ministry responsible for the project says another, and an experienced aid worker will be facing a familiar situation and should know how to live with it. Familiarity with such environments is part of the skill set of good aid workers (Fechter & Hindman, 2011).

The Overloaded Expert: Pressures for Change?

In this section, I examine possible pressures for change, focusing on the position of experts as sayers of truth. As I have already argued, the situation challenges the role of experts in articulating for donors a truth stating what predictable cause–effect relations will hold. Experts play a key role in international development assistance, as they articulate for donors those truths that give meaning to, and validate, development interventions. Erosion of their authority therefore undermines the foundations of aid interventions.¹⁴ In the evaluation manual of the Swedish bilateral donor (Sida), we find the following:

With the help of control groups who have not been exposed to the intervention it is sometimes possible to get a good idea of how the target group would have fared without the intervention. When the counterfactual cannot be estimated in this way—a common situation in development cooperation—statements about impact rest upon weaker foundations. The intervention is often taken to be the cause of the identified changes if such a conclusion appears to be consistent with expert knowledge and there seems to be no better explanation around. Although less compelling than an explanation based on control group methodology in most cases, an argument of this type can be good enough for the purpose of the evaluation. (Molund & Schill, 2004, p. 32)

Yet, evidence shows that knowledge production practices do not reliably deliver robust predictive guides to interventions that use the LFA. Even the best expertise, although asserting that it knows the truth of the matter, confronts donors with a lack of that robust predictability formally required by the LFA. Add to this the issue of multiple sovereignty, so that there is no way to force agreement between the host government and the donor agency, and experts, as any experienced aid worker knows, face familiar tangles.

The nub of the issue lies in differences between what different actors say, think, and feel about development. These tensions present, though, through devices such as the LFA, as intellectual matters. Development studies courses therefore understandably compromise and teach students about how ideas about development have varied over time and between different actors.

This suggests that, for many honest practitioners,¹⁵ the problem of development has become the tension created by formal knowledges that contain rigid assumptions about how to do development—specifically, that change is both to be done in accordance with some single activity rationale—a knowable cause–effect logic (knowable before and knowable after)—and the fact that such knowable logics cannot reasonably be said to have been found in the terms (variables) of the relevant discourse (Cowen & Shenton, 1996; Fforde, 2009). If they had been found, students would be taught like nurses, doctors, or engineers, and they are not.

For the wider governance literature, this suggests that study of international development work clarifies what happens when agencies advocate change based on evidence-based policy. Evidence-based policy, in my view, stresses the importance of either securing reliably robust (i.e., reliably predictable) research results, or of the use of sovereign power to decide what the truth is to be said to be. If, as is often the case, reliably predictive research results are unavailable, then this suggests the importance of understanding the sovereignty issue, that is, the question of deciding the truth of the matter and what happens if there is no authority to do that.

Assuming predictive knowability increasingly commands less and less support. There is a tension between increasingly powerful social beliefs in the relativity and subjectivity of beliefs, themselves related to the social issues of increasingly complex modern societies, and the pressures to organize and behave in certain ways (such as the LFA) that entail belief in predictive knowledge. This is partly because belief in predictive knowledge implies viewing as wrong any who disagree with the particular beliefs asserted (e.g., by the knowledge production processes involved in the preparation of a log frame). Thus, to quote again the AusAID Guidelines (2005),

In order to help avoid common problems associated with the use of the LFM, AusAID managers should

ensure their colleagues and partners have a common understanding of the key analytical principles and terminology used

emphasise the importance of the LFA process at least as much as the matrix product

ensure it is used as a tool to promote stakeholder participation, dialogue and agreement on activity scope, rather to impose “external” concepts and priorities

avoid using the matrix as a blueprint through which to exert external control over the activity. (p. 4)

The tensions revealed here are striking. If all participants are to have a “common understanding” of the principles used, then the process can only be intended to result in agreement about the activity description and its rationale—a single truth. If that agreement is not forthcoming, then it follows that, as the last dot point suggests, the incentives structure may encourage AusAID managers to conflict with other stakeholders and use the matrix to push. Many development practitioners are well aware that such attempts can be particularly difficult—for example, if participants, probably due to different values and perspectives, dislike the particular activity rationale and do not experience the intervention as leading to the predicted outcomes. Apparent power asymmetries can quickly vanish, as, perhaps, poor farmers decide to fail to turn up to meet key officials, or learn to game interactions so as to turn random sample trials to their advantage (Fforde, 2013, Chapter 22; Fforde & Seidel, 2015).

Aid workers learn fast (one may hope) that project activities usually require significant inputs from a range of stakeholders, delivery of which may in practice require more or less explicit “agreements to disagree” (multiple truths). In some situations, resources that could be used for negotiation (perhaps by an AusAID manager) may be limited because hierarchies above the manager are focused on implementation of a particular activity description and activity rationale. This will perhaps be expressed in a global language (perhaps referring to global best practice—see the quote from the AusAID Office of Development Effectiveness above), which suggests that “what works there, will work here,” so the activity rationale creates antagonisms.

The Reinvention of Development—Trends

Alternatives: The Possibility of Skepticism Within a Hierarchy

Many people believe that the origins of this situation lie in power asymmetries, though donors often do not get what they want, as the horror stories discussed above suggest. Clearly, though, donor officials often present as powerful and insulated from, say, the nitty-gritty of local development tangles. Yet it is possible to find and discuss, even if ethnological studies are very limited, formal organization and hierarchy that entail skepticism.

One way of discussing this through notions of vertical and horizontal consistency: What is required by superiors for activities to be deemed correct (vertical consistency)? What is seen by superiors as acceptable differences between subordinates (horizontal consistency)? If “it does not matter what color the cat is, so long as it catches mice,” then acceptance of different colors is a criterion for horizontal consistency and requiring the cat catch mice is a criterion for vertical consistency (rabbits will not do).

Consider the practice of the Department of Treasury and Finance (DTF) in the state of Victoria, Australia. The general rule applied by DTF is to pay for outputs, and reports about delivery of these, plans having previously been negotiated with suppliers (typically line Departments such as Education), then form the basis for disbursement (Fforde, 2010). In such conditions, two suppliers that have inconsistent activity rationales (to use the AusAID terms) may both get paid so long as they are said to have met agreed output target indicators: DTF is formally uninterested in how they catch mice, simply that they do so. This works because DTF is backed up by the sovereign powers of the Australian state.

Although I do no more than flag it here, histories of Prussian and then German military doctrine often argue that it was based on doctrine that asserted the inherently chaotic nature of war, and suitable organizational responses. These in most accounts focus on a stress on mission statements crafted to encourage local initiatives justified in terms of attaining that mission (Hoffman, 1994; Nelsen, 1987; Vardi, 2008; Widder, 2002; Ziegler & DeGorsky, 2008).¹⁶ The responsibility of superior officers was then not to ensure that their orders had been obeyed but to encourage initiatives to ensure attainment of the overall mission (suitably formulated).

Changes in Official Aid Doctrines?

In this section, I discuss evidence for change in official aid doctrines. I focus on issues related to the authority of doctrine and the validity and role of knowledge.

I have already pointed to signs of change—I mentioned above that a minority of citations of Levine and Zervos (1993) accepted that “there was a big issue.” I also mentioned work by Lindauer and Pritchett and the changing view of the IMF on capital account liberalization as evidence for not widespread adoption of skepticism (for the citations above of donor requirements show the contrary) but at least acceptance of it not being entirely out of the question. Relevant here is what is said at important international development conferences, and the following shows the meanings attributed to these in a recent study commissioned by USAID:

The 2002 International Conference on Financing Development in Monterrey, Mexico initiated a new dialogue between donors and developing countries aimed at creating more effective partnerships for development. The consensus of this meeting recognized the leadership role of developing countries with respect to their development plans, fostered donor support of those plans and simultaneously called for improvements in developing country governance. This consensus was expanded in 2005, through the Paris Agreement, which set forth an agenda for reforming aid delivery and management with the aim of improving its effectiveness. The principles of this agreement focused on host country ownership of the development process; improved harmonization of donor practices and their alignment with country-led strategies; better use of managing for results approaches; and mutual accountability for the use and results of aid. (Hageboeck, 2009, p. 4)

This quotation suggests that the difference between “knowledge before” and “knowledge after” is crucial.¹⁷ Shifts in donor rhetoric have not been reflected in an increase in beneficiary participation in bilateral and unilateral donor evaluations: rather, what has happened is a trend to inclusion of stakeholders in intervention design. But by this is meant actors from other donors as well as from recipient organizations, such as public officials (Hageboeck, 2009, p. 9).

Parallel with this, normative statements about how to do development have tended to retreat, framing positions in ways that while still appearing to leave broad-brush universals intact, step around overtly implying their universal validity. A good example of this is a contribution to the World Bank’s 2011 World Development Report background papers, again by Pritchett (Pritchett et al., 2011), where the analysis tiptoes around the issue of knowability “beforehand”:

The causes of . . . failure are complex and diverse and we do not pretend to explain why this failure occurs; rather, we focus on how. . . . [this happens] . . . This analysis gives rise to a policy research agenda focused on better understanding the conditions under which political space is created for nurturing the endogenous learning and indigenous debate necessary to create context-specific institutions and incremental reform processes . . . capable of effecting systemic change, at scale, while retaining flexibility and adaptability in the face of contextual idiosyncrasies and in response to local accountability norms. (pp. 44-45)

Framing here seems to seek to balance three tendencies: first, assertion of generalities (that the development process has common essential patterns; Pritchett et al., 2011, p. 3 et seq.); second, despite this to avoid general predictive statements; third, to assert that through a conceptualization of diversity that has particular definition(s) of the local, predictive certainties will be found. To put this in another way, it accepts that regularities are unknowable in global terms while asserting that they are to be found at some (as yet unspecified) local (“indigenous”) level.

What is happening here, I believe, is a defense of belief in the knowable existence of regularities prior to their empirical identification. Pritchett et al. (2011, p. 3 et seq.) thus poses and answers the essentialist question “What is development?” Traditional views would assert that such a question, if predictive knowledge is sought, could and should be answered (if it can be) through a series of dual passages through theory and empirics and back again (e.g., Crombie, 1953), but this is not happening here. The direction is from the falsification of old truths rather than from the identification of previously hidden regularities. The fate of Levine and Zervos (1993) thus was that they identified a previously hidden lack of regularities.

There is thus a now familiar tension between the assertion in Pritchett et al. (2011) that the causes of failure are unknown and the assertion that the correct policy research agenda is known. One might argue that if you have no idea what causes failure there is little reason to decide to devote research resources to anything specific at all, yet by contrast the paper asserts, supporting donors and the way they use, that systemic change can be known in advance. The cultural and organizational pressures to preserve belief in the predictive knowability of cause–effect relations are strong (Fforde, in press).

So what options have arisen to address these issues? I argue that we find basically two.

Cognitive Change—The Conservative and the Radical Options

Although the most consistent response to unknowability would be to shift practice in directions that explicitly acknowledge such ignorance, the most practical one appears to be to move forward in ways more consistent with past practices. Consider the following quote from Dunn (2000), a professor of political theory at Cambridge, United Kingdom:

In the case of the massive impetus towards economic liberalisation of the last two decades of the twentieth century . . . it is natural to see this [as] the discovery of ever clearer and more reliable techniques for fostering economic efficiency . . . This is very much the way in which a whole generation of economists actively engaged in public service have come to view it, just as their Keynesian predecessors a generation or so earlier saw the previous move in a roughly opposite direction. Viewed epistemologically, however, the sequence looks strikingly different. . . . (I)t was the increasingly evident falsity of one set of false beliefs, not the steadily growing epistemic authority of their replacements, which did most of the work. . . . The clear result is the negative result. (pp. 184-185)

The two elements that stand out are first, the preservation of the role of the expert and, second, the pattern of seeking to drive change by revelatory truth rather than observation of regularities. As Dunn puts it, the negative—the “evident falsity of one set of false beliefs”—drives change. There is no evidence I know in aid practice that formally accepts the radical option and organizes on an assumption of ignorance; rather, in a conservative manner, the reinvention of development preserves the belief that what a group thinks to be true, actually is. Donors do not euthanize the overloaded expert but protect them by a reduction in the expected scope of their expertise.

A New Pattern of “Multiple Truths”

Statistically, the assumption that sampling is from a single population is required for the making of statistical inferences; if this is awry, then results are spurious. This resonates strongly with development practitioners aware of how local specificities create tensions with the universalizing language of much development expertise. So how can one find a “single population?” An answer would be to “go local.” One particular technique has been increasingly influential: ToCs. A recent study for the U.K. bilateral, DFID, reports the mainstreaming of ToCs into aid work since the mid-noughties (Vogel, 2012, p. 8). She argues,

Theory of change comes from both evaluation and social change traditions, so it is being used both by smaller civil society organisations and by donors; development agencies and organizations are mainly using theory of change for evaluation, but it is increasingly being used for programme design and to guide implementation; [and] perceived benefits include an integrated approach to design, implementation and evaluation and better analysis of the programme context than other approaches.

Three key forces for her explain the popularity of ToCs—the results agenda, the need to cope with complexity, and the emphasis on local country ownership in development work. She also stresses a diversity of approaches to defining the term. However, as I have argued, the LFA stresses creation of some singular and particular intervention logic, a “common understanding of the work” (Vogel, 2012, p. 11). Her Table 1 on p. 13 called “Use of theory of change thinking” shows no sense that alternative ToCs may be valid in a particular context, and active within a project or intervention.¹⁸

Vogel (2012) reports, however, as a view important among practitioners, that ToCs should be carried out with a light touch and nondogmatically to “strengthen the potential of programmes to support the development outcomes they seek” (p. 62). Examination of the list of contributors to her Review and their affiliations (p. 81) shows a predominance of development professionals. The discussion of donor practices above shows how donors deploy expertise. It is clear that ToCs mean the continued use of ideas of change to organize development interventions based on belief in knowability. A ToC is a statement of what donors expect will happen and the basis for how evaluations after the event should be done: it states the ToC to use in a particular context. This particular ToC can be different from others in similar but not identical contexts, and is this limited sense one can talk of a plurality of ToC, but in a given context, the sense is singular. Other ToCs are, therefore, to be seen there as wrong.

I conclude from this that donors and aid workers are here seeking to deal with tensions created by the issues of multiple sovereignty and apparent unpredictability. They are doing so by preserving the role of expertise in establishing and validating the particular intervention logic, but the load on expertise has been reduced in two ways. Allowing for multiple intervention logics creates distance between the particular intervention and global generalities. First, this makes it easier to say that “what works there will not work here,” and this reduces the load on experts. Second, however, this creates data of only local meaning, so it is harder to monitor the design of ToCs. This second point has two possible consequences. First, it offers greater incentives for donors to monitor interventions through enquiries of end users.¹⁹ Second, it offers scope for the more powerful. In a study quoting a Finnish NGO worker,

Our partners do not understand all of these matrices, and actually we don’t either, even if we’re able to use them. But narratives are something that every human being can understand. (Holma & Kontinen, 2011, p. 184)

So ToCs, embedded in logical frameworks, are incomprehensible to laypeople—those who are not experts. Arguably, processes of education will design this into requirements of expertise, preserving the structural position of experts. ToCs cannot remove the inconsistencies of the LFA.

Conclusion

The Aid Industry

The focus of this article has been a discussion of donor interventions in developing countries—aid. Spending around one third of 1% of developed country GDP annually, organized to meet the requirements of government bureaucracies, aid is a case study of modern governance.

Arguments for treating development intervention as strongly influenced by the need to manage ignorance, itself a manifestation of the heterogeneity underneath standard terminologies, receive strong support from a number of directions. First, management of ignorance is better seen as the management of diversity: using the standard social science vocabularies, we are not sampling from a single population; in those terms, there is neither ontological nor epistemological stability. Criticisms of essentialism are thus empirically valuable, rather than theoretically valid. Ethnographic research into aid organization that tells us about how assumptions of knowability are challenged would help tell us why we continue to use languages that suggest a degree of commonality and uniformity that the evidence tells us is absent. I am not convinced that this absence is a consequence of power asymmetries.²⁰ One reason is a sense that power asymmetry would be needed to enforce respect for multiplicity of valid meanings.

Second, I have argued that one consequence is that aid workers engage with different belief fields, with no single authority to say what the truth will be said to be. Aid necessarily takes place in a situation of multiple sovereignty. Thus, in aid work resort to authority as a means to resolve disputes over what the truth is said to be is problematic. There is no Pope.

The most interesting aspect of this is that it overloads the role of experts in stating the truth of the matter, a role that interacts unhappily with an evident lack of robust cause–effect relationships required by and associated with, ideas of global best practice. If donor and recipient experts cannot agree, there is no way out of this tangle.

Third, the presence of unstable observed cause–effect relations suggesting heterogeneity offers specific lessons as the need to manage diversity engages with the issue of multiple sovereignty: the fact that aid involves at least two sovereign powers. Compared with project design or evaluation under single sovereignty, it is far harder to conceal or manage the instability of knowledge about cause–effect relationships.

This situation has led to the conservative responses I have discussed. However, the world is changing. Rapid growth in regions that used to contain large numbers of poor people means that, increasingly, poverty happens in areas with fragile states, or perhaps no state at all. The global economic capacity to support poor people is increasing. In a generation, if not earlier, the environment for aid will shift from a country transitioning to middle-income status to regions with weak states, mainly in Africa (Kharas & Rogerson, 2012). Yet desires for shared accountability have marked—not increased participation of beneficiaries, but greater involvement of the locally powerful.

Under such circumstances, a conservative rather than a radical response to the issue of diversity does make sense, and it enables donors to defend the position of their experts. This would be in preference to an intellectually consistent acceptance of the radical position that overtly accepts ignorance, mainly because donors’ experts seem to have no idea of just how donors could organize to do so.²¹

Yet, it should be argued that as the aid environment becomes tougher, so we may see emergence of mixed tactics that combine the brute force implied by existing LFA methods with the greater tactical flexibility possible when higher levels of trust allow for mission-based tactics. On one hand, massive sectoral programs based on line ministries may devote resources to push through whatever works to extend immunization programs, primary schooling, and so on. On the other hand, when mutual trust can exist, far cheaper interventions may secure mutually desired outcomes.

Wider Implications: Policy Government

So what about modern governance techniques in rich countries? Consider

[T]he apparent rejection or disappearance of religion or theology in fact conceals the continuing relevance of theological issues and commitments for the modern age. . . . [T]he process of secularisation or disenchantment that has come to be seen as identical with modernity was in fact something different than it seemed . . . the gradual transference of divine attributes to human beings (an infinite human will), the natural world (universal mechanical causality), social forces (the general will, the hidden hand), and history (the idea of progress) . . . What actually occurs in the course of modernity is thus not simply the erasure or disappearance of God but the transference of his attributes, essential powers, and capacities to other entities or realms of being. The so-called process of disenchantment is thus also a process of reenchantment in and through which both man and nature are infused with a number of attributes or powers previously ascribed to God. To put the matter more starkly, in the face of the long drawn out death of God, science can provide a coherent account of the whole only by making man or nature or both in some sense divine. (Gillespie, 2008, pp. 272-273, 274)

To develop Gillespie’s argument, we might conclude that to say that social change is unknowable in a predictive sense is simply to say that, like God, it is ineffable: too much for us to know. Gillespie links “modernity” to its theological origins, specifically to the preservation of habits of thought, especially of “belief in belief.” Fforde (2005) reported how the response of the great majority of those citing Levine and Zervos (1993) was to ignore its central findings: that there are almost no robust relations between policy and outcome viewed globally and in standard ways. Such patterns are preserved in the reinvention of development and this helps to explain the conservative rather than radical response to our revealed ignorance. It is too much to ask the mainstream to abandon the belief that, individually or in groups, they know that what they believe in is true (Friedman, 2006). Aid workers and donor officials are creatures of the rich countries most of them come from and where nearly all of them are educated. It is therefore deeply revealing about how rich countries govern themselves that the aid industry should organize in the way that it does.

Footnotes

Acknowledgements

The author thanks Bob Smith, Joerg Wischermann, Nguyen Quang Ngoc, Tran Huy Chuong, and Nguyen Dinh Huan for collegial discussion, and Adrian Wood, Doug Porter, Michael Webber, Minna Hakkarainen, Jan-Olov Agrell, Bob Smith, Richard Sacks, and Martin Gainsborough for comments on earlier drafts. In particular, he thanks anonymous referees.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biography

Adam Fforde is a professorial fellow at the Victoria Institute for Strategic Economic Stud, Victoria University, Melbourne, Australia, and has long made most of his income as a development consultant.

References

AusAID. (2005). AusGuideline: Activity design 3.3—The logical framework approach. Retrieved from www.ausaid.gov.au/ausguide/pdf/ausguideline3.3.pdf

Capolupo

(2009). The new growth theories and their empirics after twenty years. Economics, 3(2009-1), 1-72.

Cowen

Shenton

(1996). Doctrines of development. London, England: Routledge.

Crombie

A. C.

(1953). Robert Grosseteste and the origins of experimental science 1100-1700. Oxford, UK: Clarendon Press.

Development Advisory Committee (DAC), Organisation for Economic Cooperation and Development. (2015). Criteria for evaluating development assistance. Retrieved from http://www.oecd.org/development/evaluation/daccriteriaforevaluatingdevelopmentassistance.htm

De Jong

. (2011). False binaries: Altruism and selfishness in NGO work. In Fechter

A.-M.

Hindman

(Eds.), Inside the everyday lives of development workers: The challenges and futures of Aidland (pp. 21-41). Sterling, VA: Kumarian Press.

Dunn

(2000). The cunning of unreason: Making sense of politics. New York, NY: Basic Books.

Fechter

A.-M.

Hindman

(2011). Inside the everyday lives of development workers: The challenges and futures of Aidland. Sterling, VA: Kumarian Press.

Ferguson

(1990). The anti-politics machine: “Development,” depoliticization and bureaucratic power in Lesotho. Cambridge, UK: Cambridge University Press.

10.

Ferguson

(1997). Development and bureaucratic power in Lesotho. In Rahnema

Bawtree

(Eds.), The post-development reader (pp. 223-233). London, England: Zed Books.

11.

Fforde

(2005). Persuasion: Reflections on economics, data and the “homogeneity assumption.” Journal of Economic Methodology, 12, 63-91.

12.

Fforde

(2009). Coping with facts—A skeptic’s guide to the problem of development. Sterling, VA: Kumarian Press.

13.

Fforde

(2010). Responses to the policy science problem: Reflections on the politics of development. Development in Practice, 20, 188-204.

14.

Fforde

(2013). Understanding development economics: Its challenge to development studies. London, England: Routledge.

15.

Fforde

(in press). Reinventing development: Managing diversity and complexity. Sterling, VA: Kumarian Press.

16.

Fforde

Seidel

(2015). Cambodia—Donor playground? Defeat and doctrinal dysfunction in a hoped-for client state. South East Asian Research, 23(1), 79-99.

17.

Friedman

(2006). Democratic competence in normative and positive theory: Neglected implications of “The nature of belief systems in mass publics.” Critical Review, 18(1-3), 1-43.

18.

Gillespie

M. A.

(2008). The theological origins of modernity. Chicago, IL: University of Chicago Press.

19.

Government of Australia and Government of China. (2013). Memorandum of understanding for a development cooperation programme between Ministry of Commerce (MoC) of the People’s Republic of China and The Australian Agency for International Development (AusAID) of the Government of Australia. Retrieved from http://dfat.gov.au/about-us/publications/Pages/memorandum-of-understanding-for-development-cooperation-partnership-between-ausaid-and-china-ministry-of-commerce.aspx

20.

Gundlach

(1999). The impact of human capital on economic development: Problems and perspectives. In Tan

J. L. H.

(Ed.), Human capital formation as an engine of growth (pp. 7-33). Singapore: Institute of Southeast Asian Studies.

21.

Hageboeck

(2009). Trends in international development evaluation theory, policy and practices. Washington, DC: United States Agency for International Development.

22.

Hinsley

(1986). Sovereignty (2nd ed.). Cambridge, UK: Cambridge University Press.

23.

Hoffman

(1994). Auftragstaktik: Mission-based leadership. Engineer, 24(4), 50-55.

24.

Holma

Kontinen

(2011). Realistic evaluation as an avenue for learning for development NGOs. Evaluation, 17(2), 181-192.

25.

Hoover

Perez

(2004). Truth and robustness in cross-country growth regressions. Oxford Bulletin of Economics and Statistics, 66, 765-798.

26.

Hsiao

(1987). Identification. In Eatwell

Milgate

Newman

(Eds.), Econometrics (pp. 714-716). London, England: Macmillan Press.

27.

James

(2011). Theory of change review: A report commissioned by comic relief. Retrieved from http://mande.co.uk/2012/uncategorized/comic-relief-theory-of-change-review/

28.

Kenny

Williams

(2001). What do we know about economic growth? Or, why don’t we know very much? World Development, 29(1), 1-22.

29.

Kharas

Rogerson

(2012). Horizon 2025: Creative destruction in the aid industry. London, England: Overseas Development Institute (ODI).

30.

Lakatos

(1970). Falsification and the methodology of scientific research programmes. In Lakatos

Musgrave

(Eds.), Criticism and the growth of knowledge. Cambridge, UK: Cambridge University Press.

31.

Levine

Renelt

(1992). A sensitivity analysis of cross-country growth regressions. American Economic Review, 82, 942-963.

32.

Levine

Zervos

S. J.

(1993). What have we learnt about policy and growth from cross-country regressions? The American Economic Review, 83, 426-430.

33.

Lindauer

D. L.

Pritchett

(2002). What’s the big idea? The third generation of policies for economic growth. Economia, 3, 1(Fall), 1-39.

34.

Molund

Schill

(2004). Looking backward, moving forward. Stockholm, Sweden: Sida. Available from www.sida.se

35.

Mosse

(2005). Cultivating development: An ethnography of aid policy and practice. London, England: Pluto Press.

36.

Nelsen

J.T.

(1987) Auftragstaktik: A case for decentralised battle, Parameters, September, 21-34.

37.

Nisbet

R. A.

(1969). Social change and history: Aspects of the Western theory of development. Oxford, UK: Oxford University Press.

38.

Norlund

T. N.

Tuyen

N. D.

(2003). Dealing with the donors: The politics of Vietnam’s comprehensive poverty reduction and growth strategy (Policy Papers 4/2003). Institute of Development Studies, University of Helsinki.

39.

Organisation for Economic Co-Operation and Development. (2012). Development: Aid to developing countries falls because of global recession. Retrieved from www.oecd.org/dac/aidstatistics/developmentaidtodevelopingcountriesfallsbecauseofglobalrecession.htm

40.

Olasan

(2012). Cross-country growth empirics and model uncertainty: An overview. Economics, 6, 2012-2016.

41.

Prasad

Rogoff

(2003, September 1). The emerging truth of going global. Financial Times , p. 21.

42.

Pressman

J. L.

Wildavsky

(1973). Implementation: How great expectations in Washington are dashed in Oakland; or, Why it’s amazing that Federal programs work at all, this being a saga of the Economic Development Administration as told by two sympathetic observers who seek to build morals on a foundation of ruined hopes. Berkeley: University of California Press.

43.

Pritchett

Woolcock

Andrews

(2011). Capability traps: Techniques of persistent development failure (Background paper, The 2011 World Development Report on Development, Conflict and Fragile States). Washington, DC: World Bank.

44.

Proctor

Schiebinger

(Eds.). (2008). Agnotology: The making and unmaking of ignorance. Palo Alto, CA: Stanford University Press.

45.

Rahnema

Bawtree

(1997). The post-development reader. London, England: Zed Books.

46.

Rist

(2008). The history of development—From Western origins to global faith (3rd ed.). London, England: Zed Books.

47.

Rottenburg

(2009). Far-fetched facts: A parable of development aid. Cambridge, MA: MIT Press.

48.

Shore

Wright

(1997). Policy: A new field of anthropology. In Shore

Wright

(Eds.), Anthropology of policy: Critical perspectives on governance and power. London, England: Routledge.

49.

Sloan

(2012). Military doctrine, command philosophy and the generation of fighting power: Genesis and theory. International Affairs, 88, 243-263.

50.

Spruyt

(1994). The sovereign state and its competitors: An analysis of systems change. Princeton, NJ: Princeton University Press.

51.

Thomson

Barnett

Yaron

(2009, April). Monitoring and evaluation systems for DFID research (CNTR 20080542, Final Report). Oxford Policy Management. Retrieved from http://r4d.dfid.gov.uk/pdf/outputs/consultation/Monitoring_and_Evaluation_Systems_for_DFID_Research-FinalReport.pdf

52.

Tilley

(2000). Realistic evaluation: An overview. Presented at the Founding Conference of the Danish Evaluation Society. Retrieved from http://evidence-basedmanagement.com/wp-content/uploads/2011/11/nick_tilley.pdf

53.

Vardi

(2008). The enigma of German Operational Theory: The evolution of military thought in Germany, 1919-1938 (Doctoral dissertation). The London School of Economics and Political Science, London, England.

54.

Vogel

(2012). Review of the use of “Theory of Change” in international development: Review report (Report for DFID). Retrieved from http://www.dfid.gov.uk/r4d/pdf/outputs/mis_spc/DFID_ToC_Review_VogelV7.pdf

55.

Widder

(2002). Auftragstaktik and innere fuehrung: Trademarks of German leadership. Military Review, 82(5), 3-9.

56.

Willis

(2005). Theories and practices of development. London, England: Routledge.

57.

Ziegler

J. A.

DeGrosky

M. T.

(2008). Managing the meaning of leadership: Leadership as “Communicating Intent” in Wild 4 271-land Firefighting. Leadership, 4, 271-297.