Abstract
In debates about complex matters, there is a tendency for the positions of academics on the rival sides to ossify: difficulties in their own positions are simplified or overlooked, while problems in the opponents’ views are treated as devastating. Certainly this is the case with the lively debate about evidence-based practice (EBP). It is into this contested terrain that Tone Kvernbekk enters, hoping to shed some light, if not to bring peace. This is how she describes the orientation of her book: There are enthusiastic advocates who think it is irresponsible and downright unethical to try untested strategies in classrooms and who produce the kind of evidence it is assumed can serve as a base for practice. There are critics who view EBP as positivist, implying both an illegitimate instrumentalization of practice and severe restrictions on the freedom of professionals to exercise their judgment and who therefore wish to reject it. I shall do my best to walk a middle ground and be, as it were, a critical friend to EBP. (p. 3)
And indeed, her book provides a detailed, balanced, clear, and often elegant discussion of the core issues. Nevertheless, it must be said that reading this book is not a task for wimps, and it certainly cannot be accomplished at one sitting, for the complexities, generally overlooked by supporters and critics alike, are great. There are chapter-length discussions of the causal landscape, the logic of intervention, reproducibility, and what does it take for an intervention to work – all of which are wonderful and packed with material. Turning to the Index, philosophers of science Achinstein, Audi, Glymour, Hacking, Laudan, Price, Toulmin, and others each are referenced a couple of times, Nancy Cartwright – perhaps the heroine of the book – is referenced 26 times (more if you include ‘causal cakes’ and ‘nomological machines’); INUS conditions get the nod 13 times; randomized controlled trials earn 16 citations, while Gert Biesta’s less technical writings get 14.
So, what is going on in the literature Kvernbekk deals with? At first sight (and at second and third as well) the antipathy to EBP seems odd, for who would want to adopt – in such an important area of life – policies or practices for which there was no supporting evidence? We certainly would not select a surgical procedure that was lacking in evidential support, so why adopt an educational practice that was similarly lacking? The opposition to EBP becomes even stranger when a point made by Richard Pring (2004) is contemplated, namely, that it undoubtedly is the case that experienced teachers, faced with classroom problems, adopt strategies that their bodies of prior experience indicate are likely to be effective – in other words, these bodies of experience are the source of evidence. And it can be argued that the same holds true in many other areas of life – the use of evidence is pervasive. Furthermore, it is evidence of causal effectiveness, as Kvernbekk stresses – although this is often masked by the verbal ploy of using synonyms (p. 6): Practical pedagogical language is replete with causal terms such as improvement, bring about, enable, affect, contribute, impact, work, promote, facilitate, precipitate, motivate, depend, influence, encourage, discourage, produce, hinder, reduce . . .
It should be obvious from the discussion above that along with the concept of ‘cause’, ‘evidence’ also is problematized in the EBP debate, but detailed discussions of it are lacking in the burgeoning literature. The focus largely has been on whether or not evidence that is derived from randomized controlled trials or ‘true experiments’ (RCTs) is the ‘gold standard’. Many supporters of EBP seem to regard the kind of evidence used by the teachers in Pring’s example as being of dubious value, and as falling far short of the trustworthiness of the causal knowledge that is produced by well-conducted RCTs – which of course fuels the perception of opponents of EBP that it has no place for teacher expertise or teacher professionalism. On balance, here, the critics of the important role RCTs play in the EBP position have a point – much useful evidence comes from non-experimental sources, and the limitations of randomized studies tend to be overlooked.
Kvernbekk shows that there is much to think about when ‘evidence’ is problematized. She discusses the debates about evidence that have taken place in the literature of philosophy of science, she summarizes the debate on the question of whether evidence is found or made, she has a lengthy examination of what RCTs can and cannot deliver, and she discusses the so-called hierarchy of evidence. While she certainly is no enemy of the evidence that comes from personal experience and observation, perhaps she could have developed a little more strongly the point that qualitative investigations of various kinds can produce reliable causal findings. I recall a wonderful spoof in the British Medical Journal some years ago (Smith and Pell, 2003) arguing that the use of parachutes should be put on hold, for no randomized study could be found that demonstrated their causal efficacy. The evidence about parachute effectiveness was merely anecdotal and largely based upon observation. The authors called on the members of the medical research community to volunteer as subjects in a randomly assigned control-group study to remedy this lack of solid evidence!
Kvernbekk’s discussion of RCTs leads her to tackle the issue of generalization that is the logical heart of EBP – evidence gathered in a research or evaluation study can be generalized to apply in other settings (i.e., for example, in your or my classroom). So she points out that the well-done RCT provides evidence that the treatment worked there, but what is wanted is evidence that it will work here. Unfortunately for the supporters of RCTs and of EBP, the better or tighter the experiment has been, the stronger is the claim that any finding applies there – to the setting in which the study was run – but the trade-off is that the claim that the results apply anywhere else is weakened. This is usually discussed in the literature as the trade-off between internal and external validity.
Kvernbekk draws on the work of Nancy Cartwright (1999) to make the point that a great many background conditions need to be in place for a causal process or mechanism to operate – for a chemical reaction to occur, for example, the ingredients need to be pure, the environment needs to be free of contaminants, the right conditions of temperature and pressure need to obtain, a liquid medium may need to be present, perhaps a catalyst is required, and so on. All these ingredients are part of the causal or nomological machine (or system, as Kvernbekk likes to conceptualize it), and their presence or absence will determine whether the causal process will work here. Furthermore, some of these ingredients are necessary for the machine to work (e.g. oxygen is necessary for combustion to occur, but clearly it is not sufficient – other factors, members of what Cartwright calls ‘the support team’, are required), and there may be bundles of necessary and sufficient conditions each of which can cause the desired effect to appear – hence Kvernbekk’s important discussion of INUS conditions. Causation is a complex topic, but sadly (until now) the complexities generally have been ignored in the discussions of EBP in the educational literature. With Evidence-based Practice in Education, there can no longer be any excuses!
Strong antipathy to this ‘causal paradigm’ and to the ‘means/ends reasoning’ that accompanies it, both of which are part-and-parcel of EBP, seems to be what drives its critics within the philosophy of education community. The nature of this hostility, to my mind, is the central issue and Kvernbekk does a workman-like job in discussing the relevant complexities (and indeed these are complex). However, her analysis is not highlighted in any way and so tends to get lost at the end of Chapter 4. Hence, my friendly suggestion that, in the next edition, from pages 46 to 54, the text should be set in CAPS so that readers give it the attention it deserves.
There is much more to be said about Kvernbekk’s discussion of EBP. For example, what is the precise relation between a policy or practice, on one hand, and evidence, on the other? In what sense is policy or practice supposed to be based on evidence? Once again, she provides an enlightening discussion of the complexities that lie below the surface here. The relevant but oft-overlooked topic of randomness is also given an airing: There is much hidden randomness in both nature and human life . . . To bring this back to education; a preliminary conclusion is that if randomness is a general problem in education, it is particularly so in EBP, which aims at developing schemes that do not go awry, but deliver planed and desired results in a stable and reliable manner. (p. 150)
Kvernbekk mentions that the topic of unintended side effects is important, and perhaps further elaboration is warranted. Figures as different as Karl Popper (1957) and Michael Scriven (1991) have insisted that when an intervention (practice or policy) has been introduced, the intended consequences sometimes fail to appear while unintended side effects frequently are of major significance. Scriven even devised an approach to program evaluation – ‘goal free evaluation’ – to specifically deal with this issue. There is no reason to believe that practices derived from EBP are free from this complexifying phenomenon.
In light of the fact that much of the material in this fine book is complex, every little simplification that could have been made would have been welcome. The example of character education could have been dropped and her adoption of Toulmin’s theory of argumentation did not seem necessary.
So what is the upshot? Kvernbekk herself sums it up well: The book is about unpacking the causal core of EBP . . . My overall judgment of EBP… is that the idea of EBP makes good sense, but that it is much more complicated than both advocates and critics have thought. EBP is no magic bullet, but sensibly dressed up – and in recognition of limitations and complications – it can serve certain (but not all) aspects of educational practice well. (p. 11)