Evidence-based approaches to education

Evidence-based education is not merely flavour of the month, but of the year, or even of the decade. Education leaders, yearning for all that is best in terms of teaching and learning, may find the idea very appealing. It could feel like a wonderful gift – a seductive source of educational authority independent of subjective values. ‘Surely’, our leaders might think, ‘with such an approach, staff cannot dismiss us as pushing personal obsessions about direct instruction, group work or how reading should be taught. No – with evidence-based power, a new situation has arisen. We can now acquire ‘scientific’ support for our policies. We can put staff under moral pressure to follow us, under pain of being “unscientific” if they resist.’

Readers will sense a ‘but’ hovering in the wings. Cartwright and Hardie (2012) wrote a book about evidence-based policy, where they keep returning to the aphoristic question, ‘It worked there. Will it work here?’ They were dissecting evidence-based policy in general, but their question can be applied to educational policies for teaching approaches and even for educational leadership styles. The authors make many crucial points about the importance of context. I want to focus on their ‘it’. ‘It’ could be many things in educational contexts. Popular candidates include group work, direct instruction, mastery mathematics, project-based learning, synthetic phonics, whole language approaches, and transformational and transactional leadership. Yet when these policies are researched and then recommended, we may wonder whether their adherents really know what counts as implementing them. The ‘it’ can prove to be singularly elusive. For instance, what counts as a teacher employing direct instruction today, and applying that same method next week? For several teachers to adopt direct instruction in different classrooms, with different age groups, subjects, contexts, cultures and even countries?

A crude outline of ‘direct instruction’ might run as follows, though I will shortly be deconstructing it: the teacher works directly with a group of pupils, rather than the pupils being given tasks on which they work without the teacher. Direct instruction is likely to include episodes in which the teacher imparts knowledge by ‘telling’ pupils. The teacher has knowledge, and uses ‘direct’ ways to ensure that pupils acquire it.

Direct instruction is very much flavour of the month with DfE ministers such as Nick Gibb. An Ofsted review (2017) released as I write appears to encourage more direct instruction in Reception classes. Yet unless direct instruction can be clearly and unambiguously specified, robust empirical research into its ‘effectiveness’ is simply not feasible, or so I intend to argue in this article. School leaders should be very concerned about this issue at a time when they may feel under considerable pressure to steer their staff in the direction of ‘traditional’ approaches to teaching, with which direct instruction in particular is often associated.

Before I address this challenge head on, I explore an example where the ‘it’ is relatively straightforward to pin down (or so you might have thought). Consider ‘pupils sitting in rows’. If research indicates that this classroom arrangement is effective, it makes perfectly good sense for school leaders to recommend it to teachers, or so it would seem.

And yet this particular ‘it’ can cover a great variety of phenomena, depending on answers given to questions such as the following. Are pupils facing the front throughout and not talking? Are they allowed to turn and speak to each other? How often? What kind of dialogue is being encouraged? Why? What is the context? How old are the pupils? What subject is being taught? What degree and character of interaction takes place between teacher and pupils? Is the learning concerned part of long-term conceptual development, or is it knowledge or skills that can be acquired within a lesson or so? Are pupils used to ‘sitting in rows’ or is this a departure from the usual arrangements? Do they ‘sit in rows’ with this particular teacher, but not with others?

Despite the variety of legitimate responses to questions such as these, advocates of research-informed teaching might still argue for the usefulness of ‘sitting in rows’ research. They could urge that it would be good to know whether it is better to teach with the students sitting in rows than in some other seating configuration. If so, teachers and education leaders should take careful note.

Yet absurdities rapidly make their presence felt. For certain kinds of lessons are impossible like this. Sessions drawing on the performing arts such as drama and music come to mind. So, even were there to be something of an ‘effect size’, it would be unclear what teachers should do with it in their everyday decision-making. There would be so many reasons why it would sometimes be sensible to deviate from the ‘sitting in rows’ rule, and teachers would draw on their professional expertise to weigh up these reasons – an expertise that certainly does not consist of rules derived from research-supported ways of doing things.

I deliberately began with something that should have proved ‘easy’ for an empirical researcher, even though it may seem simplistic and artificial. Let us now return to ‘direct instruction’. Conventional quantitative empirical research into any given teaching method needs all its alleged instances to have something in common. Moreover, observers must be able to agree about whether it is being implemented in any particular lesson. However, as we will see shortly, this requirement and true teaching are not comfortable bed-fellows.

Unsurprisingly, characterizations of direct instruction vary. Consider one from Cook, Holland and Slemrod:

Lesson objectives that are clear and communicated in language students are able to understand…An instructional sequence that begins with a description of the skill to be learned, followed by modeling of examples and non-examples of the skill…shared practice…and independent demonstration of the skill. Instructional activities that are differentiated and matched to students’ skill levels to the maximum extent practicable. Choral response methods in which all students respond in unison to strategic teacher-delivered questions. (Cook et al., 2014: 202)

Needless to say, some of this is open to interpretation. What does ‘clear’ mean? What about that phrase ‘language students are able to understand’? Does it imply that the clarity concerned relates to how the students receive the instruction? Must all the students find it clear? How does the teacher know what they can ‘understand’? Is the teacher making a very general judgement about what pupils of that age and stage are likely to understand? Or are her decisions about the nature of her instruction drawing on some kind of assessment of this particular group of students? Or some combination? What counts as a lesson objective? Is it the same kind of thing regardless of the curriculum area concerned?

You can answer these questions in all kinds of ways, depending in part, of course, on what counts as understanding and what would be regarded as appropriate ways of detecting it. In any case, not all direct instruction lessons will begin with a skill description, because some direct instruction lessons will not concern skills, even if many will.

So, only brief scrutiny of direct instruction interpretations reveals a rich variety of classroom phenomena. In the light of this, perhaps ‘It worked there’ should be replaced with ‘That collection of processes that vary one from another – the one we have decided to place in a category named direct instruction – worked there’. Admittedly it does not trip so readily off the tongue.

If ‘direct instruction’ in this multi-faceted sense were researched, and turned out to have a significant effect size, it is far from clear what school leaders and teachers could do with such information. Perhaps teachers should never feel guilty about standing at the front and talking to students on some occasions. However, this leaves them entirely free to make all the detailed professional decisions that they have always made. The ‘direct instruction’ policy would only ever have one or two teeth. These would fail to bite unless the prevailing culture had been that teachers should never, ever stand at the front and tell pupils things. We may wonder whether any schools and teachers have ever really thought that.

Some versions of direct instruction are more extreme. Take Flores and Ganz, for instance:

The researchers implemented instructional procedures and instructor behaviors as directed in the instructor’s manual. These procedures and behaviors consisted of: (a) following the given script; (b) choral student responses; (c) the use of a clear signal to elicit student responses; (d) correction procedures for incorrect responses or responses that were not in unison; and (e) modelling skills, guiding students by responding with them, and asking students to respond independently. (Flores and Ganz, 2009: 42)

This may seem to be less open to interpretation than the first example. Yet ‘modelling skills’ could mean very different things, depending on the curriculum area, the age and stage of the pupils, and the variety of actions and activities the teacher might employ to ‘model’. ‘Modelling’ how to solve a quadratic equation with 13-year-olds would surely have little in common with modelling decoding of some simple pieces of text with 5-year-olds, or how to make good use of metaphor and simile in creative writing with 11-year-olds. Furthermore, as with the first example, ‘skills’ might not feature at all.

Setting these interpretation complexities to one side, a fundamental worry remains. Are these extreme versions of direct instruction actually teaching?

Here is an argument for the claim that they are not. Teachers always aim (and, surely, should aim) to promote understood knowledge. To explain and justify this, it must be admitted straight away that ‘understanding’ is an elusive notion, and is a matter of degree. Many years ago, the psychologist Richard Skemp coined a distinction between two sorts of understanding. He was thinking of mathematics in particular, but his ideas can be applied to many curriculum areas. He described and contrasted what he called relational and instrumental understanding (Skemp, 1989).

Skemp explains his distinction with an analogy. Imagine that Anna and Sarah are making their way to a wedding in an unfamiliar town. Each has been supplied by the host with a series of directions. ‘Left after the first roundabout on the way into town, under the railway bridge, third right, past the park and second left by the hypermarket. The church is on the right a few hundred yards down that road.’ Anna has no map of the town, whereas Sarah has one and can read it. If Anna meticulously follows the directions supplied, she will reach the church. It only needs one slip, however, for her to be lost, whereas Sarah can follow the directions with confidence, knowing that if she strays she has a remedy. Hence, she is less likely to go wrong in the first place. If she does make a mistake, she can find her way back to the sequence of directions, or even pick out a new way to the church. Sarah represents those with relational understanding of their mathematics (and, indeed, many other curriculum areas). Armed with a ‘cognitive map’, those with relational understanding are able to find a number of ways to solve a given problem. If they forget a particular procedure, or make a mistake in it, they can use and even construct an alternative route.

The map analogy captures something crucial about understanding. That church hosting the wedding (obviously) has a location. For the church to have a location is its relationships with other locations – half a mile west-south-west of the fire station, 400 yards north of the cinema and so forth. A position without these relationships makes no kind of sense. Similarly, when a knowledge item is situated on a cognitive map, its ‘situation’ comprises its relationships with an indefinite variety of other pieces of knowledge. A child learns that Michael Faraday discovered the dynamo. The content of the learning cannot be separated from who Faraday was – namely, a Victorian scientist. In turn, this is linked to calendars, the fact that Victoria was Queen until around 120 years ago, the nature of a scientist, and much more. The properties of dynamos are bound up with electricity, and again, of course, much else.

We have already noted that (relational) understanding is a matter of degree. This is one of the ways in which the cognitive map analogy is less than perfect. New knowledge needs a place on a child’s cognitive map. It might be thought that either that knowledge is on the map or it is not. Yet, when knowledge acquires a position on the map, it cannot be a matter of a momentary spasm of cognition, before which the new content has no place, and after which it is definitively located. What actually takes place is that learned material gradually becomes connected to other content in the child’s mind. Sometimes the putative connections are wrong. This is often a temporary problem, but errors may be long-lasting. Deeper understanding can only occur when at least some of these problems are overcome. The connectedness of any ‘one’ piece of knowledge is never fully comprehensive or complete.

True teachers will want to promote relational understanding in their pupils, at least to some degree, and at least for most subject areas. They know that knowledge without relational understanding, merely ‘inserted’ into pupils’ heads without regard to what else they already know and understand, fails to find a place on pupils’ existing cognitive maps. Now those maps need to undergo constant adjustment in order properly to assimilate and accommodate any new learning. If this does not happen, that new learning is isolated in the child’s mind. Hence it cannot be used and applied in an unrestricted variety of circumstances and contexts. At best, it can only be reproduced in contexts closely resembling those in which it was transmitted to them. It may be useful in a high stakes assessment regime that, in effect, gives value to certain kinds of teaching to the test, but it fails to involve the richer knowledge that a true education should prioritize.

Teachers cannot teach in a way that relates to what pupils already know and understand without permitting and even encouraging interaction between them and their students. Without it, they cannot discover anything about the current contents of their students’ minds. How much interaction? It is not possible to be precise. It will vary, depending on the age and class size concerned, the character of the knowledge being taught and many other factors. There must at least be the potential for information to emerge as a result of teacher–pupil interactions that will help the teacher decide how to teach from one moment to the next. She will choose her language, explanations, pace, tasks, questioning styles, sequencing and other aspects of pedagogy in response to what she judges to be the impact of new learning on pupils’ resident knowledge. Hence, whatever else an extreme direct instructor may be doing, and whatever value it may possibly possess, it cannot count as teaching.

Torgesen (2004) defended direct instruction, claiming: ‘Explicit instruction is instruction that does not leave anything to chance and does not make assumptions about skills and knowledge that children will acquire on their own.’ However, this approach is manifestly impossible. Teachers are bound to leave some things to chance, as they cannot comprehensively inspect the contents of, say thirty students’ minds at any one moment. At the same time, success will be less random if teachers can discover at least something of how students are responding to teaching during the process.

In arguing for this point, we should assume neither that pupils would never learn from extreme forms of direct instruction nor that no connections are made in a child’s mind unless the teacher explicitly intends this. For instance, some higher education lectures consist entirely of an academic talking, sometimes to very large numbers of students. No interaction between lecturer and students takes place, and often enough it is not intended that it should. Although sessions of this kind are sometimes rightly deplored, a skilful presenter can be engaging and students come away having gained a good deal.

However, higher education and school contexts normally differ significantly. Higher education lectures are primarily setting students up to begin their own learning about given topics, rather than attempting to ‘cover’ the content linked to any one module. Such cover would be quite impossible in the timescales concerned. School lessons, on the other hand, usually are trying to ‘cover’ content in some way or other.

I conclude by broadening the argument. Direct instruction is simply one currently fashionable way of categorizing approaches to teaching, and one contemporary ‘style’ that is being promoted by certain traditionalist educators who oppose anything that they regard as progressive education. Hattie is one well-known exponent of large-scale systematic reviews of educational research. He reports the so-called ‘effect sizes’ of various educational procedures, including ability grouping, co-operative vs individualistic learning, phonics instruction and student control over learning.

Some of Hattie’s categories embody clear membership criteria. What counts as holding pupils back a year or reducing class size is perfectly straightforward. Either the pupil is held back a year or not. Either you are reducing class size or not. For what it is worth, these examples can be subjected to ‘rigorous’ empirical research. Nevertheless, they resemble ‘sitting in rows’ in the sense that they seem to leave much of the vital and detailed decision-making to teachers. Yet they may come closest to examples of useful evidence-based policy. It is significant that they do not amount to teaching methods or approaches.

However, other descriptions of classroom approaches, such as ‘student control over learning’ do amount to characterizations of teaching approaches. These face a dilemma. Either they are characterized as a recipe or they are left as abstract and open to multiple interpretations.

On the one hand, the motivation for the recipe choice is that the method becomes readily researchable. It is clear what counts as implementing it, and observers can, in simple and straightforward fashion, agree on whether it is present in any given lesson or series of lessons. On the other hand, if the method is still researched but as flexibly interpretable, then teachers can take little from any effect size ‘proved’ by people such as Hattie. This is because they, the teachers, must still engage in their familiar day-to-day choices about which interpretation or combination of interpretations is appropriate in their particular context. Their choices may well vary from one day to another, and with good reason.

If the recipe horn of the dilemma is chosen, it is doubtful whether any such recipes have actually been researched. Few schools in their right minds would ever agree to follow a script precisely. If, despite my doubts, classroom procedures along the lines of scripts really have been tested out, what has been investigated was not true teaching, at least according to my earlier argument. This dilemma and the unsatisfactory nature of responses to either of its horns pose significant challenges to a significant body of educational research. The very idea of a researchable teaching method such as ‘direct instruction’ is seriously problematic. School leaders cannot and should not support classroom approaches of this kind on the ground that they are evidence-based.