The use and usability of evaluation outputs: A social practice approach

Introduction

The quote introducing this article is from a recent report on the way evaluations were used in the context of social cohesion ¹ and social fund programmes ² in Europe (taking Poland as a country case study). It embodies the high expectations of evaluations. At issue is their capacity to yield resources for managing developments in policy and practice. In short, evaluations ought to offer much by way of responsive policy development but this resource is often under-used or its potential remains opaque. The report (Ferry and Olejniczak, 2008) is a serious attempt to grapple with the issue of evaluation use in the European context and heralds a move to engage with an issue that has a substantial American literature, including some important syntheses or meta reviews, but has hitherto been relatively undeveloped in Europe.

This article addresses the development of this thinking by contributing some reflections that might be useful in building a systematic consideration of use into the design of evaluations in the European policy context. While it seems axiomatic that evaluations should be used, a planned and rehearsed approach to the way this may happen is not always undertaken as part of a design brief or proposal. Usually the discussion does not go beyond a description of the way in which the output from an evaluation might be disseminated. This said, we know, as others have suggested (e.g. Fleischer and Christie, 2009; Weiss, 1979), use practices are not necessarily predictable, positive or controllable.

This article emphasizes a distinction between the contexts of use of evaluations on the one hand, which refers here to the practices associated with the way in which the outputs of an evaluation may or may not be used in an organizational setting or policy context as a resource for onward practice, policy or decision making. It highlights the context of an evaluation; in particular, the organizational capacity of potential ‘users’ to respond to evaluation ‘outputs’ in its broadest sense (it also refers to what has been termed ‘process use’ (Patton, 1998) or the way in which an evaluation interacts with its ‘social context’ to create effects in an organization policy context while it is being undertaken). On the other hand, I refer to the usability of evaluation (the extent to which the design of an evaluation – both its output and the way it is undertaken – maximizes, facilitates or disables its potential use).

The focus, in particular, is the interplay between the processes and practices associated with use in the way I define above (which may be informal and tacit as well as more conventional conceptions associated with a text in the form of a report) and the processes and practices associated with usability. It will do so by suggesting that foregrounding evaluation as a social practice is a useful frame for considering the distinction. In turn, the argument for giving this approach attention is based on a view that the impact of an evaluation, in particular the use of its outputs, can be measured in terms of the changes in practice it might shape or influence (including reified practice in the form of systems or depicted in economic performance). To adopt this approach, we need to have a robust understanding of what practices are and how they might change.

The article has been developed from three sources: first, evaluative research on higher education within the UK and internationally (see Saunders et al., 2011); second, from participation in the exploration of evaluation use as part of the work of DG of Regional Policy in the European Union ³ ; third, the approach was developed and trialled in a large EU funded R&D project named PALETTE in which the evaluation work package had an important part. The focus, summarized in the final report on the project (PALETTE, 2009), synthesizes the extensive research undertaken in the project of responses of members of the PALETTE project to formative evaluation. The conclusion of the report offers summative observations and recommendations concerning evaluation, change and its use in supporting learning within Communities of Practice. It argues that:

Integrating evaluation more solidly in decision-making raises questions about the ‘usability’ of evaluation outputs for such a task. By usability we understand the relationship between the ‘design’ of the evaluative activity and the use to which it is put. (PALETTE, 2009: 37)

A note on the background to studies of ‘use’

As I note above, there has been an extensive meta-analytical literature from the USA on use and utilization of evaluation. This meta-analytical preoccupation reflects the difficulties in using the concept of ‘use’. The term has its problems, as Kirkhart (2000) reminds us. In the estimation of the effects of an evaluation, Kirkhart suggests that the term is an:

awkward, inadequate, and imprecise fit with non results based applications, the production of unintended effects, and the gradual emergence of impact over time. Second, when the history of influence is traced from the approach of results based use, the historical roots of other dimensions of evaluation impact are erased. Process use, for example, incorrectly appears as an afterthought, a late arrival. Third, fitting other types of influence under a results based paradigm continues to privilege the concept of results based use. Other types of use are secondary, ‘tacked on’ or seen as important primarily in the service of results based use. (Kirkhart, 2000: 6)

Kirkhart (2000: 7), in commenting on the work of Weiss (1979) some 20 years before, suggests that reconstructing our thinking, as Weiss does, by making a distinction between utilization and use, thus widening the scope and implications of the focus on how evaluations might create ‘effects’, still requires a discursive shift to the idea of ‘influence’. However, in examining broad questions derived from reviews of the literature about the use of what we might call ‘enquiry based knowledge’ from social research (later applied to evaluation), it reveals that a diverse array of meanings is attached to the term. Weiss suggests that much of the ambiguity in the discussion of ‘research utilization and conflicting interpretations of its prevalence and the routes by which it occurs-derives from conceptual confusion’. She argues that ‘if we are to gain a better understanding of the extent to which social science research has affected public policy in the past, and learn how to make its contribution more effective in the future, we need to clarify the concept’ (Weiss, 1979: 426).

The synthesizing work of both Kirkhart and Weiss (also in Weiss, 1998) provides us with an overview of studies on ‘use’ and ‘utilization’ that act as a resource for further overviews in which the issue of use is reconsidered. They draw our attention to the diversity of use, its unpredictability and its political nature. They identify many potentially relevant factors, but are equivocal about which are most related to increasing positive evaluation use. They tend to emphasize the diverse ways in which evaluations are used rather than the conditions under which positive use (i.e. for development) seems to occur or why, theoretically, practices of use are as they are (exceptionally, Patton’s [1997] work significantly identifies a range of factors; see below).

Recent commentators have asserted, however, that the phenomenon of ‘use’ is multifaceted (Ledermann, 2011) and prescriptions are of limited value. Also relevant is the assertion from Hofstetter and Alkin (2003), with reference to the US environment, that:

for as long as modern-day evaluations have existed, so too have questions about evaluation use. As with social science research and knowledge, there remain ongoing concerns about the level of importance of conducting evaluation and producing information that leads to action, be it in creating policy, decision making, or changing how someone may regard a particular issue. We assume that our efforts are rewarded with some importance or action, although measuring the effects evaluation has will continue to prove problematic. (Hofstetter and Alkin, 2003: 219)

A more recent review by Johnson et al. (2009), for example, with reference to the research on evaluation use between 1986 and 2005, point to shifts in focus and concern:

Moving beyond the first quarter century of use research, the new millennium has witnessed theoretical activity that has reconceptualised the field’s understanding of its impact. Scholars now view evaluations as having intangible influence on individuals, programs, and communities. Focusing solely on the direct use of either evaluation results or processes has not adequately captured broader level influences. (2009: 378)

These wider considerations in the discussion on use refer to the way the knowledge resources produced by evaluations have effects on practice that have a more subtle nuance.

Important for the Johnson et al. study was the earlier Cousins and Leithwood (1986) work in which they identified 65 empirical studies of evaluation use conducted between 1971 and 1985 through computerized searches of keywords including ‘evaluation utilization’, ‘data use’, ‘decision making’, and ‘knowledge utilization’ (Johnson et al., 2009: 379). Of particular interest to them were two groups of factors that impacted on use; namely, (a) characteristics of evaluation implementation, and (b) characteristics of the decision or policy setting (p. 379). Fleischer and Christie (2009), building on this, predominantly, American literature on use (see above), demonstrate the different emphases and preoccupations.

We also have the important contribution to this debate offered by successive editions of Utilization-Focused Evaluation, by Patton (1997), in which he asserts that the potential for use is largely determined a long time before ‘a study is completed’, thus pointing to the importance of ‘design’, or usability. He argues that ‘utilization-focused evaluation emphasises that what happens from the very beginning of a study will determine its eventual impact long before a final report is produced’ (1997: 20).

For Patton, an evaluation that is focused on use, ‘concerns how real people in the real world apply evaluation findings and experience the evaluation process . . . it is about intended use by intended users’ (1997: 20). He acknowledges that in an evaluative environment there are likely to be an array of evaluation users as well as uses of programme outputs, but he argues for a particular emphasis on primary intended users and their commitment to specific concrete uses. Further, he argues for the highly contextualized nature of these kinds of decisions, along with the overview offered by Shulha and Cousins (1997) in which they identify the rise of considerations of context as critical to understanding and explaining use.

This is broadly the stance taken in this article. While Patton offers us an emphatic approach to the problem of use, he emphasizes, in particular, a relatively narrow concern with potential users as commissioners. The enduring issue of use is graphically illustrated by the following observation by Weiss in which she emphasizes the expectations of use we may have as evaluators:

As evaluators, we undertake our studies with the intention of helping decision makers make wiser decisions. We provide evidence that shows the successes and shortcomings of programs. We identify some of the factors that are associated with better and worse outcomes. And, we often try to explain how the program works in practice and why it leads to the effects we observe. We expect that these data will feed into the decision making process and influence the actions that people take at the staff level, at management levels, or in the higher reaches of decision making. (Weiss, 2005: 286)

Fleischer and Christie (2009) note these important reviews of studies of ‘use’ in the US evaluation literature. Building on their ‘overview of overviews’ it is possible to identify the typology of use presented in Figure 1.

OPEN IN VIEWER

Figure 1.

Typologies of use.

The synthesis of ‘use domains’ identified by Fleischer and Christie (2009) are helpful, but from the approach of this article, these domains can be understood as clusters of ‘use practices’. What is missing is a way of approaching the interplay between context and design in the discussion of evaluation use and usability. This article argues that a social practice approach suggests that ‘use’ and ‘usability’ are essentially about the intersection of forms of practice and the way they may be changed. If we are to improve the capacity of evaluations to be used (to contribute to changes in practice) then we need, not only to typologize forms of use, which has a rich and well rehearsed history, but have a robust understanding of what practices are, how they might change and, further, identify practices of evaluation design that might encourage use.

Using social practice theory to examine use and usability

To emphasize use and usability as practice requires an understanding of practice as situated or contextualized. The following observation made by Ledermann in her recently completed research, is persuasive:

The study reported in this article follows the calls in recent years that it is time to abandon the ambition of finding ‘the important’ characteristic for use and to adopt a focus on context-bound mechanisms of use instead. (2011: 160)

The implication of adopting this approach is that it is important to discern the domain and the clusters of practices that are emphasized or inhabit that domain. It allows for a broad focus on the kinds of ‘effect’ an evaluation might have but draws our attention to the ‘situatedness’ of these effects and the practices that a) might bring them about and b) emphasize on the effects as essentially concerned with changes in ‘practice’.

The focus of this article is that people are carriers of practices: they enact in a specific reservoir of behaving, understanding and responding in ways that are to a certain extent particular to them in a social field (Bernstein, 1999, distinguishes between ‘reservoir’ and ‘repertoire’ to describe this process of individuation from structural characteristics). Individuals articulate repertoires of recognizable practices: ‘patterns of bodily behaviour and routinized ways of understanding, knowing and desiring’ (Reckwitz, 2002: 250). The focus on analyses of use is therefore best located in the practices, not the individuals involved.

As phenomena, practices can be defined as:

A ‘practice’ (Praktik) is a routinized type of behaviour which consists of several elements, interconnected to one another: forms of bodily activities, forms of mental activities, ‘things’ and their use, a background knowledge in the form of understanding, know-how, states of emotion and motivational knowledge . . . [A] practice represents a pattern which can be filled out with a multitude of single and often unique actions reproducing the practice . . . the single individual – as a bodily and mental agent – then acts as the ‘carrier’ of a practice – and, in fact, of many different practices which need not be coordinated with one another. (Reckwitz, 2002: 249–50)

By social practice we mean the recurrent, often unconsidered, sets of behaviours or ‘constellations’ that together constitute daily life. These behaviours are shaped by knowledge resources on which actors routinely draw. These knowledge resources can be tacit and informal as well as more explicit knowledge areas and skills. In that evaluations produce knowledge resources, it is these knowledge resources to which we refer in discussing ‘use’ and ‘usability’. For the purposes of this article then, practices can be usefully conceptualized as sets or clusters of behaviours forming ways of ‘thinking and doing’ associated with evaluation use.

The application of social practice theory to evaluation use leads us to take an approach that emphasizes the way evaluation use is embedded in specific national and regional contexts. Without this appreciation of the contextual factors that affect use, use strategies – as part of the design of an evaluation – are likely to be less effective (Pawson and Tilley, 1997: 114). This means that in this approach, evaluation use:

Stern (2006: 293) has outlined the complexities of the widely shared view that evaluation is a ‘practical craft’ by identifying at least seven taken for granted assumptions about where the emphasis should be. These range from evaluation as technical practice, as judgement, as management, as polemical and as social practice.

What is the added value of the ‘social practice’ dimension? I argue that the value is in three ways. First, understanding use and usability as ‘practice’ provides analytical purchase on the knowledge resources stakeholders use to make sense of evaluation and the response (as practice) they might have to both its process and outputs (these might be negative as well as positive). Second, it enables a nuanced approach to how evaluations have influence, leading to change, as well as how the experience of an evaluation becomes a knowledge resource for action. In that we know ‘practice’ tends to be ‘engrooved’ (see Huberman, 1994), it is difficult to get practices to change. If we adopt the perspective that use is about ‘changing practice’, what might be the nature of the knowledge resources available from an evaluation that creates the momentum for change (i.e. for it to be used)? Third, it enables the identification of practices of ‘engagement’ (with evaluation outputs, for example) that facilitate the process of establishing what changes in practice flow from the knowledge resources an evaluation might yield.

In sum, the social practice approach enables analysis that focuses on the underlying dynamic of change embedded in the idea of ‘use’.

Modelling evaluation use and usability practices

Using the social practice approach to think about use practices in Higher Education, recent research suggested evaluation use inhabits at least four ‘domains’ of practice. This model was based on the analysis of 16 international case studies of evaluative practice in the Higher Education policy sector (see Saunders et al., 2011). I address these domains in detail below in Figure 2.

OPEN IN VIEWER

Figure 2.

Use Practice clusters.

These domains are not definitive but serve the purposes of this article by illustrating how the contexts of evaluation use produce different types of ‘practice cluster’. It is in line with the view of Ledermann (2011), above, concerning the ‘situatedness’ of evaluation use. As Stern (2006) points out, these expressions may well be very different in focus and not particularly consistent; it is these differences that Figure 2 illustrates in the context of use.

The indicative differences depicted by using a social practice approach in Figure 2 are drawn from the Higher Education sector. If this broad model of domains of practice is generalizable to other areas (health, criminal justice, general education, social services), then it suggests how the practice clusters associated with evaluation use differ between the domains in which evaluative practice occurs (to regulate national or regional systems or to enhance quality within an organization). Further research is required to test the hypothesis that these domains of evaluative practice associated with use are embedded in social policy areas in general.

The Ferry and Olejniczak report (2008) is based on a study that proceeded by addressing five propositions about the factors that might influence effective use within the paradigm identified above, and is a systematic and thorough overview of a range of methodological, paradigmatic and strategic considerations.

What is helpful for the purposes of this article are the propositions the report contains about the characteristics that determine ‘use practice’ summarized below:

I suggest that these factors are conflating a distinction that is important for a use-based evaluation design. In the discourse offered in this article, Ferry and Olejniczak (2008) demonstrate the existence of practice clusters that can be further understood by distinguishing between ‘use’ and ‘usability’.

Usability practice

Prosaically, most research into evaluation usability refers to the form the evaluation outputs take and the extent to which they ‘speak’ to the intended user in an appropriate way (i.e. enable their use as knowledge resources for new practices). The contextual dimensions of this we have called the ‘use’ domain. The other key dimension refers to the design of the vehicle of the message to maximize engagement (a single, cold unresponsive text, a presentation, working in a workshop, working alongside a user to draw out practice based implications, etc.) but also the way in which the design of the evaluation lends itself to communicability and is formed in such a way that its potential as a knowledge resource is made more apparent.

In social practice discourse, this is sometimes referred to as a ‘boundary object’ or a ‘bridging tool’. In an important sense, usability is a reference to a learning process in which an ‘evaluation artefact’ (such as a report, web-based material or a social artefact like an event) is used as a knowledge resource for new practice. The term ‘bridging tool’ is guided by a specific learning theory. The idea that people might engage in a change on the basis of the use of an evaluation (moving from one activity system to another) has resonance with the notion identified by Engestrom and others (see Tuomi-Grohn and Engestrom, 2003 ) concerning the metaphor of ‘boundary crossing’. Conventionally associated with the experience of moving between different ‘activity systems’ and the learning processes, opportunities and, indeed, requirements, such crossing implies, activity theory has provided a fertile resource for depicting this process. Building on the work of Vygotsky (1999), Engestrom and his collaborators (1999, 2001, 2004) have drawn our attention to ways of thinking that emphasize how learning takes place in a social setting involving practices shaped by tools and resources (in this case knowledge resources provided through an evaluation), communities, divisions of labour and rules such that individuals and groups experience tension, creative problem solving and resolution that utilizes these elements toward an ‘object’ or ‘project’. Often, in evaluation environments, this may be improvement or development (Chelimsky, 1997).

Contandriopoulos and Brouselle (2012) focus on an approach that positions evaluative products or artefacts in the context of knowledge use and exchange. This conceptual reconfiguration is welcome, and is captured in this short extract:

We thus suggest a definition of collective-level knowledge use as the process by which users incorporate specific information into action proposals to influence others’ thought, practices and collective action rules. (Contandriopoulos and Brouselle, 2012: 63)

However, as suggested above, what strengthens this formulation is a robust conceptualization of what constitutes a ‘practice’. Action rules form the knowledge base on which potential users draw in new practices prompted by knowledge resources derived from an evaluation. In a recent case study of evaluation use in the context of Arts within higher education (Shreeve, 2011), a high usability approach to evaluation based on an interactive and user responsive design is outlined. Interestingly, the learning points Shreeve identifies have considerable resonance with the design issues prompted by adopting the practice-based approach. Although at the ‘high risk’ end of a strategy to maximize usability, her observations are telling:

We had to devolve power and control and let the respondents determine what they wanted to tell us and how they wanted to tell us. Taking a risk in this way resulted in new formats and unexpected outcomes of evaluation . . . evaluation does not always have to follow text based and verbal processes and dissemination or the ‘usability’ of evaluation is an important part of the outcome and this is dependent on the format the evaluation takes. We found acts of translation are required between different kinds of medium (words and objects) and also between different people with different previous experiences. Evaluation is also a context dependent and cultural activity. (Shreeve, 2011: 185)

While commissioners and users of an evaluation are not synonymous, they can be collapsed for the purposes of this commentary. It is self evident that once an evaluation enters the public domain, if it does, then anybody can be a potential user of the evaluation if they have access to it. This is one of evaluation’s great potentialities (see Patton, 2002) and suggests the urgency of establishing the levels of public access to evaluations very early on. Using the categories for evaluation design associated with RUFDATA (Saunders, 2000), it is possible to chart practices that are likely to enhance the usability of the evaluation and its outputs. The framework presented in Figure 3 is intended to rehearse the practices associated with evaluation at each stage of the framing and ‘pitch’ of an evaluation. The framework has been used in the context of the PALETTE project, which was a European project associated with professional learning in a variety of contexts (McCluskey et al., 2008).

OPEN IN VIEWER

Figure 3.

Usability practices in evaluation design.

The frames of reference of evaluators and those of evaluation commissioners are likely to be different from each other, and both differ from those emanating from practices on the ground as ‘users’ attempt to ‘enact’ policy or a programme idea. In particular, these differences mean that ‘engagement strategies’ with evaluation processes and outputs are best embedded in evaluation designs to a much greater extent than conventionally undertaken. These participative dimensions of evaluation design should be central, with the use of outputs at all stages ‘design savvy’.

So, as Patton (1997) has so persuasively suggested, evaluators need to engage with the commissioners and with users from the start of the evaluative process and at intervals throughout it. Simply delivering a report at the end is nowhere near enough, from a social practice point of view (we return to this issue below in considering the ways in which the voice of programme recipients can be heard). Building on previous work (Bamber et al., 2009), we know that engagement with evaluation output can be understood in the form of a continuum (from relatively low engagement and use to relatively high engagement and use). As presented in Figure 4, at the low engagement end of the continuum, we have dissemination or distributive practices (reliance on written texts alone in the form of articles, reports, summaries); in the middle of the continuum are what we could call presentational practices (seminars, presentations, conferences); and at the engaged end of the continuum we have interactional practices (working alongside users to identify situated implications).

OPEN IN VIEWER

Figure 4.

Use as engagement.

With these differences in thinking about supporting and incentivizing a thorough approach to evaluation use and usability, it is possible to undertake research of ‘use’ by using a typology of practice-based responses to an evaluation. This mechanism is particularly appropriate when an agency is involved in commissioning a series of evaluations or is overseeing a number of evaluations in devolved environments. The circumstances might involve, for example, a funding criterion that requires a development proposal or specification to have evaluative components built in or the requirement for an external evaluation to take place at various points in the intervention or programme life cycle. What is important in this formulation is that the evaluation output is assumed to have an effect on development or, put another way, is a resource for potential changes in practice. This may happen during an intervention or programme or in subsequent programmes. Either way, donors or funding agencies (like the EU) should have an expectation that evaluations paid for by the European tax payer, should have a use and the use is not rhetorical or tokenistic.

Figure 5 builds on a framework adapted from the work of Hall and Louks (1978) and used in the context of the national evaluation of a Higher Education Policy and programme intervention (see e.g. Lent and Machell, 2011) and in an evaluation of a governance programme in Chile (Saunders, 2006). It is conventionally used as a framework for depicting the responses of a target group to a change initiative or policy intervention. It is adapted here to focus on the stages through which potential users of an evaluation output might go as their use of the output becomes more sophisticated or better developed. This is a reference to the extent to which the knowledge resource implications of an evaluation are understood in increasing complexity. Note that the engagement with the implications of an evaluation report begins with simple awareness of an evaluation (2, 3) and proceeds through examining implications (3, 4), acting on them (5, 6) and finally, the creative adaptation of the implication of findings in onward action (7, 8).

OPEN IN VIEWER

Figure 5.

Stages of concern with use.

It is important to note that this approach is aspirational and explores a highly extended use of evaluation as a resource for development. In the cases reviewed in the UK-based evaluation of the CETL (Centres for Excellence in Teaching and Learning) (Saunders et al., 2009: 86), for example, use was confined to practices between stages 2 and 6. This tool might be discussed with programme managers as part of the preparatory stage of project management to alert them to the range of possibilities expected from the use of evaluation components. Reaching stages beyond 5 will require programme or project managers to have a well-developed understanding of the use and usability practices I have outlined above.

Overview

This article has presented an approach to evaluation use from social practice theory that highlights ‘use’ as a practice within clusters of evaluative practice determined by the domain context and policy area. The theoretical point here is that this approach foregrounds use as ‘behaviour’, situated in particular contexts and enables the identification of instruments that might encourage more effective use.

It has undertaken an analysis that builds on existing work by making a distinction between the use of an evaluation output and its usability. These two dimensions are based on a distinction between the organizational and political context of use on the one hand, with a particular concern with the capacity of stakeholders within an organization to make use of the knowledge resources made available through evaluation along with the systemic characteristics of the environment. On the other hand, usability refers to the dimensions of evaluation design, within the power of evaluators to affect, which are likely to inhibit or enhance the chances of evaluation output being used. The article included instruments and an epistemology that are intended to aid more effective use and usability in design and in a more active and thoughtful engagement strategy with evaluation outputs.

The argument for considering a social practice approach to the problems of use and usability is based on a concept of evaluation impact (the use of an evaluation to produce positive change) that identifies ‘changes in practice’ as central. According to this view, for evaluations to be used positively, they need to be understood as a knowledge resource for new practices (i.e. a change). The article has identified the organizational practices that are most likely to create ‘use’ and the design practices most likely to create usability. The bridging idea of ‘engagement’ itself involves practices. Engagement is used in this article to denote the practices that involve ways in which the characteristics of the evaluation design (usability) intersect with the characteristics of the organizational context of potential use. It is this interplay that is important.

Building on the work of Ojha et al. (2011), it is possible to provide an overview of these considerations influencing the extent to which use and usability are enhanced or inhibited. Figure 6 identifies the evaluation processes (usability factors) and user characteristics and the nature of the organizational context.

OPEN IN VIEWER

Figure 6.

Factors affecting evaluation use and usability.

Each of these factors embodies practices that can determine the extent to which evaluations are used. The suggestion implicit in this article is that awareness of the constellations of potential use practices will result in a more effective political role for evaluation.

The European context is one in which a regional policy is focused on using a series of instruments and mechanisms to achieve better integration of member states. The extent to which these mechanisms work effectively toward this aim is critical. Evaluation outputs are an important social capital resource for the attribution of value to these various mechanisms. It is a policy imperative, therefore, that their use becomes discerning, intelligent and better managed.