Book Review: Language assessment in practice: Developing language assessments and justifying their use in the real world

Some conceptual antecedents

In the interest of full disclosure, I should mention that my background has been mainly in educational assessment and in licensure and certification assessment, with limited experience in language assessment per se. My background necessarily limits my perspective to some extent, but divergent views can be informative.

As I read the book, I noted echoes from three distinct traditions in assessment: a focus on uses and consequences, an argument-based framework, and extensive reliance on the analyses of tasks and context to buttress the claims to be made about assessment uses. The book provides a new framework for evaluating language assessment programs, but a recognition of these conceptual antecedents can put the work in a historical context.

First, the authors make the uses of assessments and the consequences resulting from these uses the central organizing framework for the book and for their framework for assessment development and justification. The recognition that assessments should be evaluated in terms of how well they work in practice, and not just in terms of technical characteristics or the plausibility of proposed interpretations, goes back a long way, but Bachman and Palmer make it the centerpiece of their framework.

Assessments have generally been evaluated in terms of their reliability and validity, with reliability providing an indication of the consistency or stability of assessment scores, and validity addressing the meaning and utility of the scores. For the first half of the twentieth century, validity tended to emphasize the usefulness of the scores for particular purposes (e.g. predicting some criterion). In the first edition of Educational Measurement, Cureton (1951) defined validity in terms of test use:

The essential question of test validity is how well a test does the job it is employed to do. The same test may be used for several different purposes, and its validity may be high for one, moderate for another and low for a third. (p. 621)

Cronbach and Meehl’s (1955) watershed article on construct validity focused on interpretations. Messick (1981, 1988, 1989, 1994) and Cronbach (1971, 1988) redressed the balance; they gave a lot of attention to interpretations, but also emphasized the consequences resulting from various uses of assessment.

To the extent that validity is defined in terms of how well an assessment program achieves its goals, it necessarily includes some attention to consequences, positive and negative. The role of consequences in validity is controversial. Some authors (Borsboom, Mellenbergh, & van Heerden, 2004; Popham, 1997; Sackett, 1998) have argued for a focus on a semantic interpretation of scores, to the exclusion of most consequences. Others (Messick 1989, 1994; Linn, 1997; Shepard, 1997; Kane, 2006) have advocated for a conception of validity involving both the meaning of assessment scores and the consequences of their use.

There is no ambiguity about where Bachman and Palmer stand on this issue. On page 2, they make it clear that, ‘All of the various components of this book are tied in to a structure for justifying assessment use’. They go on to say that:

At the core of our approach is an Assessment Use Argument (AUA), which guides the assessment development process. The AUA consists of a set of claims that specify the conceptual links between a test taker’s performance on an assessment, an assessment record, which is the score or qualitative description we obtain from the assessment, an interpretation about the ability we want to assess, the decisions that are to be made, and the consequences of using the assessment and of the decisions that are made. (p. 30)

Bachman and Palmer subsume traditional notions of reliability and validity under the AUA, and maintain that: ‘Assessment justification consists of articulating an Assessment Use Argument (AUA) and collecting evidence to support this’ (p. 30).

They are decisive in opting for a framework that emphasizes the evaluation of assessment uses in terms of their consequences. Given past experience, it seems unlikely that everyone will hop on their bandwagon, but they provide a clear path to follow in this regard.

Second, in organizing their discussion around the role of AUAs, the authors adopt an argument-based framework for the development of the framework. Following Toulmin’s analysis of practical reasoning, the AUA is to ‘include the following elements: data, claims, warrants, backing, rebuttals, and rebuttal backing’ (p. 99).

Argument-based approaches to validation (Cronbach, 1971, 1988; House, 1980; Kane, 1992, 2006; Shepard, 1993) suggest that the claims being made in a score interpretation and use be stated clearly (e.g. in the form of an explicit interpretive argument) and that these claims be critically evaluated.

Bachman and Palmer follow in this tradition and extend it in several ways. They base their approach to language assessment development and use fundamentally on ‘the need for a clearly articulated and coherent Assessment Use Argument (AUA)’ and on ‘the provision of evidence to support the statements in the AUA’ (p. 31). They adopt the framework and terminology of an argument-based approach, but they do not emphasize the validity of a proposed interpretation per se. Rather, they are concerned with the general question of the justification for assessment uses, with the justification of proposed interpretations constituting one of several major claims in the AUA.

Argument-based approaches to validity have generally attended to both the interpretations and the uses of assessment scores (Kane, 2006; Shepard, 1993), and, as a result, have attended to the consequences of particular uses as well as interpretive frameworks, but Bachman and Palmer make score uses and the consequences of score uses the centerpiece of their discussion: ‘An AUA provides the conceptual framework for linking a claim about a particular set of consequences to the performance of individuals on a language assessment’ (p. 156). Given the high stakes of many emerging uses of assessment systems (e.g. in school and teacher accountability, in employment and immigration decisions), the analysis of consequences in justifying assessment programs is becoming increasingly important.

The third antecedent is the growing interest in performance, or authentic, assessment over the last 30 years; the authors devote a lot of attention to the fidelity of assessment tasks to the language use tasks that are of primary interest given the proposed use of the assessment. Their framework emphasizes the development of assessments that are authentic in the sense that they require the kinds of performance that are of interest.

Bachman and Palmer treat language use as a complex and highly interactive activity, involving the ‘negotiation of intended meanings between two or more individuals in a particular situation’ (p. 34). They define language ability as a ‘capacity that enables language users to create and interpret discourse’ (p. 33) and suggest that it includes two components: language knowledge and strategic competence, but they also recognize a need to consider, ‘personal attributes, topical knowledge, affective schemata, and cognitive strategies’ (p. 33).

Bachman and Palmer define language use tasks as elemental activities of language use, and they define target language use (TLU) tasks as language use tasks in some specific setting outside of the assessment itself. They suggest that:

Language use can be viewed as performance of a series of language use tasks that are interrelated, in terms of the setting, the communicative goal to be achieved, and the participants. A language assessment task can be thought of as a procedure for eliciting responses from which inferences can be made about an individual’s language ability. It therefore follows that in order for such inferences to be made, a language assessment should consist of language use tasks. In designing language assessments whose use we can justify, it is important to include tasks whose characteristics correspond to those of TLU tasks. (p. 63)

To the extent possible, it seems that Bachman and Palmer would design the assessment as a sample of performances from the TLU domain.

This approach has some clear advantages, which Bachman and Palmer discuss at some length, but it can also entail some significant disadvantages (Widdowson, 2001). Rich performance tasks tend to require a significant period of time to complete, and therefore the sampling of tasks from the TLU domain is likely to be small and potentially not representative of the domain as a whole. The art of assessment development is to balance the need for adequate sampling of the domain and consistency in scores across replications of the assessment with the need for tasks that are as authentic as possible.

Structure and content of the book

This book could serve at least three general functions. It is basically a monograph in which a framework for the development and justification of language assessments is developed in some detail. The first five chapters lay out the framework and criteria for justifying assessment programs. Chapters 6 to 12 provide a description of how to develop language assessments and associated AUAs. The third part of the book, Chapters 13 to 21, discusses the development and use of language assessments in real-world contexts. As discussed above, the framework has roots in educational assessment and in language assessment, and it provides a rich framework for assessment development and justification with a strong focus on language assessment use and on consequences.

The monograph can serve as a guide to assessment development. It provides a number of examples of language use that are carried through the book, and it discusses the claims made in language assessments in considerable detail. It does not provide a cookbook approach for assessment development, but it does include lists of claims, warrants, and rebuttals that are likely to come up or be encountered in developing language assessments, and provides detailed advice on how to deal with the many issues that are likely to arise in assessment development. In some cases, the detail, which would be useful for assessment development, slows the general development down a bit, but a reader always gets to pick and choose how they will read a book; in a section of the first chapter, the authors say of the book that they ‘cannot imagine anyone sitting down and reading it cover to cover’ (p. 15) and then give some advice on how to get the most out of their work.

Third, the book could provide the basis for an advanced course in language assessment. In addition to the development and discussion of the framework, the book includes extended examples and a set of project materials on a website. The book is set up to facilitate its use in a language assessment course.

Communication across disciplines and sub-disciplines

There is a natural tendency for distinct but related research areas to diverge over time, developing differences in terminology and methodology as a result of differences in contexts and goals. Most of this differentiation is necessary and desirable, as new models and methods are developed to meet specific needs in different contexts, but some of the differences may cause unnecessary confusion. The differences are particularly problematic if the same term is used to describe different concepts or methods, or different terms are used to label a common concept or method.

There was one set of terms on which I would have preferred that Bachman and Palmer had made different choices. They define generalizability as ‘the degree of correspondence between a given language assessment task and a TLU task in their task characteristics’ (p. 117), and they define consistency as the extent to which, ‘the assessment records are consistent across different characteristics of the assessment situation, such as different assessment tasks, different forms of the assessment, different assessors, or different times of assessment’ (p. 126).

However, there is a well-established theoretical framework, developed by Lee Cronbach and his associates, for the kinds of precision/reliability/generalizability issues considered under the heading of consistency in the AUA framework. This framework, Generalizability Theory (Brennan, 2001), goes back over 40 years, and I think that the difference between Bachman and Palmer’s usage of the term ‘generalizability’ and the usage of this term in Generalizability Theory invites confusion. Bachman and Palmer have proposed a new framework, which necessarily requires new terminology, but I suggest that if they revise the book or write a sequel, they consider changing their usage of the term ‘generalizability’.

A second labeling issue, which could also generate some ambiguity, involves the term ‘warrant’. Bachman and Palmer (2010) define warrants as, ‘statements that elaborate the qualities of a claim’ (p. 161), and they have many warrants associated with some claims. As described by Kane (1992, 2006) and Mislevy, Steinberg, and Almond (2003), Toulmin (1958) treats warrants as rules of inference that justify the adoption of a claim, based on data, or as ‘general, hypothetical statements, which can act as bridges, and authorize the sort of step to which our particular argument commits us’ (p. 98). Toulmin’s usage may be less widespread in educational measurement than the usage of ‘generalizability’, but the reader of various argument-based analyses should be aware of the variability.

Concluding remarks

I expect that a wide range of individuals who are involved in language assessment, and more generally, in educational assessment, will find this book thought provoking and helpful in the practice of assessment development and justification. It is an important contribution to the development of argument-based approaches to assessment development and justification.