Assessing vocabulary learning in early childhood

General vocabulary measures

Normed general vocabulary measures: Measures of breadth. Normed measures of general vocabulary (receptive and expressive) focus on measuring the breadth of individuals’ vocabulary (Table 1). PPVT (Dunn and Dunn, 2007) is the most common measure of receptive vocabulary used in early literacy research today. It is an individually administered assessment in which the examiner shows the child a page with four separate illustrations and asks him/her to point to the one that shows the target word. Almost all studies measuring the effects of reading aloud on general vocabulary use the PPVT as a measure of general vocabulary (e.g. the 19 studies reviewed by Dickinson and Smith, 1994; Hargrave and Sénéchal, 2000; NELP, 2008; Reese and Cox, 1999; Schwanenflugel et al., 2005; Walsh and Blewitt, 2006). We have expressed concerns about the PPVT (Hoffman, 2008), as have others (e.g. Pearson et al., 2007), but overall we find it adequate as a general measure of receptive vocabulary. Related normed measures also assess expressive vocabulary, two common examples being the Expressive One Word Picture Vocabulary Test (EOWPVT; Brownell, 2000) and the Expressive Vocabulary Test (EVT-2; Williams, 2007). In these expressive measures, rather than pointing to a picture portraying a target word, the administrator prompts the child to name the word portrayed in a picture.

Although the PPVT, EOWPVT and EVT are the most commonly used normed measures of general vocabulary in literacy research, many other examples exist. The Woodcock-Johnson III Picture Vocabulary Subtest (Woodcock et al., 2001) is widely used in clinical practice, although less frequently in literacy research. This assessment includes a receptive measure (similar to the PPVT) and an expressive measure (similar to the EOWPVT and EVT). Other more holistic assessments of oral language development appropriate for preschoolers typically include a vocabulary component. For example, the Systematic Analysis of Language Transcripts (SALT, Miller, 2012) measures the number of different root words in a spoken language sample, and the Preschool Language Scale (PLS-4, Zimmerman et al., 2002) includes receptive and expressive vocabulary components. Because assessments like SALT, PLS and other similar assessments (e.g. Clinical Evaluation of Language Fundamentals (CELF); Semel et al., 2003) are more often used by speech-language pathologists for decision-making in clinical practice and have not been commonly used in studies of preschool vocabulary learning from classroom read-alouds, our remaining discussion of general vocabulary assessments focuses on those most relevant to our specific area of analysis, namely the PPVT, EOWPVT and EVT.

Measures like PPVT, EOWPVT and EVT are useful instruments because they are normed and standardized, statistically determined to be valid and reliable over thousands of administrations, easy to train for administration and simple to score. These measures virtually guarantee a normal distribution of scores conducive to inferential statistical analyses and can support the conclusion that gains on the test represent general vocabulary growth by comparing the individual to the general population curve. Thus, PPVT, EOWPVT and EVT are useful tools for indicating the impact of preschool instruction on the growth of children’s general vocabulary over the course of their year(s) in school and for estimating the impact of vocabulary on later reading achievement (Dickinson and Porche, 2011; National Institute of Child Health and Human Development Early Child Care Research Network (NICHD ECCRN), 2005; Neuman et al., 2011; Storch and Whitehurst, 2002). The value of standardized measures like PPVT lies in their well established validity and reliability, which enable rigorous quantitative comparisons for programme evaluation and research.

In terms of measuring the breadth of vocabulary learning specifically attributed to instruction, gains on general vocabulary measures like the PPVT are typically small (Marulis and Neuman, 2010; National Early Literacy Panel, 2008). Research using other measures of vocabulary learning as well as anecdotal evidence suggest that reading aloud positively impacts on children’s vocabulary, and so it is unreasonable to conclude that the small effect sizes measured by assessments like PPVT indicate that the children in those studies did not learn new vocabulary from instruction (Marulis and Neuman, 2010). Rather, we suspect that what students did learn about new words was not well captured by the normed general vocabulary measures being used in such research. Other sources have raised similar concerns about the relative insensitivity of general vocabulary measures to detect small amounts of growth, in particular the National Reading Panel (NICHD, 2000), which suggested researchers should use both researcher-created and standardized measures to best examine vocabulary gains.

Definitional measures of general vocabulary: Measures of depth. A very different approach to vocabulary assessment involves directly prompting children for their definitions of words. Definitional measures provide information about depth of word knowledge in two ways: (1) they require the child to use their own language to describe word meanings, which provides rich qualitative data for insights into how the child understands a particular word in relation to other words and (2) they are typically scored on a scale (as opposed to correct/incorrect), which allows measurement of levels of word knowledge (Table 1). However, data about the depth of word knowledge inevitably lead to challenges with scoring the responses of young children. As opposed to the straightforward right/wrong nature of picture-based measures, with definitional measures one is forced to confront which features of a definition constitute “knowing” a word?

Different researchers have used different approaches to scoring definitional responses. Snow et al. (1989) assessed the ability of older children (grades 2 to 5) to give formal definitions of words, which had to contain an equivalency statement (e.g. a “dog” is… or “dog” means…) and superordinate (a broader category to which the word belongs, e.g. an animal). However, this may not be a realistic expectation for younger children, even though young children can communicate much about what they do know about a particular word. Alternatively, Biemiller and Slonim’s (2001) study of vocabulary of children in preschool through to grade six scored children’s informal definitions using a three-point scoring system: “known” (1), “possibly known” (0.5) or “not known” (0). Using this definitional scoring system, Biemiller and Slonim estimated the size of children’s general vocabulary based on the percentage of children in the entire sample who knew each word at each grade level (their measure of word difficulty). Because definitional measures rely on indications of the level of word knowledge, they are also useful in assessing the depth of children’s word knowledge in ways that are not possible with traditional receptive and expressive measures. However, definitional measures of general vocabulary like these are much less commonly used in research, especially research on the effects of instruction, than PPVT, EOWPVT and EVT, most likely due to the complexity of scoring.

General vocabulary measures like those we have reviewed can provide valuable insights into the breadth and depth of children’s word knowledge. However, what is not so useful about general vocabulary measures for work with teachers and children in preschool classrooms is their inability to provide insights into children’s knowledge of specific words. This is problematic because a range of preschool activities, and especially read-alouds, target specific words that children are meant to learn, and in order to drive data-based instructional decisions, teachers need information about what words children are learning and what they are learning about them.

Specific word vocabulary measures

A number of studies have attempted to measure children’s learning of specific words taught (e.g. Coyne et al., 2009; Hargrave and Sénéchal, 2000; Justice, 2002; Neuman et al., 2011; Sénéchal and Cornell, 1993; Silverman, 2007; Walsh and Blewitt, 2006; Wasik and Bond, 2001; Whitehurst et al., 1994), the clear advantages being that the researcher can target specific words related to concepts in the curriculum and/or in books to be read aloud, and simply pre- and post-test the words to measure vocabulary growth. In practice, teachers benefit from data like these because they lead to better understanding of what children are learning from instruction, in terms of both breadth and depth of word knowledge.

PPVT-like formats: Measures of breadth. The most commonly developed formats for measuring the breadth of specific word vocabulary reported in the literature are researcher-developed receptive vocabulary measures that mimic the PPVT. The researcher shows four pictures (1 correct and 3 distractors) for each of the target words and prompts the child, “Point to the picture that shows [target word].” Another related type of pictorial assessment is a researcher-developed measure of expressive vocabulary (structured like EOWPVT or EVT). In these expressive measures, the researcher shows a picture of the target word and asks the child to name the picture. Measures like these have been used extensively with preschool students to measure the breadth of word learning from instruction (e.g. Ard and Beverly, 2004; Collins, 2005; Hargrave and Sénéchal, 2000; Justice, 2002; Walsh and Blewitt, 2006; Wasik and Bond, 2001; Whitehurst et al., 1994; Table 1).

Both of these types of tests appear straightforward, but in actuality are problematic in terms of design and use as measures of specific word vocabulary learning with preschoolers. First of all, there is the issue of the three distractors included for each item in the receptive instruments. The PPVT has been through numerous rounds of item analysis and refinement to present items systematically in a progression of increasing difficulty to fit scored responses to a normal curve, thus enabling the instrument’s purpose of measuring general vocabulary knowledge and growth compared to the normed sample. Researcher-developed tests, on the other hand, have not monitored or controlled for the difficulty of items in this or any closely related way. To illustrate, in the PPVT, the distractors become more similar to the target as the items increase in difficulty in order to capture the increasingly fine-grained meanings of the target words. However, in researcher-developed tests that are not designed to fit a normal curve, how does the researcher determine the appropriate level of difficulty of each item, or more accurately, the appropriate level of disparity between the target and the distractors? For example, one of the items we wished to assess was lightning. What should the distractors look like for this word? Would illustrations of a lamp, a kitten and a dish serve the purposes of distractors? Or should the illustrations represent other weather forms, like rain, snow and wind, or conform to some other pattern? We searched through numerous studies of early childhood vocabulary learning resulting from the instructional practice of reading aloud and could find no discussion of this issue. The absence of such discussion indicated to us that early literacy researchers typically did not address this potential measurement problem at all, and simply chose distractor words and illustrations that were categorically similar to the target.

A second item construction issue to consider is the actual visual portrayal of the target words. It is interesting to note that the PPVT items administered to three- and four-year olds typically consist of mostly nouns (72% of the first 72 items in PPVT-4). There are relatively few verbs (21%), and almost no adjectives (only 5%) that are tested. Having to represent vocabulary words pictorially creates a bias toward including nouns and simple verbs and adjectives because they are the easiest to portray. This does not pose much of a problem for standardized measures of general vocabulary because it is relatively easy to find items that work well from among the thousands of words in a language, but it creates significant problems when one is seeking to assess specific words from a book or related to a concept or unit that has been taught. To get the greatest payoff in terms of instruction, teachers are encouraged to teach “Tier 2” words (i.e. words that are not frequently heard in everyday conversation but are typical of written text (Beck et al., 2002)) and words that are central to the meaning of the texts being read. Very often such words are not easily represented in picture items (Beck and McKeown, 2007). For example, one word that we wished to assess was nudge. Efforts to portray the action nudge in a still format resulted in illustrations in which animals appeared to be sniffing more than nudging. Therefore, even though it was a rich, Tier 2 vocabulary word worthy of emphasis in the story, nudge posed problems for assessment using this format.

The third item construction issue relates to children’s scores on these types of measures. In using a receptive measure with four choices, the child inevitably has a one in four chance of getting the item correct, regardless of her/his knowledge of the word meaning. For a normed measure like PPVT, this issue is largely irrelevant: standardized scores account for probability, and since each child’s score is based on dozens of responses, a child’s results are not intended to be used as indicators of knowledge of particular words. However, when assessing only a limited number of words from relatively few instructional activities, achieving 25% correct by chance can greatly diminish the practical significance of the scores, rendering teacher understanding of an individual child’s knowledge of specific words less than robust.

Fourth, there is a significant problem with interpretations of results from such an assessment format when it is used to examine preschoolers’ vocabulary growth as a result of read-aloud experience. As previously stated, studies of the effects of vocabulary instruction in read-alouds using specific word vocabulary measures not surprisingly demonstrate larger effect sizes than those reported for general vocabulary measures (Marulis and Neuman, 2010; NICHD, 2000). Thus, data from specific word measures support the use of reading aloud to promote vocabulary learning of specific words, but they cannot be used to conclude that reading aloud promotes practical, significant general vocabulary growth, as they are sometimes interpreted. This distinction can have far-reaching effects on policy and practice. As one example, the WWC, which serves as “a central and trusted source of scientific evidence for what works in education” (2012a), reported strong evidence of the positive effects for Dialogic Reading as a read-aloud intervention supporting general oral language development despite their own report of a lack of statistically significant findings for the majority of measures used across all of the reviewed studies (2007). Four studies were included in the WWC intervention ratings (met WWC evidence standards; Lonigan et al., 1999; Lonigan and Whitehurst, 1998; Wasik and Bond, 2001; Whitehurst et al., 1994). Of these four studies, only one reported a significant difference on a normed receptive vocabulary measure (Wasik and Bond, 2001, as reported in the original publication—these data were missing from the WWC analysis in the intervention report), and only one reported a significant difference on a normed expressive vocabulary measure (Whitehurst et al., 1994). Effect sizes calculated from data from normed measures reported in these studies averaged only 0.02 for receptive and 0.13 for expressive vocabulary. Effect sizes on researcher-developed measures (Wasik and Bond, 2001; Whitehurst et al., 1994) were much higher, averaging 1.58 for receptive and 1.13 for expressive vocabulary. Through careful analyses of the included studies, it is clear that this recommendation of the WWC is attributed to the strong effect sizes of one study using researcher-developed specific word vocabulary measures (Wasik and Bond, 2001). However, the WWC made no distinction between the types of measures used across different studies, and, in fact, in a related online resource developed for teachers by the U.S. Department of Education, Doing What Works (2008), clearly promoted Dialogic Reading as a research-based practice for enhancing oral language for all children. Our goal here is not to dispute studies of Dialogic Reading that have found positive effects for some children, but rather to highlight how the use of evidence from specific word measures can be and has been inappropriately applied in recommending instructional practices for general vocabulary growth, thus promoting misapplications of research in practice.

Yes/no questioning format. An alternative to representing words pictorially is the use of yes/no questions about the meanings of target words (as used in Beck and McKeown, 2007 and Coyne et al., 2009 with kindergarten and older children). In this form of assessment, each target word has one question for which the correct response is yes and another with the correct response no. To demonstrate knowledge of the target word, both questions must be answered correctly. For example, for the target word trail, the child should respond yes to, “Could you walk along a trail?” and no to, “Could you send a trail to your friend?” in order to be considered as knowing trail.

Because these forms of assessment do not require pictorial representation, it is probably easier to assess a wider variety of target words. In addition, variations of this design can indicate children’s depth of word knowledge (Table 1). For example, Beck and McKeown (2007) included questions focusing on basic definitional meaning as well as those using the word within a specific context, and Coyne et al. (2009) designed yes/no questions that required full versus partial knowledge of the word in order to respond correctly.

Nevertheless, we believe there are challenges to yes/no assessments that should be carefully considered before using them with preschool children. When we assessed preschool children with this approach, we found that they were easily confused by the structure of yes/no questions. Our assessment experience was similar to teaching experiences asking yes/no questions in large group discussions with young children, where half of the class earnestly answers “yes” while the other half just as earnestly answers “no”. This behavior suggests children are responding to what they think the teacher has in mind, rather than comprehending and responding to the meaning of the actual question. In other words, this assessment form confounds comprehension of yes/no questioning formats with word knowledge in a way that is problematic for interpreting an individual child’s knowledge of a specific word. In addition, the 50% probability that each question will be correct by chance corresponds to a 25% probability that any two questions for a target word will be correct by chance.

Definitional measures of specific words: Measures of depth. Other studies have drawn on research using definitional measures of depth of general vocabulary to design assessments that elicit definitions of specific words taught (Table 1). As explained in our discussion of general vocabulary assessments, definitional measures inevitably involve the challenge of scoring a child’s explanation of word meaning. As with general vocabulary definitional measures, different researchers have used different approaches to scoring children’s definitions of specific words taught. Blewitt et al. (2009) analyzed children’s definitions for the number of information units specified: superordinate category (e.g. “A cat is a kind of animal”), synonyms (e.g. “A ship is a boat”), perceptual properties (e.g. “A carrot is orange”), functional properties (action, “A ship floats”, or use, “You eat carrots”) or parts (e.g. “A tree has a branches”). Children earned one point for each information unit for each word and the points were totalled across all words tested. Coyne et al. (2009) used a similar three-point system to that employed by Biemiller and colleagues (Biemiller and Boote, 2006; Biemiller and Slonim, 2001), which simply indicated whether the word was unknown, partially known or completely known.

Neither of these scoring approaches can overcome the complexity of scoring levels of word knowledge in definitions from children of this age. To demonstrate some of the complexities involved, consider the following difficulties we found. After multiple encounters with/discussions of the word reflection during read-alouds (a word from the book Raccoon on His Own (Arnosky, 2003) in which the raccoon sees his reflection in water) as well as in hands-on guided explorations of reflective surfaces, we prompted children to define the word reflection. The following is a sample of five children’s responses:

“It means that your reflection is yourself. It means that there is another person that looks just like you.”

The characteristics of these five responses are not isolated examples. They are representative of common challenges we experienced in using a definitional measurement approach to assess knowledge of specific words. First, when assessing vocabulary with children this young, measurement of word knowledge through definitions is inevitably confounded with the ability to convey that knowledge in language. The concept of reflection is a complex one to explain, even for an adult (try to think of how you would formally define this word), but especially so for a four-year-old. In response 1, for example, it is impossible to determine whether the child actually understood a reflection to be another person, or just struggled to convey the idea in words.

Second, because the context in which these responses were obtained focused on assessing word knowledge from a specific book, the issue of the generalization of word meanings is highlighted. Formal definitions are intended to convey the most general meaning of the word while still distinguishing it from similar words. However, when preschool children have encountered a word in one context, they cannot be expected to generalize its meaning to others; that is to say, they do not yet know the boundaries of that word’s meaning vis-à-vis other words with a family resemblance (Wittgenstein, 1953). For example, both responses 2 and 3 evidence some knowledge that reflections can occur in something more general than water (“in stuff” or “in something”), but both also seem to indicate the limited understanding that a reflection is only an image of oneself, with no acknowledgement of the possibility of seeing a reflection of another object or person.

Third, and related to the issue of the generalization of meanings, is the inclusion of book-specific details in definitions, such as in response 3, “like water”. Children had a tendency to include such details in their definitions, which in hindsight is not at all surprising, but which definitely complicates scoring because formal definitions are not “supposed” to include such details. Yet, and especially for children of these ages, their presence is so naturally characteristic of answering a question about a word in a book that was read to you that dismissing them in analysis would be tantamount to saying that the context of the assessment is irrelevant.

In addition, problems with the definitional measurement of word learning arise from the opposite perspective: when the child offers a definition not in line with the word use in the book. A prime example arose with the word chill, as used in the book that was read to the children: “A chill ran down his spine.” During the assessment, many children proffered a more colloquial definition of the word. A favorite of ours was: “When you want to lay down and watch TV—and eat nachos.” Thus, when the measure and the task are designed to value a particular response, how does one evaluate a different, but equally appropriate response? In this “chill” example, the child does not provide any evidence of having learned the word meaning as used in the book, but was demonstrating a level of knowledge of the word. More importantly, the word usage described by the child is arguably relevant for his interpretation of the usage of the word in the text, in that, in one sense, they are both metaphors for the connection between emotional state and body temperature (Lakoff and Johnson, 1980).

With respect to the specific scoring approaches for definitions, we attempted both analyzing information units and the three-point scoring system. When analyzing information units, any one unit can be problematic to analyze. For example, for the target word swamp, does the phrase “animals live there” evidence an understanding that a swamp is a habitat and therefore represents an emergent use of a superordinate? Or does it constitute a use of a swamp? For the word downstream, one child responded, “It means you’re going lots of water [sic] that’s going really slow.” How should the inaccurate additional information, “that’s going really slow”, be analyzed? Does this count against the score in some way, or does one ignore it? In analyzing the definitions our students provided, we struggled to achieve a reasonable degree of inter-rater reliability and were unable to designate any specific combination of these formal definitional traits (i.e. superordinates, synonyms, properties) that consistently captured the holistic quality of communicating the basic premise of the word meaning. In other words, a child could convey an understanding of a word only using a list of traits (raccoon—“he got a black mask, white eyes, brown hands, brown fur, brown legs”) or one superordinate and one trait (raccoon—“it’s a kind of animal that’s sneaky”). If scored based on the inclusion of information units using Blewitt and colleagues’ system (2009), the first response would receive a 5 and the second response a 2; however, we do not believe that first response indicates a holistically deeper understanding than the second, and thus we question the validity of the measure as an indication of the depth of word knowledge.

The only approach to scoring with which we were able to achieve a reasonable degree of inter-rater reliability was on a holistic three-point scale like that used by Coyne et al. (2009) and Biemiller and colleagues (Biemiller and Boote, 2006; Biemiller and Slonim, 2001). Quantitatively, a three-point scale only provides one additional level over the correct/incorrect measures previously described. Qualitatively, however, children’s definitions provide a wealth of information about how individual children are thinking about words.

Aside from the complexity of scoring, definitional measures offer much information about children’s word knowledge. Because they are completely open-ended, they do not rely on indirect measurement of word knowledge through some instantiation of multiple-choice, nor are they confounded with the issue of pictorial representation. In addition, instead of being restricted to analyzing children’s responses as only right or wrong, definitional measures can capture more about how individual children understand and use words. This is useful information for teachers and researchers alike, because the resulting data emphasize the complexity of word learning, as became clear through the discussion of the intricacies of children’s knowledge of words in our examples. In other words, children’s responses to definitional measures (like those reported in this section) highlight an understanding of words as interrelated and deictic components of language rather than discrete, isolated objects with stable meanings.

Normed general vocabulary measures

For teachers and researchers, normed general vocabulary measures like the PPVT are a useful tool for screening groups of students for the breadth of their general vocabulary development. For policy-makers or researchers such normed measures are useful for programme evaluation and experimental comparison based on the breadth of children’s general vocabulary. But precisely because they are normed general vocabulary measures, they do not provide practically significant information to teachers about children’s learning of specific words from instruction, nor do they capture well the depth of individual children’s knowledge of specific words. In conclusion, measures like the PPVT have a place and a role in assessing the breadth of children’s general vocabulary development, but interpretations of data from such measures should not reach beyond these limitations.

Specific word measures

First of all, we have not been able to locate any published evidence to date on the use of yes/no assessments of specific words with preschoolers. Thus, we recommend that they not be used until further evidence suggests they are appropriate and meaningful assessments for preschool children. Second, we propose that researcher- or practitioner-designed PPVT-like assessments of specific words are not useful for school administrators or practitioners to use in planning instruction because the drawbacks in terms of validity and reliability (e.g. ease of pictorial representation, opportunities for an item to be correct by chance) far outweigh the benefits (namely, ease of administration and scoring). Researcher-developed PPVT-like assessments may be useful tools for measuring word learning for groups of preschool students in experimental studies (when an individual child’s knowledge of a specific word is less relevant), but only if they are crafted in careful, systematic ways with reported measures of reliability and validity. In addition, research using these measures should specify much more clearly how items were developed and should remind readers that effect sizes reported using such measures are not comparable to effect sizes from normal general vocabulary measures.

Definitional measures

We believe definitional measures of word knowledge are a useful but underutilized assessment tool in early childhood vocabulary assessment. In particular, definitional measures have the potential to provide insights into individual preschool children’s depth of word knowledge of specific words taught. For teachers, it is important to note that definitional measures need not be highly structured one-on-one assessments. Teachers can glean much information from children during normal classroom activities through informal prompts for word meanings, while simply jotting down notes to record children’s definitions and/or scoring their responses on the three-point scale previously discussed. Thus, this assessment form is conducive to on-the-fly informal assessment opportunities that are developmentally appropriate for preschool classroom instruction, but which also provide the depth of information about individual children’s knowledge of specific words that a teacher needs to plan future instruction. Nevertheless, it is important for teachers to recognize that definitional measures inevitably confound children’s word knowledge with their ability to convey their knowledge through language.

It must also be noted that there is limited research on the use of definitional measures with young children and their relationship to planning vocabulary instruction. More research into the application of definitional vocabulary measurement in classroom practice is needed, and teachers will likely require professional development in how to analyze reliably and interpret appropriately the qualitative data that definitional measures provide.

In order to build knowledge of the reliability, validity and utility of definitional measures in classroom practice, we recommend that such measures be incorporated more often into research studies of word learning. Definitional measures offer one way to measure gradients of word knowledge of the rich words that teachers actually teach without the confounds of difficulties associated with providing pictorial representation, and so we also recommend putting more research focus on deeper understandings of word learning than simple recognition.