Abstract
Most research on early identification of reading disabilities has focused on word reading problems and little attention has been given to reading comprehension difficulties. In this study, we investigated whether measures of language ability and/or response to language intervention in kindergarten uniquely predicted reading comprehension difficulties in third grade. A total of 366 children were administered a battery of screening measures at the beginning of kindergarten and progress monitoring probes across the school year. A subset of children also received a 26-week Tier 2 language intervention. Participants’ achievement in word reading was assessed at the end of second grade, and their performance in reading comprehension was measured as the end of third grade. Results showed that measures of language ability in kindergarten significantly added to the prediction of reading comprehension difficulties over and above kindergarten word reading predictors and direct measures of word reading in second grade. Response to language intervention also proved to be a unique predictor of reading comprehension outcomes. Findings are discussed in terms of their relevance for the early identification of reading disabilities.
Considerable attention has been devoted to the development of methods for the early detection of children at risk for reading disabilities (RD). Research has shown that targeted assessments in kindergarten or first grade can often foretell reading problems in later grades (Badian, McAnulty, Duffy, & Als, 1990; Catts, Fey, Zhang, & Tomblin, 2001; Elbro, Borstrom, & Petersen, 1998; Juel, 1988; O’Connor & Jenkins, 1999; Schatschneider, Fletcher, Francis, Carlson, & Foorman, 2004). Recently, the response to intervention (RTI) framework has been introduced in hopes of further improving early identification (Fuchs & Fuchs, 2006; Haager, Klinger, & Vaughn, 2007). According to this framework, children can be identified as having RD if their response to scientifically based instruction, including targeted intervention, is substantially below that of their peers. RTI is assessed by universal screening and/or progress monitoring measures. All children participate in periodic universal screening to identify children who are potentially at risk for RD. Those who “fail” universal screening receive supplemental instruction, and their response is assessed by progress monitoring measures to further gauge risk for RD. Children who continue to show poor response may be provided with more intensive intervention and, in some settings, be considered for special education placement.
Research has begun to examine the use of universal screening and progress monitoring within an RTI approach for the early identification of RD (Al Otaiba et al., 2011; Catts, Nielsen, Bridges, Liu, & Bontempo, in press; Compton, Fuchs, Fuchs, & Bryant, 2006; Vellutino, Scanlon, Zhang, & Schatschneider, 2008). This work has demonstrated that universal screening as early as the beginning of kindergarten can accurately identify children at risk for RD. For example, Catts et al. (in press) found that measures of letter naming, phonological awareness, rapid object naming, and nonword repetition at the beginning of kindergarten were very good predictors of reading outcome at the end of first grade. Vellutino and colleagues further have shown that measuring at-risk children’s response to supplemental intervention during kindergarten can improve accuracy of identification beyond that of early screening (Vellutino et al., 2008). Al Otaiba et al. (2011) and Catts et al. (in press) also found that progress monitoring of preliteracy skills in the general kindergarten population added to the prediction of RD over and above universal screening at the beginning of the school year.
Whereas these initial results are promising, most of the attention has been focused on identifying risk for problems in word reading. The ability to read words is important for academic achievement, but the ability to comprehend words (and sentences) is especially critical. Very few studies have directly considered the early identification of problems in reading comprehension. Nevertheless, studies focusing on the prediction of word reading skills should also be relevant for the identification of children at risk for difficulties in reading comprehension. The simple view of reading states that reading comprehension is a product of word recognition and language comprehension (Gough & Tunmer, 1986). Thus, the measurement of variables shown to be related to word reading should help in the early identification of reading comprehension difficulties. The simple view also suggests that measures of language ability can add to the detection of problems in reading comprehension. Studies have shown that oral language ability in the preschool years is predictive of subsequent reading comprehension (Kendeou, van den Broek, White, & Lynch, 2009; National Institute for Literacy, 2008; NICHD Early Child Care Research Network, 2005; Storch & Whitehurst, 2002). In addition, Olson et al. (2011) found that preschool vocabulary accounted for individual differences in subsequent reading comprehension beyond that explained by word reading. Few studies, however, have directly examined the additive predictive effects of language measures within universal screening batteries. Therefore, in the present study, we investigated if measures of language ability would add to the prediction of problems in reading comprehension over and above early predictors of word reading.
From an RTI perspective, it might also be proposed that growth in language ability in response to language intervention and/or classroom instruction could provide additional information about risk for reading comprehension difficulties. However, growth in language might be different than that involving preliteracy skills related to word reading. The latter skills are often more dependent on direct instruction than is language ability. For example, the ability to name letters is a good indicator of subsequent word reading skills (Catts et al., 2001; Scarborough, 1998; Schatschneider et al., 2004; Vellutino et al., 2008). Most children enter kindergarten with some knowledge of letter names but make considerable gains in this knowledge as a result of direct instruction during kindergarten. Thus, screening at the beginning of the school year can provide some information relevant to the prediction of word reading skills, but measurement of variable growth in response to instruction can add significantly to this prediction (Catts et al., in press; Vellutino et al., 2008). This may not be the case for oral language. Language acquisition takes place throughout the preschool years, and children’s relative language status can change during this period (Ellis Weismer, Murray-Branch, & Miller, 1994; Rescorla, Dahlsgaard, & Roberts, 2000). By kindergarten, however, individual differences in language ability become more established and appear to be relatively stable thereafter (Johnson et al., 1999; Tomblin, Nippold, Fey, & Zhang, 2014; Tomblin, Zhang, Buckwalter, & O’Brien, 2003). Children continue to make gains in language ability during the school years, and some of these gains may be in response to direct instruction; but differential growth is not generally observed (Storch & Whitehurst, 2002; Tomblin et al., 2014). Thus, the potential for language growth to provide information over and above preintervention language ability would seem to be low.
Despite the above argument, it still may be possible that the measurement of response to language intervention could add to the prediction of subsequent reading comprehension. Tomblin et al. (2014), who reported considerable stability in language competence across the school years, found that the correlation of a composite language score between kindergarten and second grade was .8. Whereas this value is moderately high, it still leaves variance in language growth unexplained. Thus, it is possible that a measure of RTI could capture additional variability in language and add to the prediction of reading comprehension. In fact, RTI may be well suited for this purpose. RTI is by nature a dynamic assessment of behavior and, as such, may provide a more error-free estimate of ability (Grigorenko, 2009). It may also be less culturally and linguistically biased than a static measure (Tzuriel, 2000). Dynamic assessment has had a long history of use as a method for better assessing children’s potential for academic achievement (see Grigorenko & Sternberg, 1998) and, more recently, as a less biased measure of oral language ability (Camilleri & Law, 2007; Peña, Resendiz, & Gillam, 2007). Such a measure may, thus, be more sensitive than static assessments to individual differences in language learning abilities that are related to reading comprehension. In addition, the dynamic nature of RTI may make it a more reliable measure of language ability. Therefore, in this study, we examined if RTI involving vocabulary and narration provided unique information in the prediction of subsequent reading comprehension.
In the present study, we followed a large group of children from the beginning of kindergarten through the end of third grade. Two primary aims were addressed. First, we sought to determine if measures of language ability in kindergarten added to the prediction of third grade reading comprehension difficulties over and above kindergarten measures that have been shown to be related to word reading. We also had direct measures of word reading in second grade. This allowed us to also examine if kindergarten measures of language ability predicted reading comprehension difficulties independently of direct measures of word reading. The latter examination was important because kindergarten measures shown to be predictive of word reading may also be related to language ability (Storch & Whitehurst, 2002), and their shared relationship could account for some differences in reading comprehension not explained by word reading.
Second, the present study investigated if response to language intervention in kindergarten added to the prediction of third grade reading comprehension. Whereas individual differences in language ability may be expected to remain relatively stable once children enter kindergarten, a dynamic measure of language growth may capture additional variance and/or provide a better estimate of language ability and thus add to the prediction of subsequent reading comprehension difficulties over and above screening measures.
Method
Participants
The participants in this study were 366 kindergarten children from a medium-sized school district. This district is diverse in terms of ethnicity (approximately 63% Caucasian, 11% African American, 6% Hispanic, 7% American Indian/Alaskan Native, 6% Asian/Pacific Islander, and 7% multiracial) and family socioeconomic status (24% free and 11% reduced-price lunch). Participants entered the study in two cohorts, one year apart. In selecting our participants, we oversampled children with increased risk for RD. Although these children are referred to as “at risk,” this designation is based on beginning-of-year school-based assessments and not on our screening battery. Specifically, at-risk status was determined by performance on two subtests from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS; Good & Kaminski, 2002) that were administered to all district kindergarteners by school personnel in the first week of school. To be considered at risk, children had to perform in the “some risk” or “at risk” categories on both the Letter Name Fluency and Initial Sound Fluency subtests. For the present study, it would have been preferred if at-risk children could have been selected on the basis of poor performance on a language assessment. However, no such assessment data were available for all kindergarten children in the district. Nevertheless, at-risk children selected on the basis of DIBELS subtests often had language problems and, as a group, scored approximately one standard deviation below the nonrisk children on a composite score of our two vocabulary assessments and three quarters of a standard deviation below on our narrative assessment (see below for information concerning these assessments).
Approximately 19% of kindergarteners in the district (across the 2 years) met this at-risk status criterion. The majority of these children (N = 263; 150 boys, 113 girls) served as participants in the study. The remaining were excluded because they had limited English proficiency (23), had severe disabilities such as autism or behavior disorders (13), or were unavailable for testing because they moved before testing had begun or a parent or teacher requested nonparticipation (5). In addition to the at-risk participants, we randomly selected 103 children (53 boys, 50 girls) who did not meet the risk criteria on the DIBELS subtests. These children were included to ensure that we would have a sufficient number of participants without RD in third grade.
Between the time the screening battery was administered at the beginning of kindergarten and the end of third grade, 98 children (27%) were lost from the study. Most of the children moved out of the district (91) and were unavailable for testing. Other children were dropped because of parental request (3), later diagnosis of autism or other special needs (3), or excessive absences (1). In addition to the above attrition, 4 children were missing one or more of the screening or reading outcome measures. A complete data set was available for 266 children on one reading outcome measure and 264 on the other.
Measures and Procedures
All participants were administered a battery of screening measures at the beginning of kindergarten by trained examiners from our research team. This battery included measures that previously have been reported to be predictive of word reading outcomes (Catts et al., in press). The screening battery also included measures of vocabulary, grammar, and narration. Direct measures of word reading were also administered in second grade. In addition, a portion of the at-risk participants were provided with language intervention across the kindergarten year, and response to this intervention was assessed by several progress monitoring measures. Finally, measures of reading comprehension were given at the end of third grade, and children were divided into those with good or poor outcomes. The specific measures and procedures used in each component of the study are listed below.
Screening battery
In a related study (Catts et al., in press), we found that measures of letter naming, sound matching, rapid automatized naming, and nonword repetition were highly predictive of word reading outcomes. Models including these variables had an overall accuracy, as measured by the area under the curve (AUC), of .85 to .92 depending on the outcome measure. Others have also reported a strong predictive relationship between one or more of these measures and word reading abilities (Schatschneider et al., 2004; van Weerdenburg, Verhoeven, Balkom, & Bosman, 2009; Vellutino et al., 2008; Wolf, Bowers, & Biddle, 2000). These measures are described below.
Letter knowledge
The Letter Naming Fluency (LNF) subtest of DIBLES (Good & Kaminski, 2002) was administered to asses letter knowledge. In this subtest, the participant is shown a stimulus card containing 11 rows of randomly presented upper- and lowercase letters. The child names as many letters as he or she can in 1 min. This measure was administered on four occasions at the beginning of kindergarten (approximately 2 weeks apart). For this investigation, we used the third occasion because it coincided with when our other screening measures were administered. Test–retest reliability was found to be .91.
Phonological awareness
The Sound Matching subtest from the Comprehensive Test of Phonological Processing (CTOPP; Wagner, Torgesen, & Rashotte, 1999) is an untimed test of the ability to identify the sounds in words (i.e., phonological awareness). The participant is shown a series of stimulus cards, each with a target picture and three test pictures. The examiner provides the name of each picture, and the participant is asked to identify which of the three test pictures starts or ends with the same sound as the target picture. Test–retest reliability is .83 and internal consistency is .93.
Rapid Automatized Naming (RAN)
On this subtest of the CTOPP, the participant is presented with two forms displaying pictured arrays of six common objects repeated six times in a random order. The child names all objects from each form as quickly as possible. The number of seconds required to name the objects from each form was combined to derive the raw score for this measure. The alternate-form reliability is .82.
Nonword Repetition
In this task (Dollaghan & Campbell, 1998), the participant is required to repeat 16 nonwords ranging from one to four syllables in length (four words at each length). Each of the nonwords was composed of early developing phonemes and contained syllables that did not correspond to English lexical items. Nonwords were presented to children via headphones and a high-quality audio-recorder, and participants’ responses were recorded. An examiner scored the audio-recorded responses in terms of the number of consonants in error across the 16 words. A second examiner rescored approximately 13% of the data, and interjudge reliability was 93%.
In addition to word reading predictors, measures of vocabulary, grammar, and narration were also administered as part of the screening battery.
Vocabulary
The Peabody Picture Vocabulary Test–3 (PPVT-3; Dunn & Dunn, 1997) was administered to assess children’s recognition of the meaning of spoken words. The examiner reads a list of words aloud, and the participant is asked to select one of four pictures that corresponds to the meaning of the target word. Test procedures for establishing a basal and ceiling were followed. The split-half reliability is .94 for the age range corresponding to our sample. We also administered the Picture Naming Vocabulary subtest from the Predictive Assessment of Reading (PAR; Wood, Hill, Meyer, & Flowers, 2005). The participant is presented with a series of 35 pictures and asked to provide the name of each. Cronbach’s alpha for this measure is .90.
Grammar
The Sentence Imitation subtest of the Test of Language Development–Primary–3 (Hammill & Newcomer, 1997) was administered as an estimate of grammatical abilities. The participant is presented with a series of spoken sentences that increase in length and grammatical complexity and is required to repeat each sentence as accurately as possible. Whereas this measure is designed to assess grammatical abilities, it also taps verbal memory. Test–retest reliability is .90 and internal consistency is .92.
Narration
The Test of Narrative Language (TNL; Gillam & Pearson, 2004) was administered to all participants at the beginning of the kindergarten year and in April, after the research-based intervention was completed. The TNL assesses children’s ability to comprehend narratives read aloud and to produce narratives with varying degrees of visual support. The internal consistency is .88 and the test–retest reliability is .81.
Word reading
Word reading ability was assessed directly in second grade by four measures. These included the Word Identification and Word Attack subtests from the Woodcock Reading Mastery Tests–Revised–Normative Update (WRMT-R:NU: Woodcock, 1998) and the Sight Word Efficiency and Phonemic Decoding Efficiency subtests of the Test of Word Reading Efficiency–2 (TOWRE-2; Torgesen, Wagner, & Rashotte, 2011). The WRMT-R:NU subtests are untimed measures of a participant’s ability to accurately read words and pronounceable nonwords, whereas the TOWRE-2 subtests are timed measures of these skills. Both instruments have reported reliabilities greater than .90. For the present study, we used z-scores to combine children’s performance on the WRMT-R:NU and TOWRE-2 and derive an overall estimate of second grade word reading ability.
Intervention
Participants selected into the study based on initial risk (at-risk children, N = 263) were randomly assigned to one of two conditions: (a) intervention condition (n = 156) or (b) at-risk control condition (n = 107). Proportionally more children were assigned to the intervention condition than the at-risk control condition to ensure sufficient sample size for planned analyses. Because children received intervention in small groups at the school they attended, it was not possible to use a completely random approach in group selection. Rather, for each of the two cohorts, all at-risk children at a given school were randomly assigned into groups of three (or two/four to ensure that all at-risk children in a school were grouped). Then, for each cohort, we randomly assigned these small groups across the district to the intervention condition or the at-risk control condition at a 3 to 2 rate. This resulted in a total of 47 small groups (59.5%) in the intervention condition and 32 (40.5%) in the at-risk control condition. Children in the intervention condition received the intervention described below. At-risk control children participated in business as usual practice within the district. In many cases, these children received some supplemental small-group intervention from school-based reading specialists or paraprofessionals. The latter intervention was most often focused on alphabetics and included little to no work on vocabulary or narration.
Intervention by our research team began in the third week of October and continued for 26 instructional weeks. The intervention consisted of three 30-min sessions a week. Approximately one half of each session was devoted to instruction in vocabulary and narration. The remainder of the time involved work on phonological awareness and letter knowledge (Blachman, Ball, Black, & Tangel, 2000; Schuele & Dayton, 2000). The vocabulary and narrative intervention took place within the context of 14 storybooks, each of which spanned 4 intervention days. Three vocabulary words were carefully selected from each storybook (Beck, McKeown, & Kucan, 2002; Justice, Meier, & Walpole, 2005) for a total of 42 words taught during the intervention period. The approach used to teach the vocabulary words was primarily based on the work of Beck and colleagues (e.g., Beck et al., 2002; Beck & McKeown, 2001). On the first day of each unit, the students were provided with a child-friendly definition for each of the three words (e.g., “A pebble is a small rock”) and were guided to repeat the definition. They were also shown pictures that represented the target words (not from the book) that served as additional examples. Review of the three target vocabulary words occurred in each of the 3 remaining days of that unit. The review included children repeating the definition in unison, discussing examples or characteristics of the target word (e.g., “Pebbles are small”), and producing sentences using the target word (e.g., “I picked up a pebble in my driveway”). Participants were asked to give a synonym for target words, to make a personal experience connection to the words, and/or to use the words in a new meaningful context and explain their appropriateness in that context. Interventionists assisted the children in this process with scaffolded responses and feedback.
In addition to vocabulary instruction, narrative comprehension and production were taught through repeated experiences with the selected stories. Research regarding narrative development and components (e.g., Mandler & Johnson, 1977; van den Broek & Trabasso, 1986) guided the development of this aspect of the intervention. Prior to the first reading of the book, the interventionist introduced the setting and characters briefly and set a purpose for not only listening to vocabulary, but also listening for something related to the plot (e.g., characters’ goals and actions/attempts to reach their goals). The book then was read aloud in its entirety by the interventionist on the first day, followed by scripted discussion questions, which focused on research-based story grammar elements. The interventionist read the book again on the second day, asked story-element-focused questions from Day 1 that had not been completed or those that had been difficult, then led students in a group retelling using pictures from the storybook. The book was not read again on the third and fourth days, rather the focus was on students engaging more and more in retelling. Thus, on the third day, the interventionist retold the story using a short scripted retelling of the story and selected pictures from the storybook, then guided students to engage in this brief retelling. On the fourth day, again using the short scripted retelling as a guide, the interventionist supported the students as they retold the story as a group and individually. Whenever working with students on retellings, the interventionists emphasized the inclusion of the target vocabulary as well as words that signal story grammar elements (e.g., the when and where of the setting) as well as words and phrases that signal the order of actions/events (e.g., There once was, and then, later, next, at the end, etc.), characters’ names, and important dialogue (e.g., in the book The Teddy Robber, “I’m Tom and you stole my teddy!”).
The intervention was carried out by educators or a graduate student on our research team. The interventionists attended a 2-day workshop that provided theoretical background and training on lesson specific strategies. A procedural fidelity checklist was developed for each lesson plan that was used to document the interventionist’s use of directions, pacing and sequencing of activities, monitoring of student engagement in and completion of activities, and use of any necessary materials. Following workshop training, interventionists were observed by professional research staff to ensure the integrity of implementation of lesson plans using the fidelity checklists until they had achieved 95% or higher fidelity on three consecutive sessions. Subsequently, approximately 20% of the lessons randomly selected were observed by research staff to monitor fidelity for drift.
Assessment of response to intervention
Participants’ response to intervention was assessed in several ways. Following the completion of the intervention, the TNL, which was part of the screening battery, was readministered to all children in the study. Thus, to assess response to narration intervention, groups were compared in terms of pretest–posttest performance on the TNL. To assess response to vocabulary intervention, we measured children’s knowledge of target words in pretest, midyear, and end-of-year (also referred to as posttest) vocabulary probes. Each vocabulary probe assessed children’s knowledge of the 42 target words on a multilevel scale. Children received 0 to 3 points for each item depending on their ability to define, partially define, and/or recognize the meaning of the target word. Because children who were not in intervention were not expected to learn the relatively low frequency target words, the vocabulary probes were only administered to children in the intervention group.
Reading comprehension
At the end of third grade, all participants were administered two measures of reading comprehension. One measure was an experimental measure developed and administered by our research team, and the other was a standardized assessment administered by district personnel.
Reading Comprehension Measure (RCM)
This assessment was an experimental measure similar in format to that found in informal reading inventories such as the Qualitative Reading Inventory–5 (QRI-5; Leslie & Caldwell, 2011). This measure assessed children’s ability to read, comprehend, and answer inferential and noninferential questions about one expository passage (The Puffin, 167 words in length) and two narrative passages (The New Boy, 340 words; The Frog in the Classroom, 506 words). These passages were designed to be similar to those that third graders might encounter in school and were longer than the passages commonly found in standardized reading comprehension measures at this grade level. The passages were created by the second author and had a lexile score appropriate for third graders. The 13 inferential and 13 noninferential questions across the three passages were intended to assess how well children could comprehend the words/sentences in the text, make inferences, and create a mental representation of the text (Kintsch, 1998). Participants read each passage silently and notified the examiner when finished. Following the completion of each passage, the examiner asked eight or nine open-ended questions corresponding to the passage. Participants’ responses to administered questions were audio-recorded. Trained examiners scored each audio-recorded response based on a rubric of acceptable answers. This rubric was created by the authors after carefully evaluating the appropriateness of a subset of responses by the participants. The total number of correct responses served as the raw score. Scorers were trained to 90% or higher reliability before scoring assessments. In addition, approximately 10% of the samples were scored by a second examiner, and the interrater reliability was .93. Following the completion of data collection and scoring, Cronbach’s alpha was calculated and found to be .86.
Measures of Academic Progress: Reading (MAP)
This is a standardized computer adaptive test of reading (Northwest Evaluation Association, 2009) administered as part of the district’s progress monitoring of student achievement. A primary component of this test is children’s ability to read for understanding. Children read short narrative or expository passages via a computer and answer multiple-choice questions that require literal, inferential, or evaluative understanding of the text. Some test items also tap word meaning. The marginal reliability is .95.
Classification of reading outcomes
It is generally agreed that reading outcomes are normally distributed and that there is no clear demarcation between good and poor readers (Fletcher, Denton, & Francis, 2009). For this reason, it has not been uncommon for some researchers interested in RD to use statistical procedures that rely on reading outcomes that are continuous in nature (e.g., Schatschneider et al., 2004). However, in most school or clinic settings the identification of RD represents a binary decision, that is, the child has a reading/learning disability or does not. Therefore, an analysis that divides children into those with good versus poor reading outcomes and identifies the predictors of these outcomes has the most direct implications for early identification. In this study, we used such an approach. For each of the reading comprehension measures, participants were classified as having reading disabilities (RD) or not (non-RD). RD was defined as performance equal to or below the 16th percentile, and non-RD as above this cut score. This cut score corresponds to 1 standard deviation below the mean and is a frequent choice by researchers investigating the early identification of RD (e.g., Compton et al., 2006; Hurford, Schauf, Bunce, Blaich, & Moore, 1994). We used local data to calculate percentile scores rather than normative data. This decision was based in part on the fact that norms were not available for the RCM. Because we oversampled at-risk children, we calculated local norms by using a weighting procedure. Recall that all kindergarteners in the school district were administered DIBELS subtests during the first week of the school year to determine initial risk status. As a result, we were able to ascertain the percentage of children in the district who met our criteria for risk. Using the district rate (18.9%) and the rate in our sample (69.3%), we created a weighting variable that allowed us to convert raw scores or standard scores to z-scores that would have been expected if all children in the district had been assessed on our reading comprehension measures. Thus, although our sample contained a higher percentage of kindergarten children who were at risk than that found in the district, the scores of at-risk participants were given proportionally less weighting to ensure the representativeness of our results.
Finally, it should be noted that the classification of participants as RD versus non-RD was based only on reading comprehension outcomes. No attempt was made to distinguish between those poor readers who had or did not have word reading problems. Therefore, our results speak to the identification of children with reading comprehension problems who have a range of difficulties in word reading and not exclusively to children with comprehension problems alone.
Results
Table 1 displays the correlations between kindergarten screening assessments and third grade reading comprehension measures. All correlations were significant and in the low to moderate range (approximately .3 to .6 range). Negative correlations were expected for rapid naming and nonword repetition, since a higher score on these measures reflected poorer performance. A series of logistic regression analyses were used to examine the relationship between the kindergarten screening measures and third grade reading comprehension outcome (RD vs. non-RD). Because we were primarily interested in the question of whether or not measures of kindergarten language ability would predict reading comprehension outcome over and above other kindergarten predictors, we used a nested approach. In such an approach, we first tested a model in which kindergarten word reading predictors (i.e., LNF, SM, RAN, NWR) were entered into the logistic regression analyses. We then compared the results of this model to models in which each of the kindergarten language measures was entered as an additional variable to the initial model. We also tested models with combinations of the language measures added to models with the kindergarten word reading predictors. Chi-square tests were used to compare the log likelihood scores from additive models to those of models that included only the word reading predictors. We also compared the area under the receiver operating characteristic (ROC) curves derived from these models. An ROC curve is a plot of the true-positive rate (sensitivity) against the false-positive rate (1 – specificity) for each of the cut points of a decision making instrument (Metz, 1978; Swets, 1979). AUC can be used as an overall estimate of the accuracy of the instrument. Differences in the predictive accuracy of models can be interpreted by evaluating AUC differences (Hanley & McNeil, 1983). A critical ratio z is calculated between two AUCs, and a value greater than 1.65 is designated as significant (one-tailed comparison). Critical ratio values are corrected for the correlation introduced by using the same sample of participants to derive two AUCs. Finally, pseudo-R2 values are also provided for each of the models. Similar to R2 in linear regression, pseudo-R2 in logistic regression is an index of the accuracy of a prediction based on a scale from 0 to 1. Pseudo-R2 values for good predictive models are typically low in relationship to comparable linear regression models. However, they can be useful when comparing one model to another (Cohen, 1988). Both the Cox and Snell R2 and Nagelkerke R2 are provided for each model. The latter is a modification of the former and allows for the maximum value of 1, which is not possible for Cox and Snell R2.
Correlations Between Kindergarten Screening and Reading Outcome Measures.
Note. LNF = Dynamic Indicators of Basic Early Literacy Skills: Letter Naming Fluency; MAP = Measures of Academic Progress: Reading; NWR = nonword repetition; PAR = Predictive Assessment of Reading: Picture Naming Vocabulary; PPVT–3 = Peabody Picture Vocabulary Test–3; RAN = Comprehensive Test of Phonological Processing (CTOPP): Rapid Automatized Naming; RCM = Reading Comprehension Measure; SI = Test of Language Development–Primary–3: Sentence Imitation; SM = CTOPP: Sound Matching; TNL = Test of Narrative Language; WR = combination score of Woodcock Reading Mastery Tests–Revised–Normative Update Word Identification and Word Attack subtests and Test of Word Reading Efficiency–2 in second grade.
p < .05. **p < .01.
Screening
Table 2 displays the results of the initial logistic regression analyses for each measure of reading comprehension. For the RCM, the initial model including word reading predictors explained considerable variance in reading comprehension outcome. LNF was the best predictor, but all four kindergarten variables were significant indicators of reading comprehension outcome. Table 3 shows the additive effects of each of the kindergarten language measures. These results indicated that for the RCM outcome, two of the kindergarten language measures (i.e., PPVT-3, PAR) individually added significantly to the initial model (χ2 = 6.81, 7.68, p < .01) and the others (i.e., TNL, SI) did not (χ2 = 2.44, 0.01, p > .05). Furthermore, a model that included both PPVT-R and PAR was not significantly better than a corresponding model with either measure alone (χ2 = 2.57, 1.70, p > .05). Results also showed that the additive model including PAR had a significantly higher AUC than that of the initial model (z = 1.70, p < .05). The AUC difference between the initial model and the model including PPVT-R did not reach significance (z = 1.12, p > .05).
Logistic Regression Analyses for Kindergarten Word Reading Predictors.
Note. LNF = Dynamic Indicators of Basic Early Literacy Skills: Letter Naming Fluency; MAP = Measures of Academic Progress: Reading; NWR = nonword repetition; RAN = Comprehensive Test of Phonological Processing (CTOPP): Rapid Automatized Naming; RCM = Reading Comprehension Measure; SM = CTOPP: Sound Matching.
Logistic Regression Analyses for Kindergarten Word Reading Predictors and Language Measures.
Note. MAP = Measures of Academic Progress: Reading; PAR = Predictive Assessment of Reading: Picture Naming Vocabulary subtest; PPVT–3 = Peabody Picture Vocabulary Test–3; RCM = Reading Comprehension Measure; SI = Test of Language Development–Primary–3: Sentence Imitation; TNL = Test of Narrative Language; WRP = Word reading predictors in kindergarten, including Letter Name Fluency, Sound Matching, Rapid Automatized Naming, and Nonword Repetition.
Table 2 also includes the results for the MAP outcome. The initial model indicated that all the word reading predictors except the LNF were significant predictors of the MAP outcome. It should be noted that this model was not as predictive of reading outcome as was the initial model for the RCM. The latter had an AUC of .873 compared to .774 for the MAP outcome model. Other results shown in Table 3 revealed that three of the kindergarten language measures (i.e., PPVT-3, PAR, and SI) each individually added significantly to the initial model (χ2 = 5.85, 7.45, 5.90, p < .05) and the other (i.e., TNL) did not (χ2 = 2.37, p > .05). However, models including PPVT-R, PAR, or SI did not result in a significant increase in AUC (z = 1.17, 0.99, 1.23, p < .05).
As noted in the introduction, one or more of the so-called word reading predictors could overlap partially with language measures in the prediction of reading comprehension outcome, independent of word reading. This could have the effect of reducing the impact of kindergarten language measures in the prediction of reading comprehension outcomes. To test this, we again ran a series of nested logistic regression models. In the initial models, we included only second grade word reading, and in subsequent models, added the kindergarten language measures. Results shown in Table 4 indicated that the model including the direct measure of word reading in second grade was less predictive of RCM outcome than was the model including the kindergarten word reading predictors. The AUC in the former model was significantly lower than that in the latter model (z = 3.78, p < .01). Additional results showed that models including PPVT-3, PAR, or TNL proved to have a greater AUC than the initial model including second grade word reading alone (z = 2.41, 2.72, 2.49, p < .01). No model that included multiple language measures was significantly better than a corresponding model with a single language measure (p > .05).
Logistic Regression Analyses for Second Grade Word Reading and Language Measures.
Note. MAP = Measures of Academic Progress: Reading; PAR = Predictive Assessment of Reading: Picture Naming Vocabulary subtest; PPVT–3 = Peabody Picture Vocabulary Test—3; RCM = Reading Comprehension Measure; SI = Test of Language Development–Primary–3: Sentence Imitation; TN = Test of Narrative Language; second grade WR = combination score of Woodcock Reading Mastery Tests–Revised–Normative Update: Word Identification and Word Attack and Test of Word Reading Efficiency–2.
We also reran the analyses for the MAP outcome using a model that included second grade word reading. Results indicated that the direct measure of word reading accounted for a much greater amount of variance in MAP outcome (AUC = .871) than did kindergarten word reading predictors (AUC = .774, z = 2.8, p < .01). This was the opposite of what was found for the RCM outcome, suggesting that the RCM may be less influenced by word reading than the MAP. Further results (see Table 4) showed that models that also included PPVT-3 or PAR had significantly lower log likelihoods than did a model with second grade word reading alone (χ2 = 8.78, 6.47, p < .05). Results also showed that the additive model including PPVT-R had a higher AUC than the initial model (z = 2.12, p < .05), but this was not the case for the additive model including PAR (z = 1.45, p > .05).
Response to Instruction
Further analyses were carried out to determine if response to language intervention added significantly to the prediction of reading comprehension outcome over and above screening measures. Recall that a portion of the at-risk children were provided with intervention that included work on vocabulary and narration. The remainder of the at-risk children were placed in a control group that did not receive intervention from our research team. Before examining the added contribution of response to intervention, it was necessary to investigate the effectiveness of the intervention. In the case of narration, we compared the performance of the at-risk intervention group to that of the at-risk control group and the typical control group on pretest–posttest measures of narrative abilities (TNL vs. TNL2). Figure 1 shows that the at-risk intervention group made larger gains in narrative skills than did the other groups. Overall, an ANOVA showed a significant pretest–posttest effect for narration, F(1, 310) = 523.8, p < .001, η2 = .628, and a significant group effect, F(2, 310) = 16.09, p < .001, η2 = .094. Most important, there was a significant interaction, F(2, 310) = 9.49, p < .001, η2 = .058. Follow-up Tukey honestly significant difference (HSD) tests showed that for the pretest the at-risk intervention and at-risk control groups did not differ significantly from each other (p > .05) but did perform significantly less well than the typical control group (p < .001). At posttest, Tukey HSD tests indicated that the at-risk intervention group performed significantly better than the at-risk control group (p < .05). However, the former group still performed less well than the typical control group (p < .05).
Subgroup performance on the pretest and posttest of the Test of Narrative Language.
A different approach was necessary to evaluate the effectiveness of the vocabulary intervention. Because this intervention involved teaching the meaning of a predetermined set of target words, most of which were novel to participants, we evaluated the learning of these words only in the at-risk intervention group. Thus, to measure gains, the at-risk intervention group needed to serve as its own control. Our design allowed us to control for extraneous factors in the analysis. Recall that we assessed children’s knowledge of the target words in a pretest, midyear test, and end-of-year test. Half of the target words were taught between pretest and midyear assessments and the other half between midyear and end-of-year assessments. Thus, we could examine the gains made in target words prior to and after instruction at two points in time. Table 5 shows participants’ performance on the vocabulary probe for target words taught during the first half and words taught during the second half at each assessment. A 3 (time of assessment) × 2 (when words were taught) ANOVA was carried out to evaluate the effects of the intervention. Given our design, the interaction between time of assessment and when words were taught was of most relevance and was found to be highly significant, F(2, 142) = 503.63, p < .001, η2 = .780. A follow-up ANOVA and paired sample t tests for words taught during the first half revealed significant differences between the pretest and midyear test (t = 27.10, df = 142, p < .001). There was also a significant difference between midyear and end of year (t = 6.59, df = 142, p < .001), but as predicted, it was smaller in size. Similar analyses for words taught during the second half found a relatively small significant difference between pretest and midyear test (t = 8.95, df = 142, p < .001) and a larger significant difference between midyear and end-of-year test (t = 27.75, df = 142, p < .001).
Descriptive Statistics of Vocabulary Probe.
To examine the contributions of resonse to intervention, we again used a series of nested models, but this time for the at-risk intervention group alone. Table 6 shows that for this reduced sample, LNF was the only kindergarten word reading predictor that was significant for both reading comprehension outcomes. In further analyses (see Table 7), we added the pretest vocabulary probe (VP1) and TNL to the initial model. For the RCM outcome, results showed that these additive models had significantly lower log likelihoods than the initial model (χ2 = 19.73, 11.85, p < .001). Although the AUCs for the additive models were quite a bit higher than that of the initial model, the differences did not reach significance (z = 1.26, 0.72, p > .05). To further test for the contribution of response to intervention in the prediction of reading comprehension outcomes, we used an autoregressive approach in which the end-of-year vocabulary probe and/or TNL were added to models that included the corresponding pretest measure. If both the pretest and posttest measures are significant, it can be inferred that initial language ability and growth in language ability are related to reading outcome. Our results for the RCM showed that the nested model including vocabulary probe posttest had a significantly lower log likelihood (χ2 = 16.32, p < . 001) and significantly larger AUC (z = 1.91, p < .05) than the model with the word reading predictors and the vocabulary probe pretest. Although the vocabulary posttest proved to be a significant predictor, conclusions concerning the contributions of vocabulary growth are tempered by the fact that the pretest vocabulary probe was no longer significant when the posttest probe was added. Thus, rather than growth being a contributing factor, it appears that it was the number of words the child knew at posttest that was most important. In the case of the TNL, we found that adding the pretest to the initial model with word reading predictors significantly improved the prediction of the RCM outcome (χ2 = 12.00, p < . 001). However, when the TNL posttest was added to the model with the word reading predictors and TNL pretest, it did not add significantly to the prediction (χ2 = 0.815, p > .05).
Logistic Regression Analyses for Kindergarten Word Reading Predictors in Intervention Group.
Note. LNF = Dynamic Indicators of Basic Early Literacy Skills: Letter Naming Fluency; MAP = Measures of Academic Progress: Reading; RAN = Comprehensive Test of Phonological Processing (CTOPP): Rapid Automatized Naming; RCM = Reading Comprehension Measure; SM = CTOPP: Sound Matching; NWR = nonword repetition.
Logistic Regression Analyses for Kindergarten Word Reading Predictors and Language Measures in Intervention Group.
Note. MAP = Measures of Academic Progress: Reading; RCM = Reading Comprehension Measure; TNL = Test of Narrative Language; TNL2 = Test of Narrative Language posttest; VP1 = vocabulary probe pretest; VP3 = vocabulary probe posttest; WRP = word reading predictors in kindergarten, including Letter Name Fluency, Sound Matching, Rapid Automatized Naming, and Nonword Repetition.
Table 7 shows that in the case of the MAP outcome, neither the vocabulary probe nor the TNL pretest added significantly to the initial model that included word reading predictors (p > .05). Further results revealed, as was the case for the RCM outcome, that the end-of-year vocabulary probe added significantly to the model containing the word reading predictors and the vocabulary probe pretest (χ2 = 4.06, p < .05). But again, the pretest was not significant in this model. Table 8 highlights the results for the logistic regression analyses with the intervention group when second grade word reading was used in the initial model. The pattern of results from this set of analyses is essentially the same as that found in analyses using kindergarten word reading predictors.
Logistic Regression Analyses for Second Grade Word Reading and Language Measures in Intervention Group.
Note. MAP = Measures of Academic Progress: Reading; RCM = Reading Comprehension Measure; TNL = Test of Narrative Language; TNL2 = Test of Narrative Language posttest; VP1 = vocabulary probe pretest; VP3 = vocabulary probe posttest; second grade WR = combination score of Woodcock Reading Mastery Tests–Revised–Normative Update: Word Identification and Word Attack and Test of Word Reading Efficiency–2.
Discussion
This investigation sought to determine if measures of language ability in kindergarten would provide a unique prediction of subsequent difficulties in reading comprehension. Our results showed that language measures administered at the beginning of kindergarten did add significantly to the prediction of reading comprehension outcome at the end of third grade. Whereas the contribution was relatively small, language measures did predict reading outcome over and above kindergarten measures related to word reading as well as direct measures of word reading in second grade. Such a finding is consistent with the simple view of reading that states that reading comprehension is the product of word recognition and language comprehension (Gough & Tunmer, 1986). Whereas other studies have shown a unique relationship between early language ability and subsequent reading comprehension (Kendeou at al., 2009; National Institute for Literacy, 2008; NICHD Early Child Care Research Network, 2005), ours is perhaps the first to indicate that these measures can be used as part of a screening battery to uniquely identify children at risk for problems in reading comprehension. We found that measures of vocabulary, grammar, and narration were each uniquely predictive of reading comprehension in some cases. However, vocabulary measures consistently proved to be the best unique predictors. Others have also found vocabulary to be a good proxy for language ability and its relationship to reading comprehension (Olson et al., 2011; Protopapas, Simos, Sideridis, & Mouzaki, 2012; Spencer, Quinn, & Wagner, 2014). Of particular relevance is our finding that a measure of expressive vocabulary was a good predictor of reading comprehension status. This measure, a subtest from the PAR (Wood et al., 2005), required children to provide the name of a small number of pictured items. Such a measure is relatively brief in nature and therefore could be added to a screening battery with only limited additional time demands. In fact, some screening instruments already incorporate an expressive vocabulary component in their assessment (e.g., Foorman, Torgesen, Crawford, & Petscher, 2009). Alternatively, a receptive vocabulary measure could also be useful in screening batteries. We found such a measure, the PPVT-3, to provide a unique prediction of reading comprehension outcome. Whereas the PPVT-3 requires more time to administer than is available in most screening situations, a shorter receptive measure or one delivered in an adaptive format could prove beneficial.
Another potentially relevant finding that may have implications for screening was that kindergarten-administered measures that had been shown to be predictive of word reading actually were more closely related to third grade reading comprehension in some cases than was a second grade measure of word reading itself. Specifically, this was the case for the primary analysis involving the RCM outcome measure. In this instance, a model including kindergarten measures of letter naming, phonological awareness, rapid naming, and nonword repetition had a significantly higher AUC than one including second grade word reading alone. We did not find the same result for the MAP or the RCM with the intervention sample alone (the latter difference was in the same direction but not significant). Nevertheless, this finding deserves some attention. It may be that some of the measures believed to be precursors of word reading actually are associated with general language ability as well, and it is this shared relationship that also influences reading comprehension. Some studies have shown a relationship between language ability and performance on phonological awareness and nonword repetition tasks (Gathercole, 1995; Metsala, 1999; Storch & Whitehurst, 2002; Wise, Sevcik, Morris, Lovett, & Wolf, 2012). Also, research has documented that some children with language impairments have early deficits in phonological awareness or nonword repetition, but do not develop word reading difficulties (Bishop, McDonald, Bird, & Hayiou-Thomas, 2009; Catts & Adlof, 2011). If the present finding concerning word reading predictors can be replicated, it would indicate that these measures can also be useful in identifying some reading comprehension problems that are not associated with word reading deficits.
The present study also investigated if response to language intervention would add significantly to the prediction of reading comprehension difficulties. In an RTI framework, progress monitoring in response to supplemental intervention or classroom instruction is proposed as a method of improving the accuracy of early identification. As discussed above, there is some limited evidence that progress monitoring during kindergarten can add uniquely to the prediction of word reading difficulties (Al Otaiba et al., 2011; Catts et al., in press; Vellutino et al., 2008). However, little attention has been given to response to language intervention and its relationship to the identification of reading comprehension problems. A portion of the at-risk children in our sample was provided with supplemental language intervention across the kindergarten year. This intervention focused on teaching children a set of novel words as well as how to understand and retell narratives. Results showed that the intervention was effective in improving word knowledge and narrative skills. Further analyses revealed that children’s response to this intervention was uniquely predictive of reading outcome, especially in the case of response to vocabulary instruction. Children’s performance on the end-of-year vocabulary probe added significantly to the prediction of reading comprehension status over and above word reading predictors or word reading itself. It should be noted that once the end-of-year vocabulary score was entered into the model, the pretest probe score was not significant. This suggests that growth in vocabulary per se may not be indicative of reading comprehension outcome, for it did not matter where a child started, but rather the level the child finally achieved in vocabulary was the critical factor. It could be argued that the pretest vocabulary probe was an unfair indicator of language ability since many of the words taught were unfamiliar to the children. However, there was enough variability in this measure for it to be highly significant in an additive model that did not include the end-of-year vocabulary probe. Although not discussed in the results section, we did find that the PPVT-3 was a significant predictor in an additive model with the end-of-year vocabulary probe and word reading predictors. Thus, an initial measure of vocabulary and response to vocabulary intervention may each provide unique information concerning reading comprehension outcome. As discussed above, the dynamic nature of the vocabulary intervention/assessment likely makes it a sensitive measure of language ability and thus a useful predictor of reading comprehension.
Whereas our vocabulary intervention/assessment protocol may be beyond what could be implemented in most kindergarten classrooms, it may be possible to use shorter word learning tasks to provide additional information about risk for reading comprehension problems. Numerous investigations have studied word learning in typical and impaired populations (e.g., Gray, 2004; Nash & Donaldson, 2005) and adaptations of the methodology employed could be applied to early identification. In addition, less formal assessments of children’s response to language instruction by teachers or other professionals (e.g., speech-language pathologists) might also be used to gauge risk for reading comprehension difficulties. Future studies will need to further examine how information concerning response to language intervention can assist educators in early identification.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by funding from the Institute of Education Science.