Validation of the Risk Factor Screen for Reading (RiFS-Reading) Screening Tool for the Early Identification of Reading Problems

Abstract

Early diagnosis of children with reading disorders is essential for intervention and academic success. Many children with reading difficulties have a deficit in phonemic awareness. A web-based, group-administered screening vehicle, the Risk Factor Screen for Reading (RiFS-Reading), was developed to quickly identify students who are “At Risk” for reading difficulties by assessing knowledge of phonemic awareness. The goal was to establish the validity of the RiFS-Reading measure. Results supported construct validity through factor analysis and criterion validity by comparisons with performance on well-established measures of phonemic awareness and reading. Phonemic awareness on the RiFS-Reading was independent of other language skills.

Keywords

literacy childhood classroom achievement learning assessment phonemic awareness

Reading and literacy among school-age children is a serious concern in the US, especially since the abrupt school closures in spring 2020 (Domingue et al., 2022). Reading disorders may occur in as many as 1/5 to 2/5 children, many of whom are not diagnosed until second or third grade, wasting valuable time for remediation and instruction (Lyon, 1996, 1998; Siegelman et al., 2022). Successful remediation is much more difficult and costly if intervention is delayed. Therefore, it seems reasonable that early diagnosis of reading disorders could significantly improve children’s lives.

Francis et al. (1996) showed that most school-age children with reading deficits fail to catch up with their peers. A child’s third-grade reading ability can be used to predict overall long-term academic success, as 75% of children with reading disabilities who are not identified before third grade continue to have reading disabilities by ninth grade, and fewer than 2% go on to participate in four-year programs after high school (Hamilton & Glascoe, 2006). Because these children rarely catch up, the consequence of a slow start becomes monumental as the deficits accumulate over time. The effects of failure to acquire early word reading skills range from negative attitudes toward reading to missed opportunities for vocabulary growth and the development of reading comprehension strategies to less practice in reading. The best solution therefore is to allocate resources for early identification of students with deficits in the areas that contribute to reading skills (Torgesen, 1999; Torgesen et al., 2010) and then provide remediation (Ehri et al., 2001).

The most common cause of early reading difficulties has been found to be weaknesses in knowledge of letters and letter sounds and phonological processing, including phonemic awareness (Catts et al., 2002; Milankov et al., 2021; Torgesen, 1999; Torgesen, et al., 2010). Torgeson recommended measuring phonemic awareness based on performance in three categories: sound comparison, phoneme segmentation, and phoneme blending (Torgesen, 1999; Torgesen, et al., 2010). He and his colleagues helped develop a comprehensive measure to assess these skills, namely, the Comprehensive Test of Phonological Processing, Second Edition (CTOPP-2; Wagner et al., 2013). There are also existing screening tests of phonemic awareness, such as the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) (Good et al., 2001), i-Ready (Curriculum Associates, 2019), Measure of Academic Progress (MAP) Assessment (Northwest Evaluation Association, 2016), the Phonological Awareness Skills Test (Kilpatrick, 2016), and the EarlyBird Screener (Gaab & Petscher, 2021); however, they either need to be individually administered, do not fully capture several important constructs, take a long time to administer, and/or are costly.

A novel screening measure has therefore been developed: the Risk Factor Screen for Reading (RiFS-Reading). It differs from other screening tools in that it targets students in kindergarten and first grade and is completely web-based, engaging the students with graphics on a tablet or touch-screen computer. It also can be administered in a small group format, which is significantly more efficient, as teachers and schools often do not have the time or resources to test students individually. Furthermore, results are available almost immediately and provide a cost-effective way to identify children at risk for early reading difficulties. The goal of this research was to determine whether RiFS-Reading is a valid tool to identify students who are at risk for developing reading problems so that they can receive early intervention.

Methods

Objective

The purpose of this study was to examine the reliability and validity of the Risk Factor Screen for Reading (RiFS-Reading) Scale. It was hypothesized that the scale will have (a) construct validity (as measured by item evaluation during exploratory factor analysis), (b) good reliability (as measured by Cronbach’s alpha), (c) convergent validity (as compared to Comprehensive Test of Phonological Processing, Second Edition and Woodcock-Johnson Test of Achievement, Fourth Edition (WJA-IV) Letter-Word Identification test), and (d) discriminant validity (as compared to the Wechsler Individual Achievement Test, Fourth Edition (WIAT-IV) Expressive Vocabulary and Receptive Vocabulary tests).

Procedure

The RiFS-Reading test was administered first followed by the administration of the four tests used to measure validity (i.e., CTOPP-2, WJA Letter-Word Identification, WIAT-IV Receptive Vocabulary, and WIAT-IV Expressive Vocabulary). These measures were administered within four months of the RiFS-Reading, three months of which were during the summer in which no classroom instruction was provided. However, most students received the measures within a month interval of the previous test (M = .94 months, SD = 1.59 months). Parents provided informed consent for their children to participate in the study. This study was overseen by WCG Western IRB. Analyses were conducted using Microsoft Excel and IBM SPSS Statistics, version 29 (IBM Corp, Armonk, NY, USA).

A subset of students (N = 13) were re-administered the RiFS-Reading measure to obtain valid profiles. Four students were re-administered the RiFS-Reading to decrease the time between the RiFS-Reading and the other measures used in this study (i.e., CTOPP-2, WJA-IV Letter-Word Identification, WIAT-IV Receptive Vocabulary, and WIAT-IV Expressive Vocabulary), four students had sections on the RiFS-Reading that “timed out” either due to inattention or internet instability, and five students exhibited behaviors that suggested that they were not attending to the task.

Measures

Risk Factor Screen for Reading Scale

The RiFS-Reading test is conducted on a handheld tablet that is given to each student so they can work independently. The measure was developed for each student to complete individually but can be administered simultaneously to multiple students using touch-screen devices. A small group of students can sit at their desks at the same time with separate tablets. A proctor can walk around to ensure that students are engaged and do not experience technical difficulties. The test was designed to take 10–15 minutes to complete and evaluates students’ understanding of phonics and phonemic awareness. Phonics was measured using two subtests: Letter Recognition, which assesses the student’s knowledge of the letters of the alphabet, and Letter-Sound Association, which assesses the student’s knowledge of the sounds the letters make. The Letter Recognition subtest included 14 items, where the student is prompted to select a specific letter from three options. The Letter-Sound Association subtest included nine items, where the student is prompted to select the letter associated with the sound provided. The letters utilized in the Letter Recognition and Letter-Sound Association subtests were selected based on an analysis of the most frequently utilized letters in words listed in the Concise Oxford Dictionary (Baldick, 1990).

Phonemic awareness can be grouped into three categories: Sound Comparison, Phoneme Segmentation, and Phoneme Blending. Utilizing the deductive method for item development (Boateng et al., 2018), items were generated to correspond to the relevant categories, with items utilizing one-syllable words of no more than three-to-four letters to ensure appropriateness for students as young as five years old. Phonemic awareness was measured using three subtests: (1) Sound Comparison, which assesses the student’s ability to recognize the beginning sound of a word, (2) Phoneme Segmentation, which assesses the ability of the student to break apart a word into its individual sounds, and (3) Phoneme Blending, which assesses the ability of the student to combine sounds into a word. The Sound Comparison subtest included four items, where a picture of an object was presented, and the student was prompted to choose another one-to-two-syllable word that begins with the same sound (e.g., “Find the word that begins with the same sound as pen”). Pictures and sounds were provided for each choice (e.g., van, pot, horse). The Phoneme Segmentation subtest included six items, where the student was presented with the name and picture of an object that consisted of three-to-four letters and one-syllable (e.g., “The word is pot”). The student was then asked to select from three options the group of sounds that represent the word broken down into its component phonemic sounds (e.g., /p/ /ŏ/ /t/, /p/ /ă/ /t/, /p/ /ŭ/ /p/). The Phoneme Blending subtest included six items, where the student was presented with the phonemes that comprise a three-phoneme, one-syllable word (e.g., “What word do these sounds make? /b/ /e/ /d/”). The student was asked to select from three options which word was formed when the phonemes were blended together (e.g., bet, bad, bed).

Two (of the six) items from the Phoneme Segmentation subtest and two (of the six) items from the Phoneme Blending subtest were eliminated prior to conducting the analysis. The last items from these subtests were eliminated due to observations of fatigue as the test progressed as well as occasional technical errors that occurred due to internet and/or programming issues, which occasionally made the test stop or skip those items at random times. The additional items from the Phoneme Segmentation and Phoneme Blending tests were eliminated because some of the choices sounded very similar due to the articulation of the narrator on the test (e.g., pat and pet) and therefore appeared to cause confusion in the students.

On the RiFS-Reading, a composite score was created for Phonics (Letter Recognition and Letter-Sound Association) and for Phonemic Awareness (Sound Comparison, Phoneme Segmentation, and Phoneme Blending). The total number of correctly answered items was summed across each of the relevant subtests for the corresponding composite. A student was identified to be “At Risk” in a given area if their score was less than 75% and “Not At Risk” if their score was 75% or higher. The threshold of 75% was chosen as the cutoff between “At Risk” and “Not At Risk” students to denote below average performance on the test and allow for the possibility of careless mistakes due to the young age of the students.

Comprehensive Test of Phonological Processing Second Edition.

The Phonological Awareness subtests of the Comprehensive Test of Phonological Processing, Second Edition (CTOPP-2; Wagner et al., 2013) were individually administered. This well-validated measure of phonological processing consists of three subtests: Elison, Blending Words, and Sound Matching. The Elison subtest measures the ability to remove phonological segments from spoken words to form other words, the Blending Words subtest measures the ability to synthesize sounds to form words, and the Sound Matching subtest measures the ability to select words with the same initial and final sounds. For students seven years old and older, the Phoneme Isolation test, which measures the child’s ability to isolate individual sounds within words, was used in place of the Sound Matching test, as the test changes at this age to mirror the increased developmental expectations but measures the same underlying skill. The total time to administer all three subtests was approximately 20 minutes.

On the CTOPP-2, scaled scores for each of the subtests (Elison, Blending Words, Sound Matching/Phoneme Isolation) have a mean score of 10 and a standard deviation of 3. The Phonological Awareness Composite score has a mean score of 100 and a standard deviation of 15. The scale authors reported that these subtests’ internal consistency coefficients are .86 or higher, and the Phonological Awareness Composite internal consistency coefficient is .92. Validity of the CTOPP-2 subtests and composites was demonstrated by correlations with other validated measures of phonological awareness. Coefficients range from .47 to .85 for the subtests and from .64 to .78 for the composite (Wagner et al., 2013).

Woodcock-Johnson Tests of Academic Achievement, Fourth Edition Letter-Word Identification Subtest

The Letter-Word Identification subtest of the Woodcock-Johnson Tests of Academic Achievement, Fourth Edition (WJA-IV; Schrank et al., 2014), which measures word identification skills, was individually administered to each participant and took approximately 5 minutes. Initial items require a student to recognize and identify individual letters. Later items require the student to recognize a word from an array and then read words of increasing difficulty in isolation. The WJA-IV Letter-Word Identification was scored based on age and translated into standard scores (M = 100, SD = 15). Letter-Word Identification has a median reliability of .92. Validity of the WJA-IV reading cluster was demonstrated by correlations with other validated measures of reading, including a coefficient of .83 with the Kaufman Test of Academic Achievement, Second Edition Reading Composite and a coefficient of .92 with the Wechsler Individual Achievement Test, Third Edition Basic Reading Composite (Schrank et al., 2014).

Wechsler Individual Achievement Test, Fourth Edition Receptive Vocabulary and Expressive Vocabulary Tests

The Wechsler Individual Achievement Test, Fourth Edition (WIAT-IV; Breaux, 2020) Receptive Vocabulary test is an assessment used to measure comprehension of vocabulary. Students are asked to select the picture from an array of four possibilities that best illustrates the meaning of each word. The Expressive Vocabulary test is a measure used to determine a student’s ability to retrieve words and demonstrate knowledge of vocabulary. Students are presented with a picture, are provided with a definition, and are then asked to say the word that best describes the picture and definition. Assessments were conducted one-on-one and took approximately 5 minutes to administer both subtests. The scale authors reported that internal consistency reliabilities range from .69 to .77 for the Expressive Vocabulary test and .79 to .84 for the Receptive Vocabulary test for the ages tested. Test-retest reliabilities are .84 and .66, respectively (Breaux, 2020).

Data Preparation

Prior to conducting analyses, the data were cleaned per standard procedures. Data were examined using descriptive statistics and histograms to assess for skewness, kurtosis, and outliers. All values were within normal range. Scales did not show evidence of abnormal skewness or kurtosis. Five students had missing values on the CTOPP-2 and/or WIAT-IV Expressive and Receptive Vocabulary scores, so the values were deleted pairwise in the analyses.

Data Analysis Plan

Frequencies of gender, ethnicity, and languages spoken at home were calculated in addition to the mean and standard deviation age in months of the participants in the study. An exploratory factor analysis was then conducted to determine how many factors described the skills measured in the RiFS-Reading screening tool and which items within each factor were similar to each other. The reliability of the measure was assessed using Cronbach’s alpha. Convergent validity was examined by conducting one-way ANOVAs between the “At Risk” and “Not At Risk” groups on the RiFS-Reading and scores on corresponding subtests of a measure of phonemic awareness (i.e., CTOPP-2 Phonological Processing) and a measure of reading (i.e., WJA-IV Letter-Word Identification). Discriminant validity was then examined by comparing scores on the RiFS-Reading to scores on measures of expressive and receptive vocabulary. (See pages 9 through 14 for analyses of reliability and construct, convergent, and discriminant validity). Use of the threshold of p ≤ .05 was employed to determine significance of results. Partial eta-squared values were considered to be small at the .01 level, medium at the .06 level, and large at the .14 level.

Results

Participants

A total of 88 students were approached to participate. Fifty students (29 boys, 21 girls) between the ages of 5 and 7 years (M = 79.27 months, SD = 6.33 months) were provided with consent by a parent or guardian and participated in the study. Table 1 describes the demographic characteristics of the participants. All students attended schools in upper-middle class suburban neighborhoods. All were White except for one student who identified as Black and four who identified as Hispanic. The main language spoken at home was English for all students except two who spoke Spanish and English at home and one who spoke Hebrew and English at home. A majority of students were evaluated in small groups in classroom settings, whereas the remaining students were administered the RiFS-Reading individually because they were receiving academic support due to concerns about their academic progress.

Table 1.

Demographic characteristics.

	N	%
Gender
Female	21	42.0
Male	29	58.0
Ethnicity
White	45	90.0
Hispanic	4	8.0
Black	1	2.0
Language spoken at home
English	47	94.0
English and Hebrew	1	2.0
English and Spanish	2	4.0
Mean age in months at first assessment (standard deviation)	79.27 (6.33)

Scale Validation

Construct Validity

Exploratory Factor Analysis

Initially, the factorability of the correlation matrix of the 35 items of the RiFS-Reading was examined (14 Letter Identification, 9 Letter-Sound Association, 4 Sound Comparison, 4 Phoneme Segmentation, and 4 Phoneme Blending). The exploratory factor analysis revealed that the matrix was non-positive definite such that not all of the eigenvalues were positive. A closer examination revealed that very few of the students were categorized as “At Risk” on the Letter Identification and Letter-Sound Association subscales (two and three, respectively). There was therefore very little variability in the students’ performance, which may have contributed to this characteristic of the matrix. Therefore, the Letter Identification and Letter-Sound Association items were excluded from further analysis.

The factorability of the correlation matrix of the 12 items measuring Phonemic Awareness was then examined. It was considered to be factorable, as indicated by results of the Kaiser-Meyer-Olkin (KMO) test = .75 and Bartlett’s test of sphericity χ² (66) = 218.76, p < .001. Exploratory factor analysis (EFA) was carried out using Principal Component Analysis and Promax method of rotation. To determine the number of factors to retain, the “eigenvalues greater than 1.00 rule” and the Scree test (Cattell, 1966) were used. The EFA indicated three factors with eigenvalues greater than 1.0, which explained 62.27% of the total variance. The standardized factor loadings for the retained items were acceptable using a guideline of greater than .4 for all three factors. As can be seen from Table 2, the pattern matrix revealed that the first factor was composed of the four Phoneme Blending items with saturations ranging between .729 and .851. The eigenvalue for Factor 1 was 4.26, explaining 35.51% of the total variance. Factor 2 was composed of four Phoneme Segmentation items with saturations between .408 and .884 and an eigenvalue of 1.75, explaining 14.56% of the total variance. Factor 3 was composed of four Sound Comparison items with saturations ranging from .536 to .827 and eigenvalue of 1.47, explaining 12.21% of the total variance. There were two items that cross-loaded but did so less strongly onto the other factors: one Sound Comparison item with a saturation of .345 cross-loaded with Factor 1 and one Sound Comparison item with a saturation of .393 cross-loaded with Factor 2. The Scree plot, shown in Figure 1, revealed the EFA solution and shows a substantial drop after the first, followed by the second factor.

Table 2.

Pattern Matrix for Exploratory Factor Analysis using Promax Method of Rotation.

	Component
RiFS-Reading	Factor 1	Factor 2	Factor 3	h ²
Phoneme Blending 5 (Hip)	.851			.692
Phoneme Blending 3 (Bug)	.756			.562
Phoneme Blending 4 (Can)	.743			.623
Phoneme Blending 1 (Bed)	.729			.627
Phoneme Segmentation 4 (Pot)		.884		.786
Phoneme Segmentation 5 (Stop)		.765		.550
Phoneme Segmentation 2 (Cup)		.761		.758
Phoneme Segmentation 3 (Mom)		.408		.388
Sound Comparison 3 (Hand)			.827	.650
Sound Comparison 4 (Moon)		.393	.691	.617
Sound Comparison 1 (Pot)			.685	.613
Sound Comparison 2 (Table)	.345		.536	.606
Eigenvalue	4.26	1.75	1.47
% variance	35.51	14.56	12.21

Kaiser-Meyer-Olkin (KMO)	.75
	χ²	df
Bartlett’s test of sphericity	218.76^***	66
Goodness of Fit	35.81	33	p = .34

Note. Words in parentheses indicate the stimulus items.

Note. h² refers to the communality of each item.

^***p < .001.

Figure 1.

Scree plot for exploratory factor analysis.

Intercorrelations between RiFS-Reading Subscales

Intercorrelations also were conducted between the raw scores of each of the RiFS-Reading subscales (see Table 3). The correlations between the Phoneme Blending, Phoneme Segmentation, and Sound Comparison RiFS-Reading subtests were significant.

Table 3.

Intercorrelations between the performance on each of the RiFS-Reading subscales.

	Phoneme Blending	Phoneme Segmentation	Sound Comparison
Phoneme Blending	1.00	.469^**	.378^**
Phoneme Segmentation	.469^**	1.00	.338^*
Sound Comparison	.378^**	.338^*	1.00

Note. N = 50.

^*p < .05.

^**p < .001.

Reliability

Internal Consistency Indices

The internal consistency of the RiFS-Reading subscales and its subscales were calculated with Cronbach’s alpha coefficients. The internal consistency values were alpha = .82 for the total scale, alpha = .69 for the Sound Comparison subscale, alpha = .76 for the Phoneme Segmenting subscale, and alpha = .80 for the Phoneme Blending subscale, indicating adequate internal consistencies.

Convergent Validity

Comparison to CTOPP-2 Phonological Processing Skills

An omnibus one-way Analysis of Variance (ANOVA) test was conducted to determine if there was a difference in CTOPP-2 Phonological Processing Composite scores between students who were identified as being “At Risk” and “Not At Risk” on the Phonemic Awareness Composite of the RiFS-Reading. Results revealed a statistically significant difference in mean CTOPP-2 scores between the “At Risk” and “Not At Risk” groups. Students who were “At Risk” on the RiFS-Reading (M = 92.67, SD = 11.46) had significantly lower CTOPP-2 Phonological Processing Composite scores than students who were “Not At Risk” (M = 108.18, SD = 13.91) (F(1, 45) = 11.98, p = .001). The partial eta-squared was .21 and represents a large effect size. (See Table 4).

Table 4.

Mean CTOPP-2 Composite and subtest scores (Mean for Composite = 100, SD = 15; Mean for Subscale = 10, SD = 3) for students “Not At Risk” and “At Risk” on the RiFS-Reading Composite and subscales.

	Mean	Std. Deviation	Mean	Std Deviation	F	η²_p
	RiFS-Reading Phonemic Awareness Composite
	“Not at Risk”		“At Risk”
CTOPP-2 Composite	108.18	13.91	92.67	11.46	11.98^***	.21
	RiFS-Reading Phoneme Segmentation
	“Not at Risk”		“At Risk”
CTOPP-2 Elison	10.71	2.44	8.27	1.68	9.49^**	.18
	RiFS-Reading Sound Comparison
	“Not at Risk”		“At Risk”
CTOPP-2 Sound Matching/Phoneme Isolation	11.10	2.46	8.71	1.38	6.13^*	.12
	RiFS-Reading Phoneme Blending
	“Not at Risk”		“At Risk”
CTOPP-2 Blending Words	11.20	2.64	9.45	3.67	3.01	.06

Note. N = 46; CTOPP-2 = Comprehensive Test of Phonological Processing, Second Edition.

^**p < .01.

^***p ≤ .001.

Simple effects to examine the subscale components of these composite scores were then conducted to determine whether the subscales were measuring similar constructs based on the theoretical underpinnings of the scales. A series of three one-way ANOVAs were conducted to determine if students who were identified as being “At Risk” and “Not At Risk” on the individual subscales of the RiFS-Reading (Phoneme Segmentation, Sound Comparison, and Phoneme Blending) had significantly different scores on their corresponding CTOPP-2 subscales (Elison, Sound Matching/Phoneme Isolation, Blending Words).

The first of these one-way ANOVA tests was conducted to determine if there was a difference in CTOPP-2 Elison scores between students who were identified as being “At Risk” and “Not At Risk” for the Phoneme Segmentation subtest on the RiFS-Reading. Results revealed that there was a statistically significant difference in mean exam scores between the “At Risk” and “Not At Risk” groups. Students who were “At Risk” on the RiFS-Reading (M = 8.27, SD = 1.68) had significantly lower CTOPP-2 Elison scores than students who were “Not At Risk” (M = 10.71, SD = 2.44) (F(1, 45) = 9.49, p = .004). The partial eta-squared was .18 and represents a large effect size.

A second one-way ANOVA test was conducted to determine if there was a difference in CTOPP-2 Sound Matching/Phoneme Isolation scores between students who were identified as being “At Risk” and “Not At Risk” for Sound Comparison on the RiFS-Reading. Results revealed that there was a statistically significant difference in mean exam scores between the “At Risk” and “Not At Risk” groups, whereby students who were “At Risk” on the RiFS-Reading (M = 8.71, SD = 1.38) had significantly lower CTOPP-2 Sound Matching/Phoneme Isolation scores than students who were “Not At Risk” (M = 11.10, SD = 2.46) (F(1, 45) = 6.13, p = .017). The partial eta-squared was .12 and represents a medium effect size.

A third one-way ANOVA test was conducted to determine if there was a difference in CTOPP-2 Blending Words scores between students who were identified as being “At Risk” and “Not At Risk” for Phoneme Blending on the RiFS-Reading. Results revealed that there was not a statistically significant difference in mean exam scores between the “At Risk” (M = 9.45, SD = 3.67) and “Not At Risk” (M = 11.20, SD = 2.64) groups (F(1, 45) = 3.01, p = .064). The partial eta squared was .06 and represents a medium effect size.

Comparison to Reading Skills

A one-way ANOVA test was conducted to determine if performance on the Letter-Word Identification test of the WJA-IV differed between the RiFS-Reading Phonemic Awareness Composite “At Risk” and “Not At Risk” groups. Results revealed a statistically significant difference in mean reading scores between students who were identified as “At Risk” (M = 83.81, SD = 11.67) and “Not At Risk” (M = 97.62, SD = 17.21) using the RiFS-Reading Phonological Awareness Composite score (F(1, 44) = 6.11, p = .017). The partial eta-squared was .12 and represents a medium effect size.

Discriminant Validity

Comparison to Vocabulary Skills

One-way ANOVA tests were conducted to determine if performance on the WIAT-IV Receptive and Expressive Vocabulary tests differed between the RiFS-Reading Phonemic Awareness Composite “At Risk” and “Not At Risk” groups. Results revealed that there was not a significant difference between the receptive vocabulary scores of the RiFS-Reading “At Risk” group (M = 98.46, SD = 9.80) and “Not At Risk” group (M = 99.38, SD = 12.63) (F(1, 43) = .054, p = .82) nor between the expressive vocabulary scores of the RiFS-Reading “At Risk” group (M = 101.07, SD = 7.93) and “Not At Risk” group (M = 103.34, SD = 10.43) (F(1, 43) = .494, p = .49).

Discussion

Research has shown the importance of phonemic awareness in the development of early reading skills (Milankov et al., 2021). Children who possess weak phonemic awareness have trouble accurately sounding out words and have been shown to have trouble catching up if not addressed by third grade (Hernandez, 2011). The purpose of this study was to develop and validate the Risk Factor Screen for Reading (RiFS-Reading) as a tool to quickly identify students who are at risk for early reading difficulties. This differs from other assessments of phonemic awareness because it is web-based, can be group-administered in a short period of time, requires minimal training for the assessors, and provides electronically scored results immediately after completion.

To measure construct validity, exploratory factor analysis was conducted. This revealed a three-factor structure. This corresponds with the three factors considered to be the most important components of phonemic awareness as described by Torgeson (1999). Results revealed that this model accounts for over 60% of the variance, with items loading onto the factors corresponding with their respective subtests at levels of .69–.88, with the exception of two, one of which was .54 and the other which was .41. Although there were two items that cross-loaded onto other factors, their primary loadings were on their intended factors. Furthermore, results showed the RiFS-Reading had adequate reliability as demonstrated by tests of internal consistency of each of the subtests and the total RiFS-Reading Phonemic Awareness Composite scores that revealed alphas between .69 and .82. It is important to note that although many studies have found phonemic awareness tasks to be part of one unified construct, many others (e.g., David et al., 2022; Høien et al., 1995) suggest that the skill involves slightly different abilities, an idea supported by this study.

In order to assess the convergent validity of the RiFS-Reading, students’ scores on the RiFS-Reading were compared to their performance on the CTOPP-2, a well-validated, reliable measure of phonemic awareness. Students who were identified as “At Risk” on the RiFS-Reading were found to score significantly lower on the Phonological Processing Composite and subtests of the CTOPP-2 with the exception of one (i.e., Blending Words), which trended towards statistical significance (p = .06). The CTOPP-2 Composite scores of students who were identified as “At Risk” on the RiFS-Reading fell around the 30^th percentile, which is suggestive of weak skills for their age group. In addition, students who were identified as “At Risk” on the RiFS-Reading scored significantly lower than students who were “Not At Risk” and approximately at the 14^th percentile on the WJA-IV Letter-Word Identification test, a well-validated measure of reading. This indicates that the RiFS-Reading is predictive of phonemic awareness abilities as well as reading skills. This is consistent with previous research, which has shown that phonemic awareness is highly related to early reading abilities (Wagner & Torgesen, 1987). The combination of these results supports the convergent validity of the RiFS-Reading.

To ensure that the RiFS-Reading is a measure of phonemic awareness and not other language skills, scores on the RiFS-Reading were compared to scores on the WIAT-IV tests of expressive and receptive vocabulary. Results showed that the vocabulary scores of the “At Risk” and “Not At Risk” groups on the RiFS-Reading were not significantly different, suggesting that phonemic awareness as measured by the RiFS-Reading is a distinct factor in the development of reading skills.

Limitations of the Study

While the RiFS-Reading proved successful in quickly predicting the phonemic awareness of students, it is a screening tool that has some limitations. As RiFS-Reading is meant to be administered in 10–15 minutes, the phonemic awareness sections only include four questions, with two incorrect answers yielding an “At Risk” score. Also, as the participants were almost all from upper-middle class areas, nearly all were able to identify the letters and letter sounds, making it difficult to assess the efficacy of the Letter Recognition and Letter-Sound subtests due to limited variability in the students’ performance. It would therefore be beneficial to examine the validity of the Phonics subtests of the RiFS-Reading among students towards the beginning or middle of kindergarten and among students from a variety of backgrounds and educational experiences, as there is likely to be more variability in these groups to evaluate these factors as predictors of early reading success. Future research should also determine the utility of using the RiFS-Reading with students from more diverse and lower socioeconomic backgrounds, as they may have less experience with tablets, potentially making the navigation of the RiFS-Reading more difficult.

Other limitations include the sometimes unreliable WiFi access at schools. Also, while the RiFS-Reading allows for limited proctor training, some training is necessary for administrators to ensure that they are able to address any technical malfunctions. The length of the current test may also cause fatigue in students. This was evident in the Phoneme Blending subtest of the RiFS-Reading where the “At Risk” and “Not At Risk” groups did not have significantly different scores on the CTOPP-2 Blending Words subtest (although the result trended towards significance). As it is the last section, the Phoneme Blending subtest may be less reliable than the other subtests, as young students may be fatigued by this time, and without one-on-one administration, it can be difficult to monitor students’ attention to ensure they perform their best.

Conclusion

In summary, these results show that the RiFS-Reading is a reliable and valid assessment that can be used to identify students who are at risk for early reading problems. Students who exhibit reading problems in kindergarten and first grade run a high risk of never catching up to their peers. Educators acknowledge that deficits in phonics and phonemic awareness are important contributors to students’ reading performance; unfortunately, many schools do not screen for those skills in a meaningful way due to training requirements, the need for one-on-one administration, administration time, long wait times for results, and cost. Those considerations were central in the development of RiFS-Reading. Educators will be able to capitalize on the RiFS-Reading screener results to target students who need more practice in phonemic awareness (even 10–15 minutes three times a week for the entire school year), making the screening effort highly valuable in the context of early education.

Supplemental Material

Supplemental Material - Validation of the Risk Factor Screen for Reading (RiFS-Reading) Screening Tool for the Early Identification of Reading Problems

Supplemental Material for Validation of the Risk Factor Screen for Reading Screening Tool for the Early Identification of Reading Problems by Bennett Kuttler, and Elliot G. Levy in Journal of Education

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Bennett Kuttler

Supplemental Material

Supplemental material for this article is available online.

References

Baldick

(1990). The concise oxford dictionary of literary terms. Oxford University Press.

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quiñonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6(149), 149–218. https://doi.org/10.3389/fpubh.2018.00149

Breaux

K. C.

(2020). WIAT 4: Wechsler Individual Achievement Test: Manual. - Gordon College. Pearson Education, Inc.

Cattell

R. B.

(1966). The Scree test for the number of factors. Multivariate Behavioral Research, 1(2), 245–276. https://doi.org/10.1207/s15327906mbr0102_10

Catts

H. W.

Gillispie

Leonard

L. B.

Kail

R. V.

Miller

C. A.

(2002). The role of speed of processing, rapid naming, and phonological awareness in reading achievement. Journal of Learning Disabilities, 35(6), 509–524. https://doi.org/10.1177/00222194020350060301

Curriculum Associates . (2019). i-Ready efficacy: Research on i-Ready instruction program impact. (Curriculum Associates research report No. RR 2019-55). North Billerica, MA: Author.

David

Adrian

Moldovanu

A. E.

(2022). The factorial structure of the phonological awareness construct. A qualitative review. Education, Reflection, Development - ERD 2021 , vol. 2. European Proceedings of Educational Sciences (pp. 23–40).

Domingue

B. W.

Dell

Lang

Silverman

Yeatman

Hough

(2022). The effect of COVID on oral reading fluency during the 2020–2021 academic year. AERA Open, 8(1), 233285842211202. https://doi.org/10.1177/23328584221120254

Ehri

L. C.

Nunes

S. R.

Stahl

S. A.

Willows

D. M.

(2001). Systematic phonics instruction helps students learn to read: Evidence from the National reading panel's meta-analysis. Review of Educational Research, 71(3), 393–447. https://doi.org/10.3102/00346543071003393

10.

Francis

D. J.

Shaywitz

S. E.

Stuebing

K. K.

Shaywitz

B. A.

Fletcher

J. M.

(1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88(1), 3–17. https://doi.org/10.1037/0022-0663.88.1.3

11.

Gaab

N.,

Petscher

(2021). Early bird dyslexia and early literacy screener. Technical manual. EarlyBird Education. https://earlybirdeducation.com/earlybird-for-educators/

12.

Good

R. H.

Gruba

G. G.

Kaminski

R. A.

(2001). Best practices in using dynamic indicators of basic early literacy skills (DIBELS). In Thomas

Grimes

(Eds.), Best practices in school psychology IV (pp. 679–700). National Association of School Psychologists.

13.

Hamilton

S. S.,

Glascoe

F. P.

(2006). Evaluation of children with reading difficulties. American Family Physician, 74(12), 2079–2084.

14.

Hernandez

D. J.

(2011). Double Jeopardy: How third-grade reading skills and Poverty Influence high school Graduation. Annie E. Casey Foundation. https://www.aecf.org

15.

Høien

Lundberg

Stanovich

K. E.

Bjaalid

I. K.

(1995). Components of phonological awareness. Reading and Writing, 7(2), 171–188. https://doi.org/10.1007/bf01027184

16.

Kilpatrick

(2016). Equipped for reading success: A comprehensive guide step-by-step program for developing phoneme awareness and fluent word recognition. Casey & Kirsch.

17.

Lyon

(1998, March). Why reading is not a natural process. Educational Leadership, 55(6), 14–18.

18.

Lyon

G. R.

(1996). Learning disabilities. The Future of Children, 6(1), 54–76. https://doi.org/10.2307/1602494

19.

Milankov

Golubović

Krstić

Golubović

Š.

(2021). Phonological awareness as the foundation of reading acquisition in students reading in transparent orthography. International Journal of Environmental Research and Public Health, 18(10), 5440. https://doi.org/10.3390/ijerph18105440

20.

Northwest Evaluation Association . (2016). Linking the Kentucky K-PREP assessments to NWEA MAP tests. Northwest Evaluation Association. https://www.nwea.org/about/

21.

Schrank

F. A.

Mather

McGrew

K. S.

(2014). Woodcock-johnson IV: Tests of Achievement. Riverside Publishing Company.

22.

Siegelman

Rueckl

J. G.

van den Bunt

Frijters

J. C.

Zevin

J. D.

Lovett

M. W.

Seidenberg

M. S.

Pugh

K. R.

Morris

R. D.

(2022). How you read affects what you gain: Individual differences in the functional organization of the reading system predict intervention gains in children with reading disabilities. Journal of Educational Psychology, 114(4), 855–869. https://doi.org/10.1037/edu0000672

23.

Torgesen

J. K.

(1999). Assessment and instruction for phonemic awareness and word reading skills. In Catts

H. W.

Kamhi

A. G.

(Eds.), Language Basis of reading disabilities. Allyn & Bacon.

24.

Torgesen

J. K.

Wagner

R. K.

Rashotte

C. A.

Herron

Lindamood

(2010). Computer-assisted instruction to prevent early reading difficulties in students at risk for dyslexia: Outcomes from two instructional approaches. Annals of Dyslexia, 60(1), 40–56. https://doi.org/10.1007/s11881-009-0032-y

25.

Wagner

R. K.,

Torgesen

J. K.

(1987). The nature of phonological processing and its causal role in the acquisition of reading skills. Psychological Bulletin, 101(2), 192–212. https://doi.org/10.1037/0033-2909.101.2.192

26.

Wagner

R. K.

Torgesen

J. K.

Rashotte

C. A.

Pearson

N. A.

(2013). Comprehensive test of phonological processing–2nd ed. (CTOPP-2). Pro-Ed.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.57 MB