Developing and Validating a Tool to Assess Young Children’s Early Literacy Engagement

Abstract

There is growing recognition that literacy learning takes place in the years prior to formal schooling and that young children develop literacy-like behaviours through exposure to interactions in shared contexts in which literacy is a component. Despite this, there are few assessments that measure the very early literacy skills that children develop before 36 months of age. This article reports on the design and validation of a new instrument – the Early Literacy Engagement Assessment (ELEA). This tool was developed to provide insights into the impact of Conversational Reading, a key pedagogical strategy implemented at Families as First Teachers playgroups, on young children’s early receptive and expressive vocabulary and literacy skills. The instrument was trialled with 104 children living in locations across Melbourne, Victoria, and 39 Aboriginal children living in remote communities in the Northern Territory. The trial process was undertaken in two phases: (1) a technical assessment to test item consistency, characteristics and placement and (2) concurrent validity testing against items from the Clinical Evaluation of Language Fundamentals Preschool-2 tool. The findings from the trial and validation process indicate that overall the ELEA discriminates well between children of high and low ability, and it is a useful tool in the authentic assessment of expressive and receptive vocabulary skills in young children.

Keywords

Early language and literacy assessment early childhood education

The home and early childhood education and care (ECEC) settings are important contexts in which children can be exposed to a world of texts, print and images from a very young age. Mediated by adults, children’s early interactions with texts ensure they build symbolic literacy and meaning-making skills to develop an increased awareness of the relationships between oral and visual representations (Department of Education and Training, 2016; Raban, 2014). Further, providing young children with frequent and intentional language-focused interactions is key to building strong literacy foundations in the years prior to school (Dickinson et al., 2012). This premise is grounded in the ‘ineradicable and critically important idea that literacy grows out of and alongside language development’ (Lo Bianco, 2016, p. v).

The Early Literacy Engagement Assessment (ELEA) tool described in this article was developed as part of a larger Australian Research Council (ARC) study, ‘Building a Bridge Into Preschool in Remote Northern Territory Communities’ (Page et al., 2019). This study explored whether the Abecedarian Approach Australia (3a), an educational program comprised of a set of language-rich teaching strategies (Language Priority, Enriched Caregiving, Conversational Reading and LearningGames®) with strong cultural adaptations, can advance young Aboriginal children’s early learning prior to attending preschool (for a detailed discussion of the 3a strategies, see Cooke & Piers-Blundell, 2019; Page et al., 2019). The underlying premise of the research was that the most optimal outcomes for Aboriginal children will occur when a ‘learning bridge’ for children from birth to 3 years of age is created – supported at one end by the strength and content of local culture and at the other end by proven learning techniques from research and practice in early childhood education. The Abecedarian teaching and learning strategies were adopted as the educational program for Families as First Teachers (FaFT) playgroups in the Northern Territory on account of the strong evidence base generated by studies in the United States (Campbell et al., 2012; Gonski et al., 2018). In recognition of the importance of adapting early childhood educational program materials to improve cultural responsivity, the 3a LearningGames® were modified in consultation with Aboriginal families and staff at FaFT playgroups prior to the start of this study (Cooke & Piers-Blundell, 2019). Building on this work, a key outcome of the ARC study was the development of a culturally sensitive instrument that could be administered in English and local languages to assess young children’s early literacy skills as they engage with text to create meaning, as well as their receptive and expressive vocabulary skills.

Assessment in early childhood

Assessment in early childhood frequently involves educators observing, recording and documenting children’s learning (Tayler & Ishimine, 2013). Such sensitive observations of children within naturally occurring contexts of learning highlight the need for well-informed assessment tools and analysis frames (Clay, 2019). Ideally, these will support the systematic observation of children across a variety of authentic tasks that capture a wide range of skills beyond a single test on an isolated occasion. Moreover, any assessment undertaken in the early years should also reflect the curriculum that is intended or in place in order to assess children’s performance based on what they have had the opportunity to learn (Gee, 2003). Hence, our focus in this research was to co-develop an assessment that aligned with the shared book-reading practices that were both a regular, everyday learning activity at FaFT and one that could be used to assess young children’s receptive and expressive vocabulary and early literacy skills over time, with the information gleaned immediately transferable to teaching practice.

Joint attention and shared book reading

Shared reading with young children is a social practice. At the core of this activity, supporting children’s engagement with text is joint attention. Joint attention involves individual children coordinating their attention with a social partner via an object or image and the process of following a partner’s direction of gaze or drawing the young child’s attention away from an image or object to attend to another (Frischen et al., 2012). Through the process of joint attention, adults label objects that are referenced by children, and in this way children can begin to build their lexical knowledge (Tomasello & Todd, 1983). However, there is a cautionary aspect to labelling during joint attention. Silvén et al. (2003) noted that the simple act of labelling objects may result in children interacting less and using fewer multi-word utterances and therefore suggest that as young children’s language skills increase, adults should begin to use other techniques to engage them in language interactions. Educators and parents should be encouraged to expand on children’s comments, restate them and use a variety of questions as they create a conversation or dialogue about the book.

Dickinson et al. (2012) report that book reading and children’s active engagement in talk about text fosters oral language development, and when integrated, these activities create opportunities for children to both hear and produce new forms of language. Recent research reports the heightened impact of shared reading conversations that occur about, around and beyond the text, exposing children to new words and concepts (Gonzalez et al., 2014). However, the instruction around shared book reading matters, and this includes extratextual conversations, encouraging children to use words in new contexts that promote the construction of meaning outside their own experiences (Massey et al., 2008). Further, interactivity during shared reading is important, as this ‘both engages the child and supports producing language at the appropriate linguistic level, while conceptual challenge (supported for example by the content of appropriate books) helps children develop the more sophisticated cognitive and linguistic skills they will need in school’ (Rowe & Snow, 2019, p.13). High levels of engagement in shared reading with adults are considered necessary to maximise the positive impact on children’s language outcomes (Noble et al., 2019).

Some aspects of oral language are clearly more related to literacy outcomes than others. The report by the National Early Literacy Panel, 2009, Developing Early Literacy, notes ‘complex aspects of oral language, such as grammar, definitional vocabulary, and listening comprehension, as having more substantial predictive relations with later conventional literacy skills’ (p. 78). Language-rich conversations, such as those afforded by interactive shared book reading, are identified as building the language resources facilitative of children’s literacy learning; with extended, decontextualized language positively associated with later measures of literacy (Dickinson & Tabors, 2001). Longitudinal research evidence suggests that preschool children’s exposure to storybook reading predicts vocabulary development and comprehension skills at the beginning of Grade 1, which in turn predicts reading comprehension at the end of Grade 3 (Sénéchal & LeFevre, 2002). Yet there remains wide variation in children’s language use and opportunities for literacy learning (Hart & Risley, 2003; Hoff, 2006; Kidd et al., 2018). Aboriginal children living in remote communities are considerably more likely than their non-Aboriginal counterparts to be developmentally vulnerable, particularly in the language and cognitive skills domains (Australian Early Development Census, 2019) and that the gap in literacy learning widens as Aboriginal children continue throughout the stages of schooling (Australian Curriculum, Assessment and Reporting Authority [ACARA], 2018). Working with families and community members to understand more deeply how young Aboriginal children’s first languages and English are taught, supported and assessed in education programs is critical to understanding how best to address this gap. Integrating Aboriginal ways of doing, being and knowing (Lee, 2019; Martin, 2005) has been central to the adaptation of the Abecedarian Approach Australia (3a), proven to advance the personal, social and academic achievements of very young children (Campbell et al., 1998; Campbell et al., 2012).

Conversational reading and the Early Literary Engagement Assessment

Joint interaction and language-rich book reading are central tenets of Conversational Reading, with a number of studies both in Australia and internationally attesting to the efficacy of this approach (e.g. see Brookes & Tayler, 2016; Cooke, 2013; Ramey et al., 2012). Conversational Reading focuses on the shared, reciprocal interaction of an adult reading a book with a child as an active participant. Using the 3S strategy – ‘See, Show, Say’ – the adult engages the child in a series of back-and-forth interactions to participate in the reading session (Sparling & Meunier, 2019). At the ‘See’ level, the adult follows the child’s lead by engaging verbally, acknowledging what the child is looking at or touching – ‘You are looking at the beetle’. Then, the adult, positioned so they can see the child’s head and eye movements, assists the child to follow her lead by pointing to and naming a picture the child is not looking at and checks to see whether the child’s eyes are following – ‘I can see the crab’. When children are more experienced in the ‘See’ step, the adult can just name the picture that the child is not looking at and check to see whether the child’s eyes are following. This level of Conversational Reading builds young children’s receptive language. At the ‘Show’ level, the adult prompts the child to respond in a way that requires a physical gesture, for example, ‘Point to the ladybird’ or ‘Show me the grasshopper’. This level of Conversational Reading assists the adult to prompt the child’s receptive language. The ‘Say’ level involves the adult using questioning to encourage the child to provide an extended verbal response (e.g. ‘What will the grasshopper do next?’). This level of Conversational Reading provides opportunities for utterances that include more complex, integrated language structures (Scull et al., 2013). Print concepts were also included in the ELEA, with children asked to point to the writing and to indicate where the reader might start reading.

Conversational Reading underpins the development of the ELEA. Through children’s participation in Conversational Reading, the ELEA was designed to assess children’s early literacy and receptive and expressive vocabulary skills in a familiar context. The tool is administered as children participate in a reading interaction using a book called Lookout! Here comes Cheeky! (James et al., 2015). The text and illustrations were designed to facilitate engagement and prompt children to respond to instructions and questions, modelled around the 3S strategy, either by looking, gesturing, speaking or turning pages. To ensure the cultural and linguistic appropriateness of the text, feedback was sought from Aboriginal Family Liaison Officers and playgroup staff as the text was developed and trialled. The development of a new text was intended to contribute to the reliability of the tool; ‘taken together, a standard task, with standard administration and with standard scoring procedures provides sound measurement conditions’ (Clay, 2019, p. 12).

The analysis reported below considers whether such an instrument can be administered with fidelity with Aboriginal children (phases 1 and 2) and with children experiencing socio-economic disadvantage (phase 2). Phase 2 considers whether the tool can be used to reliably assess young children’s receptive and expressive vocabulary skills.

Research aims

This research study comprises two phases. In phase 1, we examined the properties of the ELEA. The aim of this phase of the study was to ascertain whether the range of items captured children’s experiences with Conversational Reading, whether the items were of an appropriate level of difficulty and whether the items were able to account for a range of achievement levels for children aged 2–4 years, with the actual age of children participating in both studies ranging from 1.8 to 4.6 years. The aim of phase 2 was to test the concurrent and convergent validity of the ELEA alongside another standardised language assessment, the Clinical Evaluation of Language Fundamentals Preschool-2 (CELF P-2; Wigg et al., 2006). Specifically, we were interested in how well the ELEA tool compared to a well-established test of young children’s receptive and expressive vocabulary and the degree to which the two measures were related. The CELF P-2 test measures a broad range of language skills in young children. It provides a comprehensive range of seven norm-referenced subtests. Two of these subtests – ‘Basic Concepts’ and ‘Expressive Vocabulary’ – were used for the purpose of validating the ELEA. The Basic Concepts subtest, as a measure of receptive vocabulary, was matched to the ‘See’ and ‘Show’ aspects of Conversational Reading, and the Expressive Vocabulary subtest was matched to the ‘Say’ aspects of Conversational Reading. The research was performed with approval from the University’s Human Research Ethics Committees and with permission from the participating state government and educational institutions. Parental consent was obtained prior to the participation of children in the study, and child assent obtained before the testing commenced.

Phase 1

Participants

Early childhood centres where 3a strategies were regularly used were invited to engage in this study so that children participating were familiar with Conversational Reading procedures. Early childhood centres in the Northern Territory and Victoria participated in this phase of the study. The Northern Territory cohort consisted of 23 Aboriginal children, aged from 1.8 years to 3.3 years, who attended FaFT programs in the two remote communities engaged in the broader ARC-funded research study. The Aboriginal children were all dual language speakers; there were 13 different languages spoken by the children in one community, and the most common language in the other community was Yolŋu Matha (Djambarpuyngu) (Page et al., 2019). The Victorian cohort comprised 83 non-Aboriginal children, aged from 1.8 years to 4.2 years, who attended two preschools in Melbourne, Victoria. Of the Victorian cohort, three children were from non-English-speaking backgrounds, with all children able to complete the task in English.

Procedure

The task was administered by FaFT staff or research assistants trained in Conversational Reading. Reliability was achieved through training procedures to ensure consistency of the ELEA administration, with any apparent differences discussed to reach consensus (Bradley et al., 2007). Each child was read the text Lookout! Here comes Cheeky! once only, during which they were prompted to identify, by looking and pointing, and to label pictures and words in the text and comment on the story. The administration of the task ranged from 10 to 15 minutes and was undertaken within the preschool setting. All administrators were familiar with the children and able to assess if the children were tired or lacked interest in the task; on such occasions, testing was discontinued and resumed at another time. In the Northern Territory, Aboriginal staff engaged in the process of administering the assessment, presented in both English and in first language(s), with the language(s) used for questions and answers recorded. This aligns with our advocacy for dual language use, recognising the important role first language maintenance plays in children’s language learning as well as in the maintenance of culture (Clarke, 2009).

Data analysis

The percentage of correct answers, item discrimination, weighted mean square, Rasch analysis and an item–person map were used to analyse the data. The percentage of correct answers was calculated based on the children’s raw test scores. Here, it should be noted that items 14 to 17 are partial credit items, so the averaged percentages were reported. Item discriminations were undertaken to measure of the relationship between the score on an item and the overall test score (Wu & Adams, 2007). The correlation between the item score and the total score of the remaining items in the test is presented. An item discrimination index smaller than 0.3 is considered low, and as a rule of thumb, these items do not discriminate between children who are highly engaged and children who are not engaged (Wu et al., 2016). Weighted mean squares were calculated to show how well items are in agreement with others to measure a single trait underlying the items. The weighted mean square fit statistics take account of extreme cases for each item, and the mean square fit statistics should be close to 1 to show a good fit to the Rasch model (Nguyen & Griffin, 2007). Rasch item difficulty is defined in terms of the ability level needed to obtain a 50% chance of getting the item correct (Wu & Adams, 2007). A Thurstonian threshold (also referred to as item threshold in this article) for a category is the point on the scale at which the probability of responding in that category or higher reaches .50. This is the point on the continuum where the skills described by a category are likely to be attained, since beyond that point children have greater than 50% chance of reaching that category of performance. One of the important features of the Rasch model is the ability to place person ability and item difficulty estimates on the same scale (Wright & Masters, 1982) and to ascertain whether the items work together to define a single construct (Wright & Masters, 1982). In this study, Rasch modelling was used to determine how well the items defined an increasing sophistication in the construct of Conversational Reading and how well the items targeted children’s abilities and discriminated between children with high and low ability (Nguyen & Griffin, 2013). An item–person map was also produced by Conquest (Adams et al., 2015) to check whether the items covered the range of children’s ability and whether the range of item difficulty was comparable to the range of children’s abilities.

Phase 1 results

The demographic characteristics of the children involved in this phase of the study are described in Table 1; 78% of children were from Victoria while 22% of the sample were from the Northern Territory. Children from Victoria (M = 3.4, SD = 0.4) were older than children from the Northern Territory (M = 3.1, SD = ±0.3; p < .05). The number of male students (43) was lower than that of female students (63).

Table 1.

Phase 1 participant demographics.

	Northern Territory	Victoria
Number of children	23	83
Male	10	33
Female	13	50
Age (years)	M = 3.1, SD = 0.3	M = 3.4, SD = 0.4

Across the two cohorts, there was a range of total scores for the ELEA, confirming that the tool captures a broad range of young children’s abilities. The mean score for children in Victoria (M = 19.2, SD = 4.4; p < .01) was higher than that for children in the Northern Territory (M = 13.1, SD = 4.4) for all three domains: ‘See’ (M = 4.0, SD = .1 vs. M = 3.7, SD = 0.6; p < .01), ‘Show’ (M = 8.8, SD = 1.9 vs. M = 6.3, SD = 3.0; p < .01) and ‘Say’ (M = 6.4, SD = 2.9 vs. M = 3.1, SD = 2.1; p < .01). There was no significant difference between female and male students for the total ELEA score (M = 18.3, SD = 5.1 vs. M = 17.3, SD = 5.0; p = .8) or any of the three domain scores for ‘See’, ‘Show’ and ‘Say’.

Children’s results for three age groups were compared (younger than 3 years old, from 3 to 3.5 years old and older than 3.5 years old). This analysis showed an increasing trend in children’s performance from the youngest to the oldest group. When the age groups were compared, older children outperformed the younger children, with higher total ELEA scores and higher ‘Show’ and ‘Say’ domain scores. For the ‘See’ domain of the ELEA, no difference was observed in relation to children’s age.

Item characteristics are presented in Table 2. Column 1 of Table 2 presents the test domains. Column 2 presents the item/question number in the test. Column 3 presents the percentages of correct answer (score 1). There are four items (items 14–17) where children were given partial credit scores ranging from 0 to 3. The average percentages of children with correct scores for each of the subquestions of these items are also presented in column 3 with an asterisk. Column 4 presents correlations between item scores and the total scores of the remaining items in the test. Columns 5–7 present weighted mean square fit, their interval confidence and t values. Column 8 presents item difficulty in logit scores.

Table 2.

Item characteristics.


(1)	Item (2)	Percentage of correct answers (3)	Item discrimination index item-test correlation (4)	Weighted mean square (5)	Confidence interval (6)	T (7)	Item difficulty (logit score) (8)
See	1	97.2	.03	1.37	0.03	1.97	0.8	−2.56
See	2	97.2	.20	1.23	0.03	1.97	0.6	−2.56
See	3	98.1	.11	1.26	0	2.22	0.6	−3.02
See	4	99.1	.06	1.16	0	2.84	0.5	−3.80
Show	5	93.4	.51	0.74	0.37	1.63	−0.8	−1.47
Show	6	95.3	.40	0.85	0.26	1.74	−0.3	−1.92
Show	7	83.0	.55	0.93	0.63	1.37	−0.3	−0.06
Show	8	92.5	.50	0.77	0.41	1.59	−0.8	−1.28
Show	9	88.7	.37	1.10	0.52	1.48	0.5	−0.69
Show	10	59.4	.57	0.89	0.8	1.2	−1.1	1.61
Show	11	21.7	.46	0.90	0.72	1.28	−0.7	3.89
Show	12	84.0	.55	0.90	0.62	1.38	−0.5	−0.15
Show	13	81.1	.64	0.78	0.66	1.34	−1.3	0.12
Say	14	42.2∗	.56	1.15	0.75	1.25	1.1	2.74
Say	15	54.7∗	.64	1.09	0.74	1.26	0.7	1.93
Say	16	42.5∗	.65	0.94	0.74	1.26	−0.4	2.69
Say	17	50.3∗	.61	1.14	0.74	1.26	1.1	2.10
Show	18	82.1	.38	1.06	0.65	1.35	0.4	0.03
Show	19	45.3	.28	1.25	0.8	1.2	2.4	2.40

*Children given partial credit scores ranging from 0 to 3 for thes items.

In Table 2, items 1, 2, 3, 4 and 19 have a low discrimination index, and items 1, 2, 3, 4 and 6 were recorded as easy items (more than 95% correct). Given that the test may be used with children with a range of receptive and expressive language abilities and with children from diverse linguistic communities, test items covering a wide range of ability levels are appropriate. Items 10 and 11 assess children’s early understandings of print, with only 59.4% and 21.7% of children providing correct responses to these items.

The mean square fit statistics of all 19 items are within the confidence interval, which means that these items fit the Rasch model, and there is evidence of a single construct underlying the set of items. Item and person reliability are .96 and .83, respectively, which is considered high. A high item reliability demonstrates that items in a test are well separated to define a direction and meaning of a variable. A high person reliability indicates that the children within the sample are well separated or that the measurement errors are so small that there is no overlap between the estimates (Nguyen & Griffin, 2013).

The analysis of item difficulty and child abilities was also assessed using an item–person map, which shows a range of child engagement levels (see Appendix). ‘Say’ items appeared to be the most difficult (having the highest logit value) while items 1, 2, 3 and 4 appeared to be the easiest items (‘See’ items). In addition, there were fewer items at the top of the scale (‘Say’ items). The analysis suggested the assessment of a child’s ability around this level would be more accurate if there were additional items at this higher level of difficulty.

Phase 1 summary

The findings indicate the ELEA has good psychometrics properties, with evidence of strong construct validity. However, the analysis indicated the ELEA could be improved with the addition of more difficult items. Following the results from phase 1, the ELEA was modified and an additional ‘Say’ item was included to increase the level of task difficulty. The revised version of the tool was used for phase 2.

Phase 2

Participants

Participants for this phase of the study (n = 37) were recruited from two early childhood centres where Conversational Reading strategies were implemented. The Aboriginal children (n = 16) attended a centre in a third remote community (not within the funded ARC study) in the Northern Territory (aged 3.2–4.6 years) and 21 non-Aboriginal children (aged 2.3–4.1 years) attended a preschool in Melbourne, Victoria. The Aboriginal children were all dual language speakers, with the majority speaking Kunwinjku as their first language, with Kriol also commonly spoken alongside English. The Victorian children were all from English-speaking backgrounds. In phase 2, the centres selected served children experiencing disadvantage, including Aboriginal children living in a remote community and children from a low SES community in an outer suburb of Melbourne.

Procedure

The administration of the tests for this phase of study was similar to that of phase 1, with training and consensus moderation processes in place to support reliability. The ELEA and two CELF P-2 subtests were administered by staff familiar with the children at each centre, under the supervision of a speech pathologist. Practice items were used to familiarise children with the process, and, where possible, all tests were completed on the same or consecutive days. For this phase of the study, the tests were administered to the Aboriginal children in English and local languages, with community language speakers supporting the children with translations as necessary, and the language(s) used for questions and answers recorded.

Data analysis

Children’s results from the ELEA were compared to results from the CELF P-2 subtests. Analyses included an independent t test for the age and gender of children. Concurrent validity was determined by comparing the score on the instrument of interest (ELEA) with the score on a reference standard (CELF P-2). Bivariate Pearson’s correlations were performed to measure the linear relationships between the ELEA (‘See’, ‘Show’ and ‘Say’ components) and the CELF P-2. The CELF Basic Concepts subtest assesses receptive vocabulary and was matched with ‘See’ and ‘Show’ items from the ELEA. The CELF Expressive Vocabulary subtest was matched with ‘Say’ items from the ELEA. Analysis was performed with the participant groups together and individually. The strength of the correlation was considered weak if r was between .1–.3, moderate if r was between .3–.5 and strong if r was greater than .5 (Cohen, 1988).

Phase 2 results

The demographic characteristics of the children involved in this study are described in Table 3. Children from Victoria (M = 3.2, SD = 0.5 years) were younger (p < .05) than children from the Northern Territory (M = 4.1, SD = 0.5 years). There were more female than male children in the study.

Table 3.

Phase 2 participant demographics.

	Combined schools	Northern Territory	Victoria
Number of children	37	16	21
Male	16	7	9
Female	21	9	12
Age (years)	M = 3.6, SD = 0.7	M = 4.1, SD = 0.5	M = 3.2, SD = 0.5

Table 4 summarises the data in terms of the number of correctly answered questions for 20 questions on the ELEA for children in the Northern Territory and Victoria and for both schools combined. Questions 1–4 comprise the ‘See’ component, questions 5–13 and 19–20 comprise the ‘Show’ component and questions 14–18 comprise the ‘Say’ component. When the data for Northern Territory and Victoria are combined, items 2, 3, 4, 5, 6 and 8 were recorded as easy items (more than 95% correct), while items 11 and 18 were deemed as difficult, with the percentage of correct responses at less than 5% and 25%, respectively (Table 4). The percentage of correct answers was high for both groups for items 1–6. For items 7–20, there appears to be no consistent trend in the percentage of correct answers between the two groups.

Table 4.

ELEA test results.

		n = 37		n = 16		n = 21
	Item	# correct	% correct	# correct	% correct	# correct	% correct
		All	All	NT (C)	NT (C)	VIC	VIC
See	1	35	94.6	14	87.5	21	100.0
See	2	36	97.3	15	93.8	21	100.0
See	3	36	97.3	15	93.8	21	100.0
See	4	37	100.0	16	100.0	21	100.0
Show	5	37	100.0	16	100.0	21	100.0
Show	6	36	97.3	15	93.8	21	100.0
Show	7	30	81.1	10	62.5	20	95.2
Show	8	35	94.6	15	93.8	20	95.2
Show	9	29	78.4	15	93.8	14	66.7
Show	10	14	37.8	9	56.3	5	23.8
Show	11	1	2.7	0	0.0	1	4.8
Show	12	33	89.2	13	81.3	20	95.2
Show	13	29	78.4	8	50.0	21	100.0
Say	14	16	43.2	9	56.3	7	33.3
Say	15	23	62.2	12	75.0	11	52.4
Say	16	20	54.1	6	37.5	14	66.7
Say	17	20	54.1	9	56.3	11	52.4
Say	18	9	24.3	7	43.8	2	9.5
Show	19	32	86.5	16	100.0	16	76.2
Show	20	23	62.2	15	93.8	8	38.1

Note: ELEA = Early Literacy Engagement Assessment; NT = Northern Territory; VIC = Victoria.

When the data from the Northern Territory and Victoria are combined, there is a strong correlation with a positive linear relationship (r = .58, p < .001) between ELEA (total score) and CELF P-2 (total scores for Basics Concepts and Expressive Vocabulary subtests; see Figure 1). The strong positive relationship was also observed for the two early childhood centres individually (Northern Territory: r = .66, p < .06; Victoria: r = .67, p < .001). When the data from both centres are combined, there is a strong correlation with a positive linear relationship (r = .53, p < .001) between ELEA scores (‘See’ and ‘Show’) and CELF P-2 scores (Basics Concepts); see Figure 2. The strong positive relationship was also observed for the centres individually (Northern Territory: r = .69, p < .003; Victoria: r = .53, p < .01). When the data are combined, there is a moderate correlation with a positive linear relationship (r = .46, p < .004) between ELEA ‘Say’ scores and CELF Expressive Vocabulary scores; see Figure 3. A strong positive relationship was also observed for the Victorian centre (r = .63, p < .002), while the moderate positive relationship was not significant for the Northern Territory centre (r = .39, p < .13).

Figure 1.

ELEA (total) and CELF (Basic Concepts and Expressive Vocabulary), Northern Territory and Victoria. Note: ELEA = Early Literacy Engagement Assessment; CELF = Clinical Evaluation of Language Fundamentals.

Figure 2.

ELEA (‘See’ and ‘Show’) and CELF (Basic Concepts), Northern Territory and Victoria. Note: ELEA = Early Literacy Engagement Assessment; CELF = Clinical Evaluation of Language Fundamentals.

Figure 3.

ELEA (‘Say’) and CELF (Expressive Vocabulary), Northern Territory and Victoria. Note: ELEA = Early Literacy Engagement Assessment; CELF = Clinical Evaluation of Language Fundamentals.

Phase 2 summary

In phase 2 of the study, we examined the performance of 37 children experiencing disadvantage from two early childhood centres on the ELEA compared with two subtests from the CELF P-2 assessment. Analysis of the data combined and at both centres independently showed the overall correlation between the two measures was strong and statistically significant. The results for all data show the ELEA test is valid for the components ‘See’, ‘Show’ and ‘Say’ combined, ‘See’ and ‘Show’ combined and ‘Say’ independently.

Discussion

Construct and concurrent validity

The results of the two phases of this study provide clear evidence of the validity of ELEA as a measure of early reading engagement and of children’s receptive and expressive vocabulary skills. The results of these studies indicate that the ELEA can be used with confidence to assess children’s attainment across the three domains of Conversational Reading – ‘See’, ‘Show’ and ‘Say’. Construct, including concurrent and convergent validity, was affirmed, as the results show the ELEA provides a clear measure of the three domains ‘See’, ‘Show’ and ‘Say’, and shows a range of item difficulty levels that take account of children’s background, age and gender. That said, after completing phase 1, an additional item was added to increase the difficulty level and capture a broader range of achievement levels. While overall the Northern Territory cohort were younger and their scores were lower, their performance mapped well to the test items, indicating that the ELEA was able to capture young Aboriginal children’s engagement with picture book texts. Furthermore, the use of known animals as characters, the simple, repetitive plot progression and clear illustrations makes the text accessible to a wide range of young listeners (Hearne & Stevenson, 2000).

The ELEA also maps well to subtests of the CELF P-2 test that measure children’s receptive and expressive vocabulary and hence has concurrent validity as an assessment of young children’s language skills in these specific areas. When comparing the two cohorts, the correlations were similar for the ELEA, indicating the assessment tool is valid with both Aboriginal and non-Aboriginal children. Only the ‘Say’ component of the ELEA was different between the two cohorts, with scores for the children from Victoria showing a strong and significant correlation, while the score for the Northern Territory children showed a moderate and nonsignificant correlation, indicating that English as a second or third language for young Indigenous learners may have an impact on children’s early expressive language development. This finding is consistent with studies of bilingual children that report rates of language acquisition are impacted by dual language input, with children’s language skills a function of exposure, dependent on how much of any one language the bilingual child hears (Hoff et al., 2012; Mancilla-Martinez & Lesaux, 2011). While beyond the scope of this study, these results can be balanced against the later positive literacy outcomes of bilingual teaching programs (Molyneux et al., 2016) and the recognised benefits of additive approaches to language teaching for bilingual children (Cummins, 2017).

Authentic assessment

The findings from the statistical analysis lend support to the ELEA as contributing to effective assessment practices, assisting educators to collect and analyse data to inform their planning (Department of Education and Training, 2016). However, no one technique is sufficiently reliable on its own, and assessment methods have been widely critiqued (Grolig, 2020). As such, educators need to make informed decisions as they assess young learners, taking into account the social dimensions of language and literacy to be assessed, the selection of assessment tools and the way these data are used to inform teaching and support students’ learning (Johnston & Costello, 2005).

Complementing the range of assessment tools and techniques available, we now propose the use of the ELEA as form of authentic and ethical assessment (Gee, 2003). In the first instance, the tool is authentic as it was developed in consultation with FaFT staff and families in each of the research communities and with Aboriginal colleagues involved in each stage of the design. Through engaging in Dadirri as deep listening (Atkinson, 2002), FaFT staff and the participant families’ knowledge and sensitivities influenced the style of the narrative, the characters selected, the title of the text and the administration of the task. Test items and the scoring of responses were also checked to ensure the tool was culturally appropriate, with cultural knowledge essential to assist with translations when the tool was delivered in local Indigenous languages. Through community involvement, the tool speaks to and reflects community aspirations for their children and supports families and educators to build children’s engagement in book reading in a meaningful and supportive way. These views were expressed by Aboriginal community members who participated in Yarning Circles (Lee, 2019), where they demonstrated an appreciation of both cultural and Western ways of teaching and learning:

And our kids are [all need] learn both ways being together, Western way and Indigenous way.

You know while we’re lucky we’ve got FaFT here and, communities and you know family engaging, through our culture, knowledge and Western society comes together.

The ELEA is also an authentic assessment tool in that it provides a clear way to monitor and track young children’s interactions in shared reading, a known and practised strategy embedded in early educational programs (Gonzalez et al., 2014). This shifts the emphasis from isolated tests of language and school readiness devoid of context to tools that draw on familiar, authentic classroom practices (Clay, 2019). The tool goes beyond the type of book reading where the infant or child is a passive listener and creates opportunities for increased understanding of their capabilities and increased engagement in text reading. Furthermore, it is reasonable and ethical that when we assess children, we do so after they have engaged in experiences that build a repertoire of relevant practice and competence (Gee, 2003). Aspects of shared reading, particularly labelling and adult–child interactions that extend children’s receptive and expressive vocabulary skills, can be assessed using the ELEA. This assessment tool is in many ways unique in its focus on young children and adult–child interactions, with text that fosters engagement with meaning and attention to print.

Assessment to guide teaching

Assessment is integral to effective teaching and is deeply connected to the ways early childhood educators guide and refine their teaching interactions (Department of Education and Training, 2016; McLachlan et al., 2013). The careful observation of young children as they complete the ELEA can provide educators with information to ‘design interactions, which purposefully challenge, scaffold, and extend children’s skills’ (Pianta, 2003, p. 5). A tool of this nature can support educators to build evidence of individual children’s skills and to plan for and assess their engagement with text reading in a more focused and intentional way. Importantly, this tool aligns with standard book-reading practices within many services and in so doing provides a systematic approach to assessment within the naturally occurring context of the curriculum. Adult–child conversations during reading interactions, as a formative assessment practice, enable educators to attend to aspects of joint attention and text engagement that young children can control, while identifying the aspects of early reading at the edge of the children’s level of development that with further support might show improvement (Wiliam, 2013). By drawing on relevant assessment data, educators are able to increase the benefits to children ‘by engaging in continuous supported experiences and tasks that enhance learning and development’ (Tayler & Ishimine, 2013, p. 299).

Limitations

The research reported in this article does have some limitations. More nuanced findings may have been achieved had the sample size for both phases of the research been increased, with greater numbers of Aboriginal children and children from linguistically and culturally diverse backgrounds included. Further, there are limitations to using single forms of assessment, especially with children from diverse cultural and linguistic backgrounds. As such, additional data collection, analysis and informed interpretation of the results will be required to accurately assess children’s competence during book reading and to report patterns of achievement. Drawing on the ELEA, alongside a broader set of data, we now propose to investigate how educators can use the tool to strengthen their teaching.

Conclusion

This article has provided details of the process undertaken to validate an assessment tool for very young learners, including young Aboriginal children and children living in disadvantaged circumstances. Within the context of Conversational Reading, the findings suggest the ELEA provides data that may assist teachers to assess children’s receptive and expressive vocabulary skills in a familiar context. Focusing on joint attention and children’s interactions with texts during book reading, the tool focuses on key aspects of early literacy as a process of understanding the semiotics used to create meaning (New London Group, 2000) that support children’s progression to formal learning. While we acknowledge the small sample size, particularly the Aboriginal children cohort numbers, we propose that the tool has potentially wide application for teaching practice with children in a range of contexts and communities. In particular, the tool draws attention to the role of authentic assessment practices designed to support early childhood educators to use assessment data to evaluate and modify practices towards improved language and literacy outcomes for young children, particularly those experiencing disadvantage.

Footnotes

Acknowledgments

We acknowledge the lead work of Professor Collette Tayler in this research. Collette had a keen commitment to improving the educational outcomes of Australian Aboriginal children and was instrumental in setting up this study. Collette passed away in December 2017. We are grateful to the families and community members who participated in the research and to the FaFT staff who contributed their cultural knowledge and wisdom throughout the study. We also thank the Indigenous Early Childhood Parenting Reference Group for their guidance and contribution.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This study was funded by the Australian Research Council (LP130100001), the Northern Territory Government's Department of Education and Department of Health who were partners to this grant, and the Kimberley Foundation.

ORCID iD

Janet Scull

Appendix

References

Australian Curriculum, & Assessment and Reporting Authority (ACARA) (2018). NAPLAN achievement in reading, writing, language conventions and numeracy: National report for 2018. Author.

Adams

R. J.

M. L.

Wilson

M. R

(2015). ACER ConQuest: Generalised item response modelling software Version 4. Australian Council for Educational Research.

Atkinson

(2002). Trauma trails, recreating song lines: The transgenerational effects of trauma in Indigenous Australia. Spinifex Press.

Australian Early Development Census . (2019). Findings from the AECD. Commonwealth of Australia. https://www.aedc.gov.au/early-childhood/findings-from-the-aedc.

Clarke

(2009). Supporting children learning English as a second language in the early years (birth to six years). Melbourne, Australia: Victorian Curriculum and Assessment Authority.

Bradley

E. H.

Curry

L. A.

Devers

K. J.

(2007). Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Services Research, 42(4), 1758-1772.

Brookes

Tayler

(2016). Effects of an evidence-based intervention on the Australian English language development of a vulnerable group of young Aboriginal children. Australasian Journal of Early Childhood, 41, 4-15.

Campbell

F. A.

Helms

Sparling

J. J.

Ramey

C. T

(1998). Early childhood programs and success in school: The Abecedarian Study. In Barnett

W. S.

Boocock

S. S.

(Eds.), Early care and education for children in poverty: Promises, programs and long-term results (pp. 145-166). State University of New York Press.

Campbell

F. A.

Pungello

E. P.

Burchinal

Kainz

Pan

Wasik

B. H.

Barbarin

O. A.

Sparling

J. J.

Ramey

C. T.

2012). Adult outcomes as a function of an early childhood educational program: An Abecedarian project follow-up. Developmental Psychology, 48(4), 1033-1043.

10.

Clay

M. M

(2019). An observation survey of early literacy achievement (4th ed.). Heinemann.

11.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.

12.

Cooke

(2013). Empowered parents: Book reading practices of Indigenous parents coached in Conversational Reading techniques in Galiwin'ku – A case study [Unpublished master’s thesis. The University of Melbourne.

13.

Cooke

Piers-Blundell

(2019). Embedding evidence-based practice in a remote Indigenous early learning and parenting program: A systematic approach. In Rennie

Harper

(Eds.), Literacy education in the Australian Indigenous context (pp. 185-201). Springer.

14.

Cummins

(2017). Teaching minoritized students: Are additive approaches legitimate? Harvard Educational Review, 87(3), 404-425.

15.

Department of Education and Training (2016). Victorian Early Years Learning and Development Framework (VEYLDF). Department of Education and Training.

16.

Dickinson

D. K.

Griffith

J. A.

Golinkoff

R. M.

Hirsh-Pasek

(2012). How reading books fosters language development around the world. Child Development Research, 2012, 602807.

17.

Dickinson

D. K.

Tabors

P. O

(2001). Beginning literacy with language: Young children learning at home and school. Paul H. Brookes.

18.

Frischen

Bayliss

A. P.

Tipper

S. P.

(2012). Gaze cuing of attention: Visual attention, social cognition, and individual differences. Psychological Bulletin, 133(4), 694-724. doi:10.1037/0033-2909.133.4.694

19.

Gee

J. P.

(2003). Opportunity to learn: A language-based perspective on assessment. Assessment in Education: Principles, Policy & Practice, 10(1), 27-46.

20.

Gonski

Arcus

Boston

Gould

O-Brien

Perry

Roberts

(2018). Through growth to achievement: The report of the review to achieve educational excellence in Australian schools. Australian Government.

21.

Gonzalez

J. E.

Pollard-Durodola

Simmons

D. C.

Taylor

A. B.

Davis

M. J.

Fogarty

Simmons

(2014). Enhancing preschool children's vocabulary: Effects of teacher talk before, during and after shared reading. Early Childhood Research Quarterly, 29(2), 214-226.

22.

Grolig

(2020). Shared storybook reading and oral language development: A bioecological perspective. Frontiers in Psychology, 11, 1818. doi:10.3389/fpsyg.2020.01818

23.

Hart

Risley

T.R

(2003). The early catastrophe: The 30 million word gap by age 3. American Educator, 27(1), 4-9.

24.

Hearne

B. G.

Stevenson

(2000). Choosing books for children: A commonsense guide. University of Illinois Press.

25.

Hoff

(2006). How social contexts support and shape language development. Developmental Review, 26(1), 55-88.

26.

Hoff

Core

Place

Rumiche

Senore

Parra

(2012). Dual language exposure and early bilingual development. Journal of Child Language, 39, 1-27.

27.

James

Sparling

Page

Scull

(2015). Lookout! Here comes Cheeky! The University of Melbourne.

28.

Johnston

Costello

(2005). Principles for literacy assessment. Reading Research Quarterly, 40(2), 256-267. doi:10.1598/RRQ.40.2.6

29.

Kidd

Donnelly

Christiansen

M. H.

(2018). Individual differences in language acquisition and processing. Trends in Cognitive Sciences, 22, 154-169. doi:10.1016/j.tics.2017.11.006.

30.

Lee

W. Y

(2019). Understanding early childhood teaching and learning in two northern Arnhem Land family and playgroup contexts. University of Melbourne.

31.

Lo Bianco

(2016). Foreward: Understanding “growing up literate”. In Scull

Raban

(Eds.), Understanding growing up literate (pp. III-VI). Eleanor Curtain Publishing.

32.

Mancilla-Martinez

Lesaux

N. K.

(2011). Early home language use and later vocabulary development. Journal of Educational Psychology, 103(3), 535-546.

33.

Martin

(2005). Childhood, lifehood and relatedness: Aboriginal ways of being, knowing and doing. In Phillips

Lampert

(Eds.), Introductory Indigenous studies in education (pp. 27-40). Pearson Australia.

34.

Massey

S. L.

Pence

K. L.

Justice

L. M.

Bowles

R. P.

(2008). Educators’ use of cognitively challenging questions in economically disadvantaged preschool classroom contexts. Early Education & Development, 19(2), 340-360.

35.

McLachlan

Fleer

Edwards

(2013). Early childhood curriculum: Planning, assessment, and implementation. Cambridge University Press.

36.

Molyneux

Scull

Aliani

(2016). Bilingual education in a community language: Lessons from a longitudinal study. Language and Education, 30(4), 337-360.

37.

National Early Literacy Panel . (2009). Developing early literacy: Report of the National Early Literacy Panel. Jessup, MD: National Institute for Literacy. http://lincs.gov/publications/pdf/NELPReport09.pdf.

38.

New London Group (2000). A pedagogy of multiliteracies designing social futures. In Cope

Kalantzis

(Eds.), Multiliteracies: Literacy learning and the design of social futures (pp. 9-38). Routledge.

39.

Nguyen

T. K. C.

Griffin

(2007). Development of a scale to measure economic status of students in rural Vietnam. Asia Pacific Education Review, 8, 205-215.

40.

Nguyen

Griffin

(2013). Developing of two instruments to measure attitudes of Vietnamese parents and students toward schooling. Journal of Applied Measurement, 14(3), 282-298.

41.

Noble

Sala

Peter

Lingwood

Rowland

Gobet

Pine

(2019). The impact of shared book reading on children’s language skills: A meta-analysis. Educational Research Review, 28, 100290. doi:10.1016/j.edurev.2019.100290

42.

Page

Cock

M. L.

Murray

Eadie

Niklas

Scull

Sparling

(2019). An Abecedarian approach with Aboriginal families and their young children in Australia: Playgroup participation and developmental outcomes. International Journal of Early Childhood, 51(2), 233-250.

43.

Pianta

R. C

(2003). Standardized classroom observations from pre-k to 3rd grade: A mechanism for improving access to consistently high quality classroom experiences and practices during the P–3 years. Foundation for Child Development.

44.

Raban

(2014). TALK to think, learn, and teach. Journal of Reading Recovery, Spring, 1-11.

45.

Ramey

C. T.

Sparling

J. J.

Ramey

S. L

(2012). Abecedarian The ideas, the approach and the findings. Sociometrics Corporation.

46.

Rowe

M. L.

Snow

C. E.

(2019). Analyzing input quality along three dimensions: Interactive, linguistic, and conceptual. Journal of Child Language, 47, 5-21. doi:10.1017/s0305000919000655

47.

Scull

Paatsch

Raban

(2013). Young learners: Teachers’ questions and prompts as opportunities for children’s language development. Asia-Pacific Journal of Research in Early Childhood Education, 7(1), 69-91.

48.

Sénéchal

LeFevre

J.-A.

(2002). Parental involvement in the development of children’s reading skill: A five-year longitudinal study. Child Development, 73, 445-460. doi:10.1111/1467-8624.00417.

49.

Silvén

Ahtola

Niemi

(2003). Early words, multiword utterances and maternal reading strategies as predictors of mastering word inflections in Finnish. Journal of Child Language, 30(2), 253-279.

50.

Sparling

Meunier

(2019). Abecedarian: An early childhood education approach that has a rich history and a vibrant present. International Journal of Early Childhood, 51, 207-216. doi:10.1007/s13158-019-00247-2

51.

Tayler

Ishimine

(2013). Assessment. In Pendergast

Garvis

(Eds.), Teaching early years: Rethinking curriculum, pedagogy and assessment (pp. 285-301). Allen & Unwin.

52.

Tomasello

Todd

(1983). Joint attention and lexical acquisition style. First Language, 4(12), 197-212.

53.

Wigg

E. H.

Secord

W. A.

Semel

(2006). Clinical evaluation of language fundamentals: Preschool second edition: Australia and New Zealand. Pearson.

54.

Wiliam

(2013). Assessment: The bridge between teaching and learning. Voices from the Middle, 21(2), 15-20 http://www.ncte.org.ezp.lib.unimelb.edu.au/journals/vm/issues/v21-2.

55.

Wright

B. D.

Masters

G. N

(1982). Rating scale analysis. Mesa Press.

56.

Adams

(2007). Applying the Rasch model to psycho-social measurement: A practical approach. Educational Measurement Solutions.

57.

Tam

H. P.

Jen

T.-H

(2016). Educational measurement for applied researchers: Theory into practice. Springer.