The effect of workshop training on rater variability in children’s oral narrative assessment

Abstract

Oral narrative assessments are important for diagnosis of language disorders in school-age children so scoring needs to be reliable and consistent. This study explored the impact of training on the variability of story grammar scores in children’s oral narrative assessments scored by multiple raters. Fifty-one speech pathologists and 19 final-year speech pathology students attended training workshops on oral narrative assessment scoring and analysis. Participants scored two oral narratives prompted by two different story stimuli and produced by two children of differing ages. Demographic information, story grammar scores and a confidence survey were collected pre- and post-training. The total story grammar score changed significantly for one of the two oral narratives. A significant effect was observed for rater years of experience and the change in total story grammar scores post training, with undergraduate students showing the greatest change. Two story grammar elements, character and attempt, changed significantly for both stories, with an overall trend of increased element scores post-training. Confidence ratings also increased post-training. Findings indicated that training via an interactive workshop can reduce rater variability when using researcher-developed narrative scoring systems.

Keywords

adult learning monitoring change oral narrative professional development story grammar

I Introduction

The way in which children tell stories (narratives) provides important information about their language and learning ability. Oral narrative assessment accounts for all domains of language, including morphosyntax, semantics and pragmatics (Botting, 2002). As narratives are a regular component of conversation and school activities, they allow language to be assessed in an ecologically valid way within a natural context (Gillam et al., 1999). Narrative assessment can describe developmental changes in children’s language abilities (John et al., 2003; Ukrainetz et al., 2005), and contributes to differential diagnosis of developmental language disorders and learning difficulties (Docking et al., 2016; Miniscalco et al., 2007; King and Palikara, 2018). Oral narrative skills are longitudinal predictors for reading development (Babayiğit et al., 2021). Piasta et al. (2018) suggest early narrative skills provide a foundation for the development of emergent literacy, word reading, and reading comprehension. For these reasons, oral narrative assessment has become an important practice for many disciplines, including speech pathologists and educators (Malec et al., 2017). Moreover, reliable and consistent analysis of oral narratives is essential to support accurate diagnosis and measurement of change in response to therapy.

1 Story grammar

While oral narratives may be analysed across both macrostructural and/or microstructural elements, this study focused on macrostructure. Macrostructure describes a narrative’s overall coherence and story grammar (SG) elements such as characters, setting, initiating events, attempts and consequences (Stein and Glenn, 1979). Microstructure refers to smaller linguistic elements such as sentence length, or grammatical language (Justice et al., 2010). SG assessment results may be interpreted using norms or criterion descriptors and inform diagnosis and goals for education or intervention (Petersen et al., 2008).

The SG approach to the assessment of narrative macrostructure, initially developed by Stein and Glenn (1979), is widespread amongst Western populations (Urbach, 2012). This framework defines stories as the completion of goal-directed behaviours by a protagonist and considers the story setting (place and time), characters and episodes. While elements and their descriptors differ across models developed from this framework, all models agree that stories involve a protagonist in a specific time and place (setting) who is faced with a problem (initiating event). The character responds emotionally (internal response) and thinks of ideas to solve the problem (plan). The character sets their plan into action (attempt) and the outcome (consequence) leads to the resolution of the story and a final concluding statement (resolution and ending) (Urbach, 2012).

While SG may appear clearly defined, its elements can be challenging for narrative assessors to accurately identify when assessing children’s oral narratives. Depending on the child’s ability, SG elements may be omitted or incomplete within the narrative. While it is possible to assess a narrative’s SG on a binary scale of element absence or presence, use of a more descriptive scoring method, such as a rubric, is arguably more informative and holds greater validity (Heilmann et al., 2010a). Rubrics provide descriptive criteria and score ratings for each SG element based on the quality of the element and are used more broadly in educational contexts to facilitate consistent marking criteria across study domains (Jonsson and Svingby, 2007). Assessors must identify and appraise the SG element in question, often relying on brief descriptions in rubrics as a basis for their scores (Heilmann et al., 2010a). Differing interpretation of scoring criteria can lead to greater variability between raters in assessment settings such as clinics or classrooms (Petersen et al., 2008).

2 Reliability of oral narrative assessment tools

Interrater reliability determines whether different raters using the same tool to measure the same variable will reach an appropriately similar score. A tool with high interrater reliability should thus demonstrate low rater variability. Many oral narrative assessment tools report adequate interrater reliability amongst their psychometric properties; however, trained raters are usually used to establish interrater reliability for specific narrative assessment protocols (Gillam et al., 2017; Gillam and Pearson, 2017; Heilmann et al., 2010a; Petersen et al., 2008; Schneider et al., 2005). While appropriate within a research context, this raises issues when viewed from a pragmatic perspective outside the research context.

Narrative assessment tools are designed to assess children in clinical and school education settings and it is therefore the ‘natural setting’ in which they must demonstrate reliability. Assessment tool developers recognize that people rating assessments (‘raters’) may not have access to training and expert support in narrative scoring (Gillam et al., 2017), and have questioned application of reliability data to a natural setting (Justice et al., 2010). Assessment tools validated by trained research staff may report adequate interrater reliability; however, these tools cannot be deemed effective for general use if there is high variability among raters for real-world utility.

3 Training

One solution to high variability is provision of training to raters in natural settings. Multiple studies illustrate the benefits of training in improving agreement in other areas of speech pathology practice including, perceptual voice assessment (Barsties et al., 2017; Brinca et al., 2015; Chan and Yiu, 2002; Eadie and Baylor, 2006; Lee et al., 2009; Schaeffer, 2013; Shewell, 1998), stuttering (Bainbridge et al., 2015; Einarsdóttir and Ingham, 2008), and cleft-palate speech (Chapman et al., 2016). There have also been calls to increase training in narrative assessment for education professionals (Malec et al., 2017).

Only two efficacious oral narrative assessment protocols were identified that offer universally available training programs: the Narrative Assessment Protocol-2 (NAP-2; Bowles et al., 2020) and the Narrative Scoring Scheme (NSS; Heilmann et al., 2010a). The NAP-2 assesses preschool children’s narrative language abilities. Training includes an introduction to terminology, revision of scored transcripts, and practice scoring using exemplars. The NSS (Heilmann et al., 2010a) provides a rubric for rating SG, and normative data within the Systematic Analysis of Language Transcripts (SALT) software (Caygill, 1998; Heilmann et al., 2010a). These two assessment protocols and training packages are designed for self-paced online learning and developed in the United States.

Despite widespread availability to raters, limited research supports real world effectiveness for either assessment tool when used by professionals other than researchers, a limitation recognized by Justice et al. (2010). Similarly, while normative research has established efficacy for NAP-2 and NSS, neither the assessments nor training have been evaluated in natural settings to determine effectiveness (Bowles et al., 2020; Heilmann et al., 2010a). Furthermore, while online training can provide global reach as a vehicle for professional development in assessment protocols, it may be less effective than face to face workshops (Lowman, 2016).

A workshop approach for speech pathologists, used by Lowman (2016), mirrored calibration or moderation methods used in schools and universities to ensure uniform marking of student assessments. Calibration relies on peer discussion of student assessment, with raters evaluating student work then discussing their results until they reach a consensus in scoring interpretation (Gardiner et al., 2020). Calibration training workshops have demonstrated reduction in rater scoring variability for tertiary accounting and business assessments (Gardiner et al., 2020; O’Connell et al., 2016). Improved scoring knowledge and skills have also been associated with greater confidence in use of scoring systems (Alanazi and Nicholson, 2019; Doble et al., 2019; Kamal et al., 2012; O’Connell et al., 2016). Therefore, the use of workshop training which incorporates discussion and calibration for oral narrative assessment may reduce variability in practitioner SG ratings, enhance homogeneity and proficiency, and increase confidence.

The present study investigated whether face to face training workshops, using calibration methods, and a researcher-designed rubric could significantly decrease rater variability in scoring SG of narratives produced by typically developing children. Three specific questions were:

Does workshop training lead to change and reduced variability in SG scoring across speech pathologists?

What is the impact of speech pathologists’ experience on SG scoring in response to workshop training?

Does workshop training improve speech pathologists’ confidence in scoring oral narrative assessment?

II Method

Ethical approval for the study was obtained from the Australian Catholic University Human Research Ethics Committee (2018-270EAP). A quasi-experimental, single group pre-/post-test design was used to explore the research questions.

1 Participants

Fifty-one speech pathologists and 19 fourth year undergraduate speech pathology students participated in the research. Participants were attendees at professional development workshops advertised through professional networks and to final year speech pathology students at Australian Catholic University and participation in the research component was optional. Workshops were conducted in Sydney (n = 18), Melbourne (n = 32), and Brisbane (n = 20) during May 2019. All workshop attendees consented to participate in the research and submitted anonymized response sheets to the researchers. Years of professional experience for the 51 speech pathologists ranged from 0.2 – 38 years (mean = 12.3 years). All speech pathologists worked with a paediatric caseload. One speech pathologist participant submitted pre-training data but did not submit post-training data.

2 Narrative samples

Thirteen oral narrative samples were gathered for use within the study from seven children of university speech pathology staff. The children included four boys and three girls between ages 6 and 12 years who had typically developing language. Eight transcripts were selected for use within the study. Longer transcripts were excluded in consideration of time constraints for training purposes.

Narratives were collected and audio-recorded by parents using one or two wordless picture book stimuli: William’s Baby Brother (WBB) and/or Sam’s Skateboard (SS) from the Oral Narrative Assessment Package (ONAP) (McCandlish and Schaefer, 2003). WBB was designed for younger (4–7 years) and SS was designed for older (8–11 years) children. Both stories have three characters and provide potential for complex episodes, with more than one attempt to solve the problem. These story stimuli were selected due to their ease of use, appropriateness to Australian children, and accessibility to participants. The ONAP allowed users the choice of generation or retell. For this study, examiners told the story, then asked comprehension questions and then asked the child to retell the story.

Narratives were transcribed by the researchers, with two used to assess the research outcomes and six others used for training purposes. An additional two artificial narratives based on WBB and SS were generated by the second author to provide exemplars of less complex narratives for training purposes within workshop activities. Thus, a total of 10 transcripts were provided for use in the workshops. An equal number of transcripts from each story, WBB and SS was used for scoring exemplars and practice. The least complex training sample was an isolated description of seven utterances, and the most complex training sample comprised 25 C-units (an independent clause and its modifiers) and had a complication (obstacle to the first attempt to solve the problem). Participants also reviewed the ONAP model scripts for WBB and SS during the workshop.

3 Narrative analysis tool

Analysis checklists in the ONAP were designed to report presence or absence of SG elements with no researched psychometric properties. The SG subscale from the Monitoring Indicators of Scholarly Language (MISL; Gillam et al., 2017) was selected as the analysis tool for this study because it provided a more nuanced rubric. The MISL was designed for use with children’s self-generated narratives and retells in response to varied picture stimuli and for describing stages of developing narrative complexity (from simple descriptions to multiple episodes) as a means of tracking progress. Its psychometric properties were explored using the Aliens story from the Test of Narrative Language (Gillam and Pearson, 2017) and considered acceptable for reliability, validity and responsiveness to therapy. The MISL rubric includes seven SG elements (character, setting, event, response, plan, attempt and conclusion) with four rating levels: Not Present (0), Emerging (1), Mastery (2) and Elaborated Knowledge (3). Definitions for each element are provided in Table 1. Elements rated at the emerging level may include non-specific descriptions or lack clear causal connections. A rating at the mastery level requires evidence of causal connections (e.g. an event motivates a character’s action) while a rating at the elaborated knowledge level requires multiple instances of well-developed elements (e.g. more than one character with proper names). All ratings were summed to create a total SG score for each narrative.

Table 1.

Definitions for Monitoring Indicators of Scholarly Language (MISL) story grammar elements.

Story grammar element	Definition
Character	An agent who performs an action.
Setting	Information about location or time.
Initiating event	Event(s) that motivate characters to take action; explicitly stated.
Internal response	Feelings stated about the Initiating Event. Must be made by the character taking the actions related to the Initiating Event.
Plan	Thoughts stated by characters related to a decision to take action.
Attempt	Actions taken by characters motivated by an Initiating Event.
Consequence	End result of characters actions in relation to the Initiating Event; not an Internal Response.

Source. Gillam et al., 2017.

4 Training and data collection

Data was collected directly before, and immediately after training on the same day. Three primary data sets were: (1) demographic data for the participants; (2) pre- and post-training SG scores for one WBB and one SS narrative; and (3) pre- and post-training confidence scores.

Pre-training data collection occurred after a 15-minute introduction addressing the purpose of oral narrative assessment and its relevance to language disorder. Participants were informed that the study aimed to investigate whether training can reduce variability in oral narrative analysis, but the nature of variability was not discussed. Assessment and analysis procedures were not addressed during the introduction and participants were invited to score two narrative test samples from a rubric independently without discussion or support.

The narrative test samples were written transcripts of two oral narratives: WBB, a less complex narrative from a younger child (5-years-old, first year of school); and SS, a more complex narrative from an older child (9-years-old, fifth year of school) (see Appendix 1). Child age was not provided to the participants. The SS sample was longer and more linguistically complex with a higher number of C-units (16, cf. 13), mean length of C-units (11.80, cf. 8.00), total number of words (170, cf. 94) and number of different words (88, cf. 48). The SS sample also showed more complexity in SG components. For example, the WBB sample did not include any SG element of Plan, while the SS sample included two C-units describing this element:

5 Sam was trying to think of something to do.

6 Finally, he came up with an idea to sell some of his old toys.

Participants were provided with the SG subscale from the MISL narrative rubric (Gillam et al., 2017) and definitions of SG components, but were not provided with exemplars to illustrate scoring application. Participants also completed a demographic questionnaire and a 12-item questionnaire exploring their confidence in administering and scoring oral narrative assessment. Confidence was rated on a Likert scale from 1 (very unsure) to 5 (extremely confident). Consent forms and all data were collected from participants after the pre-training scoring. Participants added a unique identifier code to all documents to enable linking by the authors for analysis.

The second stage involved training in oral narrative assessment. Training was delivered by the second author as a single-day workshop which participants attended at three locations, each containing 4.5 hours of instruction. Content was supported by a PowerPoint and hard copy participant handouts. Formal workshop training commenced with an introduction to oral narrative assessment tools and methods, description of MISL scoring categories and facilitated discussion around areas of confusion or concern. Exemplars of scored narratives were provided and discussed. Participants then practiced scoring oral narrative samples from transcripts individually and in small groups, followed by whole-group discussion of scoring outcomes and considerations. The second author responded to questions or issues arising from practice sessions and presented rationales to assist with scoring dilemmas.

The final stage of data collection occurred immediately post-training. Participants independently scored the same oral narrative assessments as the pre-training stage, without access to their previous scores, and completed the same confidence questionnaire. No discussion or questions were permitted; however, participants were able to refer to handouts as well as their own notes taken during the workshop. Following the post-training data collection, the workshop addressed intervention approaches and resources.

5 Analysis

Descriptive statistics were used to explore changes in SG scoring pre- and post-training for WBB and SS. A paired samples t-test determined if any significant change had occurred in total SG scores. A repeated measures ANOVA determined if there was an interaction effect between experience and total SG scores pre- and post-training. Years of professional experience was categorized into three subgroups that provided viable participant numbers for analysis: students and less than one year (n = 24), one to 10 years (n = 21), and more than 10 years (n = 24). The Wilcoxon signed-rank test was used to determine if any significant change had occurred in ratings of individual SG elements pre- and post-training and to compare Likert scale ratings pre- and post-training for rater confidence on aspects of assessment and analysis.

III Results

1 Narrative scores and variability

Descriptive statistics for both WBB and SS pre- and post-training total SG scores are shown in Table 2. Changes in range and standard deviations demonstrated a reduction in variability for WBB post-training, but not for SS. A paired samples t-test indicated a significant increase in total scores for WBB pre- and post-training (t(69)= −3.42, p = .001, d = 0.48) but no significant change in total scores for SS pre-and post-training (t(69)= −0.68, p = .50, d = 0.10).

Table 2.

Descriptive statistics for Monitoring Indicators of Scholarly Language (MISL) total scores pre- and post-training.

	William’s Baby Brother (WBB)		Sam’s Skateboard (SS)
	Pre-training	Post-training	Pre-training	Post-training
n	70	69	70	69
Mean	8.23	9.64	15.26	15.74
Minimum	3	6	10	10
Maximum	16	14	21	20
SD	2.94	2.06	2.45	2.12

Descriptive statistics and Wilcoxon signed-rank test results for the WBB and SS SG element scores are shown in Table 3. Both stories showed significantly increased scores for the elements Character and Attempt. The score for SS Consequence also significantly increased. Scores significantly decreased for WBB Plan and SS Setting.

Table 3.

Monitoring Indicators of Scholarly Language (MISL) element scores pre- and post-training for William’s Baby Brother (WBB) and Sam’s Skateboard (SS).

Element	Story	Pre-training scores: Mean (SD)	Post-training scores: Mean (SD)	z-score	p
Character	WBB	2.21 (0.82)	2.94 (0.24)	−5.313	< .001*
Character	SS	2.27 (0.51)	2.51 (0.53)	−2.667	.008*
Setting	WBB	0.43 (0.65)	0.33 (0.59)	−.951	.342
Setting	SS	2.29 (0.76)	1.74 (0.70)	−4.072	< .001*
Initiating event	WBB	1.64 (0.82)	1.86 (0.60)	−1.792	.073
Initiating event	SS	2.33 (0.61)	2.43 (0.63)	−.982	.326
Internal response	WBB	0.44 (0.58)	0.57 (0.68)	−1.184	.236
Internal response	SS	1.00 (0.90)	1.19 (0.86)	−1.416	.157
Plan	WBB	0.36 (0.62)	0.16 (0.50)	−2.062	.039*
Plan	SS	2.66 (0.63)	2.71 (0.52)	−.465	.642
Attempt	WBB	1.49 (0.61)	2.00 (0.79)	−4.443	< .001*
Attempt	SS	2.33 (0.56)	2.58 (0.53)	−2.795	.005*
Consequence	WBB	1.66 (0.85)	1.78 (0.79)	−.808	.419
Consequence	SS	2.39 (0.53)	2.58 (0.50)	−2.335	.020*

Note. * indicates significant change.

2 Professional experience and confidence

ANOVA identified a significant interaction between years of experience and WBB total SG score change pre- and post-training (F(2, 66) = 3.43, p = .038, $η_{p}^{2}$ = .09). There was no significant interaction effect for SS (F(2, 66) = 1.89, p = .16, $η_{p}^{2}$ = .05). The interaction between WBB total SG scores and participant professional experience is shown in Figure 1. The WBB total SG scores increased significantly for the most inexperienced group (students and < 1 year) but score increases were not significant for the two more experienced groups. Confidence rating significantly increased between pre- and post-training for all 12 items in the confidence questionnaire (Table 4). Variability in participant confidence also decreased following training demonstrated by reduced standard deviation.

Figure 1.

Total scores for William’s Baby Brother (WBB) pre- and post- training by years of experience.

Table 4.

Mean scores (standard deviation), z-score and p values for confidence pre- and post-training.

Question: Indicate your level of confidence in performing the following activities:	Pre-training scores: Mean (SD)	Post-training scores: Mean (SD)	z-score	p
1. Choosing an appropriate assessment tool	2.75 (1.04)	4.16 (0.56)	−6.44	< .001
2. Eliciting a narrative sample	3.56 (0.91)	4.12 (0.66)	−4.69	< .001
3. Recording a narrative sample	3.67 (0.99)	4.32 (0.70)	−4.89	< .001
4. Transcribing a narrative sample	3.33 (1.03)	3.99 (0.76)	−4.71	< .001
5. Finding an overall SG score	2.32 (0.91)	3.79 (0.66)	−6.42	< .001
6. Analysing characters	3.00 (1.13)	4.31 (0.50)	−5.92	< .001
7. Analysing setting	3.01 (1.08)	4.19 (0.55)	−5.76	< .001
8. Analysing initiating event	2.88 (0.99)	3.85 (0.61)	−5.67	< .001
9. Analysing internal response	2.69 (1.03)	3.87 (0.71)	−5.75	< .001
10. Analysing plan	2.76 (1.06)	3.82 (0.71)	−5.40	< .001
11. Analysing action/attempt	2.70 (0.94)	4.01 (0.61)	−6.27	< .001
12. Analysing consequence	2.87 (0.99)	4.01 (0.50)	−6.07	< .001

IV Discussion

The present study examined whether provision of workshop training would impact variability of oral narrative assessment scoring. It also questioned whether experience influenced response to training, and whether practitioner confidence was impacted by training. The results showed significantly diminished variability for one story sample, and no change for the other. An overall trend for decreased variability was seen for individual SG elements. In the story which showed significant change, an interaction between years of experience and response to training was noted with students and new graduates showing the greatest change in scores compared to more experienced participants. Confidence in all areas of oral narrative assessment improved following training.

1 The impact of training on score variability

Results indicate that training has some benefit on reducing rater variability in oral narrative scoring, and guided examples related to target stories with accompanying discussion are likely to lead to greater consensus in interpretation of story element definitions. One possible reason why WBB showed reduced variability in total score after training and SS did not, is the influence of story complexity. SS was a story from an older child and showed more complexity than WBB, as illustrated by more and longer utterances, more diverse vocabulary, and higher MISL scores. Previous research comparing ratings of narrative quality found agreement was highest at the ‘polar extremes’ of the rating scale (Streit Olness et al., 2005). It is possible that, since SS was a more complex and developed story, it was clearer for all raters, regardless of experience, to identify strengths and score consistently. Conversely, training to assess a less sophisticated narrative had a greater impact on consensus, highlighting the importance of including narratives of varying complexity in training.

Another potential factor is that speech pathologists marking a less mature story may mark it lower to ensure the child’s difficulties are noted, and to leave room for improvement over treatment. However, mean scores for both WBB and SS increased, despite discussion during the workshop guiding raters to mark a story lower if they were unsure about whether the criteria were sufficiently reached. Noting this instruction, the increase in scores may indicate that uncertainty in scoring was minimized following training.

Observations of variability were less consistent for individual SG elements. Overall, 71% of SG element scores showed a decrease in standard deviation following training, indicating a trend towards lower variability, with half of these also showing a significant increase in scores. This is likely due to discussion and use of exemplars during training, which clarified definitions of key concepts within the rubric. For example, discussion about proper names occurred, with participants debating whether ‘Mum’ was a proper name in WBB. It was decided that this was a proper noun in the context of the narrative, as most children only call their mother ‘Mum’. The significant increase in score and decrease in standard deviation for Character in WBB reflected this understanding following training. Participants also debated whether Mum and ‘the shopkeeper’ could be considered main characters in SS. Discussion about how to score attempts for WBB required clarification about the problem and goal (to pacify the baby).

For SS, discussion about how to score attempts focused on identifying the complication which increased the attempt score (didn’t have enough money). Discussion about the consequence score focused on identifying multiple occurrences (getting $10 for selling old toys, and $5 for mowing the lawn). It is also possible that the workshop process and focus on SG elements helped participants to assess these independently of influencing factors, such as the microstructural complexity of the narrative sample. This may explain the significant decrease in scores and standard deviation for setting in SS. In the test sample, the setting was expressed thus:

1 One day Sam was walking around the city when he saw a big garage sale sign.

2 Sam walked to the behind of the house and found an old box with a skateboard in it.

Although the setting was expressed in complex language, it did not meet the MISL criteria of “reference to place denoted using proper name, or reference to specific time” required to be classed as elaborated knowledge.

2 The impact of years of experience

Training was most beneficial for participants with the least experience, who also had the largest change in total scores following training, resulting in a more homogenous post-training result overall. This is consistent with findings across other areas of speech pathology showing inexperienced raters made greater improvements than more experienced raters following training (Silbergleit et al., 2018). It is also consistent with the notion that interrater reliability of narrative assessment tools will improve as raters gain experience (Gillam et al., 2017).

Some change, although not significant, was seen for raters with more than 10 years of experience, suggesting variability may increase over time. This may relate to the phenomenon of ‘coder drift’, which describes variability of ratings increasing over time due to a lack of calibration (Syed and Nelson, 2015). These results indicate that training is most beneficial to students and early career practitioners, yet also supports more experienced practitioners.

3 The impact of training on confidence

A significant increase in confidence after training was likely due to opportunities presented for asking questions, comparing responses and consolidating understanding throughout the collaborative workshop. In this flexible training, participants were able to air specific concerns and clarify individual aspects of scoring. The improvement in confidence is consistent with other studies in the speech pathology field (Alanazi and Nicholson, 2019; Doble et al., 2019; Kamal et al., 2012).

In the education sector, Millet (2018) suggested that low confidence in assessment grading leads to increased variability between raters. Calibration studies have reflected that increase in confidence following workshops may contribute to reduced variability, with the potential for variability to continue to decrease as confidence in the scoring process increases (O’Connell et al., 2016). In the current study, while confidence increased and overall variability diminished for total scores, confidence in scoring individual story elements did not always lead to a reduction in variability for the same story elements. Hence, confidence in rating did not guarantee homogeneity of scoring.

4 Limitations

While several narrative exemplars were provided during the training workshops, only two stories were analysed by participants for pre- and post- testing, which limited data available for analysis as well as training experience. Inclusion of stories from children with language disorders may have provided a more thorough training for participants, providing opportunities to contrast narratives of varying quality and practice differential scoring. The stories in both research analysis and training were retells of the same two narratives, limiting exposure of participants to stories gathered using different protocols and procedures. Follow-up was beyond the scope of the study and could have provided insight into the long-term outcomes of training. Furthermore, only brief demographic data was gathered. A more extensive review of participant experience with oral narrative scoring may have been informative.

5 Implications for practice and future research

The outcomes of this study indicate workshop training involving group discussion and exemplars, facilitated by a trainer with expertise in narrative assessment, is beneficial when applied to oral narrative assessment scoring. Collaborative training processes may contribute to increased confidence in the delivery of narrative assessments, and greater consistency of narrative assessment, thereby increasing the validity and effectiveness of narrative assessment in natural contexts.

Further research into the long-term effects of training would be beneficial in evaluating the efficacy of workshop training as well as providing recommendations for the frequency of training to maintain skills and confidence. In speech pathology, increased validity of scoring is essential to consistent practice when assessing children for involvement in language or narrative intervention groups, or on handover to another practitioner. As a diverse range of professions utilize oral narrative assessment procedures, further investigation into the application of this training across professions, particular the education sector, would also be beneficial (Malec et al., 2017).

V Conclusions

The findings of this study suggest that interactive workshop training led by an experienced professional is an effective tool for decreasing variability of speech pathologist scores in oral narrative assessment. While likely to be of most benefit to early career professionals, workshops which involve peer discussion and calibration is beneficial in increasing participant confidence and skills regardless of years of experience. Further investigation into areas of disagreement between raters, particularly after training, would assist in improvements to both assessment protocols and training packages. Workplaces may consider implementing workshop training amongst their teams to facilitate reduced variability in scoring and increase practitioner confidence.

Footnotes

Appendix 1. Transcripts used for pre- and post- scoring

Acknowledgements

This study was undertaken as part of a Bachelor of Speech Pathology (Honours) awarded by Australian Catholic University. Appreciation is extended to Dr Michael Steele for statistical support, participating speech pathologists and to the parents and children who provided oral narrative samples.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Wendy M Pearce

Michelle Donaghy

References

Alanazi

Nicholson

(2019) Audiology and speech-language pathology simulation training on the 1–3–6 early hearing detection and intervention timeline. American Journal of Audiology 28: 348–61.

Babayiğit

Roulstone

Wren

(2021) Linguistic comprehension and narrative skills predict reading ability: A 9-year longitudinal study. British Journal of Educational Psychology 91: 148–68.

Bainbridge

Stavros

Ebrahimian

, et al. (2015) The efficacy of stuttering measurement training: Evaluating two training programs. Journal of Speech, Language and Hearing Research 58: 278–86.

Barsties

Beers

Ten Cate

, et al. (2017) The effect of visual feedback and training in auditory-perceptual judgment of voice quality. Logopedics Phoniatrics Vocology 42: 1–8.

Botting

(2002) Narrative as a tool for the assessment of linguistic and pragmatic impairments. Child Language Teaching and Therapy 18: 1–21.

Bowles

Justice

Khan

, et al. (2020) Development of the Narrative Assessment Protocol-2: A tool for examining young children’s narrative skill. Language, Speech, and Hearing Services in Schools 51: 390–404.

Brinca

Batista

Tavares

, et al. (2015) The effect of anchors and training on the reliability of voice quality ratings for different types of speech stimuli. Journal of Voice 29: 776.e7–776.e14.

Caygill

(1998) SALT: Systematic Analysis of Language Transcripts. Child Language Teaching and Therapy 14: 285–98.

Chan

KMK

Yiu

(2002) The effect of anchors and training on the reliability of perceptual voice evaluation. Journal of Speech, Language and Hearing Research 45: 111–26.

10.

Chapman

Baylis

Trost-Cardamone

, et al. (2016) The Americleft Speech Project: A training and reliability study. Cleft Palate-Craniofacial Journal 53: 93–108.

11.

Doble

Short

Murray

, et al. (2019) Evidence-based practice self-efficacy of undergraduate speech pathology students following training. Disability and Rehabilitation 41: 1484–90.

12.

Docking

Munro

Marshall

, et al. (2016) Narrative skills of children treated for brain tumours: The impact of tumour and treatment related variables on microstructure and macrostructure. Brain Injury 30: 1005–18.

13.

Eadie

Baylor

(2006) The effect of perceptual training on inexperienced listeners’ judgments of dysphonic voice. Journal of Voice 20: 527–44.

14.

Einarsdóttir

Ingham

(2008) The effect of stuttering measurement training on judging stuttering occurrence in preschool children who stutter. Journal of Fluency Disorders 33: 167–79.

15.

Gardiner

Kim

D-GD

Helms

(2020) Key recommendations for improving AoL assessments: A longitudinal analysis of rater bias and reliability in embedded rubric-based measurements. Journal of Education for Business 95: 227–33.

16.

Gillam

Pearson

(2017) Test of Narrative Language: 2. Austin, TX: Pro-Ed.

17.

Gillam

Peña

Miller

(1999) Dynamic assessment of narrative and expository discourse. Topics in Language Disorders 20: 33–96.

18.

Gillam

Fargo

, et al. (2017) Monitoring Indicators of scholarly language: A progress-monitoring instrument for measuring narrative discourse skills. Communication Disorders Quarterly 38: 96–106.

19.

Heilmann

Miller

Nockerts

, et al. (2010a) Properties of the narrative scoring scheme using narrative retells in young school-age children. American Journal of Speech-Language Pathology 19: 154–66.

20.

John

Lui

Tannock

(2003) Children’s story retelling and comprehension using a new narrative resource. Canadian Journal of School Psychology 18: 91–113.

21.

Jonsson

Svingby

(2007) The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review 2: 130–44.

22.

Justice

Bowles

Pence

, et al. (2010) A scalable tool for assessing children’s language abilities within a narrative context: The NAP (Narrative Assessment Protocol). Early Childhood Research Quarterly 25: 218–34.

23.

Kamal

Ward

Cornwell

(2012) Dysphagia training for speech-language pathologists: Implications for clinical practice. International Journal of Speech-Language Pathology 14: 569–76.

24.

King

Palikara

(2018) Assessing language skills in adolescents with autism spectrum disorder. Child Language Teaching and Therapy 34: 101–13.

25.

Lee

Whitehill

Ciocca

(2009) Effect of listener training on perceptual judgement of hypernasality. Clinical Linguistics and Phonetics 23: 319–34.

26.

Lowman

(2016) A comparison of three professional development mechanisms for improving the quality of standards-based IEP objectives. Communication Disorders Quarterly 37: 211–24.

27.

Malec

Peterson

Elshereif

(2017) Assessing young children’s oral language: Recommendations for classroom practice and policy. Canadian Journal of Education 40: 362–92.

28.

McCandlish

Schaefer

(2003) Oral narrative assessment package. Adelaide, SA: Department of Education and Children’s Services. Available at: http://web.seru.sa.edu.au/product/oral-narrative-package/ (accessed June 2021).

29.

Millet

(2018) The relationship between grading leniency and grading reliability. Studies in Higher Education 43(9): 1524–1535.

30.

Miniscalco

Hagberg

Kadesjö

, et al. (2007) Narrative skills, cognitive profiles and neuropsychiatric disorders in 7–8-year-old children with late developing language. International Journal of Language and Communication Disorders 42: 665–81.

31.

O’Connell

De Lange

Freeman

, et al. (2016) Does calibration reduce variability in the assessment of accounting learning outcomes? Assessment and Evaluation in Higher Education 41: 331–49.

32.

Petersen

Gillam

(2008) Emerging procedures in narrative assessment: The Index of Narrative Complexity. Topics in Language Disorders 28: 115–30

33.

Piasta

Groom

Khan

Skibbe

Bowles

(2018) Young children’s narrative skill: Concurrent and predictive associations with emergent literacy and early word reading skills. Reading and Writing 31: 1479–98.

34.

Schaeffer

(2013) Student training to perceptually assess severity of dysphonia using the dysphonic severity percentage scale. Journal of Voice 27: 611–16.

35.

Schneider

Dubé

Hayward

(2005) Edmonton Narrative Norms Instrument (ENNI). University of Alberta. Available at: https://www.ualberta.ca/communications-sciences-and-disorders/resources/clinical-supervisors/edmonton-narrative-norms-instrument/index.html (accessed June 2021).

36.

Shewell

(1998) The effect of perceptual training on ability to use the Vocal Profile Analysis Scheme. International Journal of Language and Communication Disorders 33: 322–26.

37.

Silbergleit

Boettcher

Cook

, et al. (2018) Impact of formal training on agreement of videofluoroscopic swallowing study interpretation across and within disciplines. Abdominal Radiology 43: 2938–44.

38.

Stein

Glenn

(1979) An analysis of story comprehension in elementary school children. In Freedle

(ed.) New directions in discourse processing, volume 2 (pp. 53–120). Norwood, NJ: Ablex.

39.

Streit Olness

Ulatowska

Carpenter

, et al. (2005) Holistic assessment of narrative quality: A social validation study. Aphasiology 19: 251–62.

40.

Syed

Nelson

(2015) Guidelines for establishing reliability when coding narrative data. Emerging Adulthood 3: 375–87.

41.

Ukrainetz

Justice

Kaderavek

, et al. (2005) The development of expressive elaboration in fictional narratives. Journal of Speech, Language, and Hearing Research 48: 1363–77.

42.

Urbach

(2012) Beyond story grammar: Looking at stories through cultural lenses. Education and Urban Society 44: 392–411.