Interrater Reliability of the SNiPS Screening Tool

PURPOSE: The purpose of this study is to determine the interrater reliability (IRR) of the SNiPS Screening Tool, which is part of the SNiPS Scissors Skills Program currently in development (Podvey et al., 2015).

BACKGROUND: Few formal scissors skills programs currently exist, and no psychometric data are available for existing programs (Podvey et al., 2015). Two authors of this study are developing an evidence-based scissors skills program, SNiPS. The SNiPS Screening Tool is used to determine the current level of scissors skills for program users.

DESIGN: This study used a quantitative interrater reliability design. Following institutional review board approval, participants were recruited via a snowballing technique and included two adult volunteers who read and spoke English. Exclusion criteria included those with formal training in assessment, such as teachers and occupational therapists.

METHODS: The current study followed the methods used to establish IRR for two gold-standard assessment tools used in occupational therapy: the Peabody Developmental Motor Scales (Folio & Fewell, 2000) and the Beery–Buktenica Developmental Test of Visual–Motor Integration (6th ed.; Beery, Buktenica, & Beery, 2010). Study participants viewed a presentation to explain how to complete the scoring for the SNiPS Screening Tool, then rated 30 completed protocols, each comprising eight dimensions. All data collected from participants were entered into IBM SPSS Statistics for statistical analysis. Average-measures interclass correlation (ICC) was used to determine the IRR of the SNiPS Screening Tool, and an independent auditor verified the data and data analysis for veracity.

RESULTS: Using Cicchetti’s (1994) ratings, the SNiPS Screening Tool had excellent IRR overall (ICC = .996, p < .001). Six of eight dimensions (grasp, line, angle, circle, triangle, rainbow) also showed excellent IRR (ICC range = .797 to .945, p < .001). Two dimensions (curve, snips) showed good IRR (ICC = .636 and .658, p < .001).

DISCUSSION: The findings of this study indicate that the SNiPS Screening Tool has strong IRR. There was one study limitation of note. Raters noted having a difficult time using photos to evaluate hand placement on scissors and would have preferred to use videos or live children to rate the grasp dimension of the SNiPS Screening Tool. Future research is needed to test concurrent and discriminant validity of the SNiPS Screening Tool, develop baseline norms with typically developing children, and see the complete impact of the SNiPS Program on the development of scissors skills.

IMPACT STATEMENT: The SNiPS Program has been developed to provide occupational therapy practitioners and others with a science-based program to help children develop scissors skills. The results of this study support the continued development of a psychometrically sound screening tool to use with the SNiPS Program to determine a child’s current level of scissors skill development.

References

Beery, K. E., Buktenica, N. A., & Beery, N. A. (2010). The Beery–Buktenica Developmental Test of Visual–Motor Integration: Administration, scoring, and teaching manual (6th ed.). Minneapolis: Pearson.

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. https://doi.org/10.1037/1040-3590.6.4.284

Folio, M. R., & Fewell, R. (2000). PDMS–2: Peabody Developmental Motor Scales. Austin, TX: Pro-Ed.

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23–34.

Podvey, M., Hoover, K., Tumin, J., Hayek, A., Biedermann, A., & Murdoch, E. (2015, April). SNiPS ^© : Development of an evidence-based scissors skills program. Poster presented at the AOTA Annual Conference & Expo, Nashville, TN.