Abstract
We determined whether a widely used assessment of visual–motor skills, the Beery–Buktenica Developmental Test of Visual–Motor Integration (VMI), is appropriate for use as an outcome measure for handwriting interventions. A two-group pretest–posttest design was used with 207 kindergarten, first-grade, and second-grade students. Two well-established handwriting measures and the VMI were administered pre- and postintervention. The intervention group participated in the Size Matters Handwriting Program for 40 sessions, and the control group received standard instruction. Paired and independent-samples t tests were used to analyze group differences. The intervention group demonstrated significant improvements on the handwriting measures, with change scores having mostly large effect sizes. We found no significant difference in change scores on the VMI, t(202) = 1.19, p = .23. Results of this study suggest that the VMI may not detect changes in handwriting related to occupational therapy intervention.
The Beery–Buktenica Developmental Test of Visual–Motor Integration (6th ed.; VMI; Beery, Buktenica, & Beery, 2010) is commonly used to determine eligibility for occupational therapy services, especially in school-based practice. The VMI is designed to assess a person’s ability to integrate visual and motor skills so that the proper treatment can be provided (Beery, Buktenica, & Beery, 2010). However, the VMI is used not just as a screening or assessment tool but also as an outcome measure to determine improvements in visual–motor integration skills after handwriting interventions (Howe, Roston, Sheu, & Hinojosa, 2013). Although many reports in the literature have documented significant correlations between handwriting and visual–motor coordination (Barnhardt, Borsting, Deland, Pham, & Vu, 2005; Brossard-Racine, Majnemer, Shevell, Snider, & Bélanger, 2011; Cornhill & Case-Smith, 1996; Daly, Kelley, & Krauss, 2003; Parush, Lifshitz, Yochman, & Weintraub, 2010; Volman, van Schendel, & Jongmans, 2006; Weintraub & Graham, 2000), the evidence is inconclusive as to whether the VMI is appropriate for use as an outcome measure after handwriting interventions.
Occupational therapy practitioners in both school-based and private practice commonly implement handwriting interventions and programs to improve participation and occupational performance in school and community environments. When implementing handwriting interventions, practitioners often break the task down into its core components, teaching each component separately before teaching the task as a whole. One of the primary difficulties with teaching handwriting is that the research is unclear about exactly what components make up handwriting. Clearly, a combination of visual and motor skills is required (Cornhill & Case-Smith, 1996). Cornhill and Case-Smith (1996) wrote that
visuomotor integration seems to be an important variable to a child’s handwriting skill, particularly when copying or transposing from printing material to cursive or manuscript writing. In copying, the child must visualize the letter form and shape, assign a meaning to the form, and then manipulate a writing tool to reproduce the same letter. (p. 734)
Research has consistently identified significant correlations between visual–motor integration (as measured by the VMI) and handwriting. The correlational research has strong evidence demonstrating moderate relationships between the VMI and handwriting measures (Barnhardt et al., 2005; Brossard-Racine et al., 2011; Cornhill & Case-Smith, 1996; Daly et al., 2003; Parush et al., 2010; Volman et al., 2006; Weintraub & Graham, 2000), although some authors have been quick to recommend caution in interpreting correlational studies (Goyen & Duff, 2005; Tseng & Cermak, 1993). Twenty years ago, Tseng and Cermak (1993) wrote, “Authors have assumed that if there is a correlation between performance on the component and handwriting, that problems in the component underlie the handwriting problem, and that remediation of the component will result in improvement in handwriting” (p. 924). Current research has interestingly continued to rely heavily on correlational methods to examine visual–motor integration and handwriting. More recently, Goyen and Duff (2005) advised caution when interpreting these correlations, stating, “Correlational studies reflect an association between visual–motor integration and handwriting, as expected considering the nature of the task, and do not imply causation” (p. 111).
Although research has demonstrated that VMI scores are a predictor of handwriting legibility (Brossard-Racine et al., 2011; Cornhill & Case-Smith, 1996; Volman et al., 2006), recent research has suggested that it is not an effective outcome measure for handwriting interventions (Howe et al., 2013; McGarrigle & Nelson, 2006; Poon, Li-Tsang, Weiss, & Rosenblum, 2010). McGarrigle and Nelson (2006) found that the VMI was not sensitive in detecting change in visual–motor coordination after a 6-wk occupational therapy intervention. They noted changes in handwriting, but no significant change in VMI scores. Bazyk et al. (2009) conducted a study with a longer intervention period, and the results were similar. After 7 mo of occupational therapy intervention, children with disabilities showed significant improvements in fine motor skills, pencil grip, and literacy test scores; however, improvements in VMI scores were not statistically significant in this group (Bazyk et al., 2009). Significant changes in VMI scores were noted in the typically developing students involved in this study (Bazyk et al., 2009). In a more recent study by Howe et al. (2013), children improved in handwriting legibility after occupational therapy intervention but did not improve in visual–motor integration skills postintervention.
Although the available literature seems to indicate that the VMI should not be used as an outcome measure when examining handwriting and related interventions, further research is needed to confirm these results. To implement best practices and determine evidence for occupational therapy interventions, measures that are appropriate and sensitive enough to determine outcomes are essential. Therefore, the purpose of this study was to provide more definitive information on whether a widely used assessment of visual–motor skills, the VMI, is appropriate for use as an outcome measure for handwriting interventions.
Method
Research Design
This study was part of a larger study designed to investigate the effect of the Size Matters Handwriting Program (SMHP) on handwriting legibility in kindergartners, first graders, and second graders. A two-group pretest–posttest design was implemented at two public schools in Massachusetts and New York. In Massachusetts, students were randomized by classroom assignment to either a treatment or a nontreatment control group by simple random selection. In New York, a convenience assignment of classrooms was used because of teacher availability and administrative factors.
Procedures
Institutional review board approval was obtained, as well as consent from the parents of all participants. Teachers leading the experimental groups were provided with fidelity manuals and participated in an in-service on proper implementation of the SMHP before the start of the intervention. Experimental groups received the SMHP intervention for forty 20-min sessions over a period of 8 wk. The control groups participated in their usual handwriting instruction in the classroom. The Test of Handwriting Skills–Revised (THS–R; Milone, 2007), Minnesota Handwriting Assessment (MHA; Reisman, 1999), and VMI were administered pre- and postintervention, and a demographic form for each student was completed at pretest. All measures used standardized scores; raw scores were converted to standard scores provided by the test manuals on the basis of the child’s chronological age at the time of test administration. A single occupational therapist scored all VMI assessments to ensure interrater consistency. This therapist was blinded to student demographic information and condition (experimental vs. control group and pretest vs. posttest) while scoring.
Measures
Developmental Test of Visual–Motor Integration, Sixth Edition.
The VMI is a standardized, norm-referenced assessment involving copying geometric forms that is used to determine the level of integration between visual and motor systems in people of all ages. The VMI has high content and person reliability, with total group item separation of 1.00 and total group person separation of .96. The VMI’s interscorer reliability for the children’s norming group is .93. No studies have established validity for the sixth edition of the VMI, although concurrent, predictive, and content validity were established for previous versions. The 21-item Short Form was used in this study because it is designed for use with children ages 2–7 yr. It was administered in a group format in the classroom.
Test of Handwriting Skills–Revised.
The THS–R is a comprehensive handwriting assessment standardized for administration to children ages 6–18 yr. Although it includes both manuscript and cursive booklets, only manuscript writing was assessed for our purposes. The THS–R was selected because it is a widely used handwriting assessment in school-based practice. Published reliability coefficients for the THS–R range from .59 to 1.00, with a mean of .82. Test–retest reliability ranges from .75 to .90. Simple yet common writing tasks were chosen to minimize confounding variables to establish content validity. Construct validity studies for the THS–R considered developmental changes and exceptional group differences and compared mean subtest scores, mean ancillary scores, standard scores, and error scores between varying populations.
Permission was sought from the authors to adapt the THS–R test booklet to include writing lines. This modification was desired because the premise of the intervention was the primacy of letter size in legibility. Accuracy in letter size was defined as having letter line contours touch the writing lines without projecting under or over them. All scorers participating in this study were required to watch a 2-hr training video before they were deemed qualified to assess the nuanced differences between printed letters that would earn a score of 0, 1, 2, or 3.
The THS–R has nine subtests, but only those subtests assessing the target of the handwriting intervention were administered for each grade. For kindergarten students, only those subtests that assessed uppercase letters were administered; for first-grade students, subtests assessing lowercase letters were administered; and for second-grade students, subtests assessing both upper- and lowercase letters were administered. Detailed instructions as well as illustrations for scoring are provided in the manual (Milone, 2007). Because our sample population included students who were 5 years old, the tables referenced were for the youngest norms available, that is, age 6.
Minnesota Handwriting Assessment.
The MHA is a popular norm-referenced assessment of manuscript writing. Interrater reliability ranges from .87 to .99, and intrarater reliability ranges between .97 and 1.00. Test–retest reliability for legibility is .62; for alignment and size, it is .89. Content validity of the MHA was based on a review of the literature and input from teachers. The MHA demonstrated moderate to strong correlations (.37–.76) with the VMI in studies of concurrent validity and discriminated students who were receiving occupational therapy from those who were not. Scorers participating in this study were all given the same sample test to score and needed to achieve at least these levels of reliability before continuing. Six different criteria are assessed during this test: rate, legibility, form, alignment, size, and spacing. Scoring guidelines are provided in the manual (Reismann, 1999). The MHA is norm referenced for the second half of first grade and all of second grade; therefore, it was administered to only the first and second graders in this study.
Analyses
To assess whether the VMI is a useful outcome measure in handwriting studies, we (1) used paired t tests to examine whether experimental group participants reported significant over-time changes on the VMI, as they did on the THS–R and MHA (Pfeiffer, Rai, Murray, & Brusilovskiy, in press), and (2) used independent-samples t tests to examine whether the pre–post intervention change scores were significantly different for the experimental and control groups on the VMI, as they were on the THS–R and MHA (Pfeiffer et al., in press). In addition, to examine the strength of the relationship between the VMI and the THS–R and MHA measures, we ran a series of Pearson correlation coefficients.
Results
Demographics
Of the 207 children in the study, 55 (26.6%) were kindergarteners, 74 (35.7%) were first graders, and 78 (37.7%) were second graders. Slightly less than half of the sample, ranging between 41.8% of kindergarteners and 48.7% of second graders, were children from the New York school. The remaining students were from Massachusetts. The children were all between 5 and 8 years old. More comprehensive demographic information by grade is presented in Table 1. On the basis of χ2 tests, no statistically significant differences were found between the experimental and control groups on any of the demographic variables for the entire sample. Likewise, no significant differences were found between the experimental groups in each grade.
Demographic Characteristics of the Sample at Baseline
Note. No significant baseline differences were found between the experimental and control groups on any variable for kindergarten, first-grade, and second-grade students and for the whole sample. IEP = individualized education program; OT = occupational therapy.
VMI: Group Differences and Changes Over Time
Table 2 shows mean VMI scores for children in the experimental and control groups in each grade level and at each time point; the over-time changes in VMI scores are also presented. Paired t tests showed that the over-time changes in VMI scores were not significant in the experimental group, meaning that despite receiving the intervention, postintervention scores of children in that group were not significantly higher than their preintervention scores.
Group Differences in Standardized VMI Scores Pre- and Postintervention and Changes Over Time
Note. M = mean; SD = standard deviation; VMI = Beery–Buktenica Developmental Test of Visual–Motor Integration.
Positive means indicate increases in scores over time; negative means indicate decreases in scores.
Table 2 also includes results of independent-samples t tests, which show that the experimental and control groups were not significantly different in terms of VMI scores at both time points. Similarly, independent-samples t tests show that the changes on the VMI from the preintervention time point to the postintervention time point were also not significantly different in the experimental and control groups. These findings show that the experimental and control groups had similar VMI scores before the intervention, but even after the experimental group received the intervention, their VMI scores (and the over-time change in the VMI scores) were not significantly different from the corresponding scores in the control group. All these findings hold for the entire sample and for each grade level separately.
Changes on the Handwriting Measures
Table 3 shows mean changes in pretest and posttest scores on the THS–R and MHA. As shown in the table, the over-time changes on most subscales of the THS–R and MHA were significant in the experimental group; similarly, the changes on several subscales were also significant in the control group. For most of the subscales, the changes in the experimental group were significantly higher than those in the control group.
Difference in Differences: Comparing Change Over Time in the Experimental and Control Groups
Note. From “The Effectiveness of the Size Matters Handwriting Program,” by B. Pfeiffer, G. Rai, T. Murray, and E. Brusilovskiy, in press, OTJR: Occupation, Participation and Health. Used with permission. Positive means indicate increases in scores over time, whereas negative means indicate decreases in scores. MHA = Minnesota Handwriting Assessment; THS–R = Test of Handwriting Skills–Revised; VMI = Beery–Buktenica Developmental Test of Visual–Motor Integration.
Paired t test shows that within-group over-time change is significant: Bonferroni-adjusted p value (p × number of tests) < α = .05. Here, the number of tests = 60 (30 in experimental group and 30 in control group).
Bonferroni-adjusted p value, calculated as min[(unadjusted p × number of tests), 1]. Here, the number of tests is 30.
Cohen’s d (Cohen, 1988) provided as measure of effect size. Cohen considered ds = .2, .5, and .8 to be small, medium, and large effect sizes, respectively.
Correlations Between the VMI and the Handwriting Measures
Table 4 shows Pearson correlation coefficients between the VMI scores and the MHA and THS–R handwriting scores at baseline, for the whole sample and by grade level. Even though many of the correlations were statistically significant, they ranged in size from .05 to .41, indicating that the correlation between the VMI and the handwriting measures was small to moderate.
Pearson Correlation Coefficients Between Preintervention Standardized VMI Scores and Handwriting Measures
Note. MHA = Minnesota Handwriting Assessment; THS–R = Test of Handwriting Skills–Revised; VMI = Beery–Buktenica Developmental Test of Visual–Motor Integration.
Discussion
The results of this study did not support the use of the VMI as an outcome measure for assessing changes after a handwriting intervention. These results are consistent with previous research using the VMI as an outcome measure after handwriting and related interventions (Howe et al., 2013; McGarrigle & Nelson, 2006; Poon et al., 2010). In each of these cases, the VMI was not sensitive enough to measure changes in handwriting and related visual–motor integration skills after occupational therapy interventions. In addition, results of this study support previous research identifying correlations between the constructs of visual–motor integration (as measured by the VMI) and handwriting. Numerous studies examining relationships between the VMI and handwriting have demonstrated moderate correlations (.30–.70; Barnhardt et al., 2005; Brossard-Racine et al., 2011; Cornhill & Case-Smith, 1996; Daly et al., 2003; Parush et al., 2010; Volman et al., 2006; Weintraub & Graham, 2000). This study’s findings reveal small to moderate correlations between VMI scores and handwriting test scores (THS–R and MHA). Although the literature has consistently supported some level of correlation between VMI scores and handwriting, results of this study reinforce the need to incorporate additional measures when making decisions regarding which students will qualify for services and when attempting to measure changes resulting from these interventions.
If the VMI and handwriting are correlated, as the literature suggests, it is curious that handwriting outcome measures in this study, and others, show consistent gains after interventions, and VMI scores do not (Howe et al., 2013; McGarrigle & Nelson, 2006). There are likely several reasons for this, but the most obvious is that the VMI measures a related but different construct from handwriting. Children in the current study’s handwriting intervention regularly learned, practiced, and copied letters. They were most likely not practicing and copying shapes. Most shapes in the VMI, after the triangle, are unique, complex, and novel. These shapes get progressively more difficult and are not shapes children typically draw. Motor learning theory emphasizes the need for consistent practice for permanent learning to take place and for a skill such as handwriting or shape copying to become automatic (Poole, 1991). If a child has only one attempt to copy a shape that he or she has seen only once before (in pretest conditions), chances of error are much higher than would be expected if the child had practiced the shape.
If, however, the purpose of the VMI is to determine whether a child demonstrates age-appropriate visual–motor integration skills (rather than to measure motor learning or handwriting skills), then practicing before posttest would not be appropriate. Although this study did not identify significant changes in VMI scores after handwriting interventions, certainly further research is needed to determine whether different types of handwriting interventions may use methods that have a more direct impact on visual–motor integration skills than a handwriting program founded in motor learning theory, such as the one used in the current study.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The results of this study add to the current research suggesting that the VMI may not be an effective outcome measure for handwriting interventions (Howe et al., 2013; McGarrigle & Nelson, 2006).
If the VMI had been the only posttest measure used in this study to examine the outcomes of the intervention, the results would have suggested that the children did not demonstrate progress, although handwriting measures including the THS–R and MHA demonstrated significant handwriting improvements.
It is important to ensure that measurement tools are appropriate for, and sensitive to, the construct being measured, which in turn must be the construct that the intervention is addressing.
Although visual–motor integration is consistently correlated with handwriting, the VMI was never intended to assess handwriting ability, nor was it designed to screen specifically for handwriting dysfunction (Chang & Yu, 2009; Goyen & Duff, 2005).
The literature has consistently supported some level of correlation between VMI scores and handwriting, lending some support to the continued use of the VMI as a tool to measure the component skills of visual–motor integration and to help determine which children could benefit from further assessment and treatment. However, because correlations in this review were small to moderate, additional forms of assessment are suggested, including handwriting, fine motor, and environmental assessments, as well as clinical observations, when making determinations regarding treatment and progress.
Footnotes
Acknowledgments
Publication of this article was partially supported by the U.S. Department of Education, National Institute on Disability and Rehabilitation Research (Grant H133B100037). We thank the therapists and teachers who supported the implementation of the study in their schools.