Motor Assessment With the STEGA iPad App to Measure Handwriting in Children

Study Overview

This was a cross-sectional single-arm study involving a single laboratory visit at an academic research institution. All participants gave informed consent, and all procedures were approved by the Institutional Review Board at Washington University in St. Louis School of Medicine. Data were stored and managed with the Research Electronic Data Capture system (Harris et al., 2009).

Participants

Participants were 57 typically developing children (34 girls and 23 boys; M age = 10.4 yr, SD = 0.7, range = 9.0–11.8). Inclusion criteria were as follows: Grade 4 or 5 (or, in summer, recently completed Grade 4 or 5); English speaking; right-handed, as defined by an Edinburgh Handedness Survey (Oldfield, 1971) score ≥ 40; and ≥6 mo of cursive training in school (confirmed by curriculum information from the school if available; otherwise by self-report). Exclusion criteria included attentional disorders as determined by score on the Vanderbilt Assessment Scale (Wolraich et al., 2003); cognitive, sensory, or motor disorders as determined by custom survey; or uncorrected visual impairment worse than 20/30 as determined by the Snellen vision test (O’Brien et al., 2008). All individuals who responded to recruitment materials were screened, and all 62 individuals who passed screening were enrolled in the study and completed all portions; of those 62, data for 5 were discarded because they demonstrated a lack of knowledge of cursive handwriting (unable to attempt >50% of letters).

For this initial validation study, the criteria and task were chosen to maximize the potential for a strong relationship with motor skills. Cursive was chosen because of its high motor demands compared with manuscript (print) handwriting, and the age range (Grades 4–5) was chosen to include children who had begun to learn cursive but had not yet achieved ceiling performance.

The sample size was chosen on an a priori basis to detect a Pearson correlation of magnitude r ² = .25 (i.e., r = .5) between one kinematic variable from STEGA (speed, velocity smoothness, direction error, or position error) and handwriting letter legibility, after adjusting for other variables. A multiple linear regression analysis (Hsieh et al., 1998) at α = .05 would detect such a relationship with a power of .8 at a target sample size of 56.

Procedures

All participants completed STEGA, followed by the Evaluation Test of Children’s Handwriting–Cursive (ETCH–C; Amundson, 1995) and then a custom survey on their experience with STEGA. A parent observed the study through a one-way mirror.

STEGA is an app developed by PlatformSTL (St. Louis, MO) and is based on a precision drawing task with a successful track record in neuroscience research (Philip & Frey, 2014, 2016; Philip et al., 2021). Participants used an iPad 6th Generation and Apple Pencil 1st Generation to draw within the bounds of abstract symmetrical shapes, as illustrated in Figure 1. Participants were instructed to draw as quickly as possible while staying within the bounds. To focus the current study on other innovations, drawing shapes were reused from previous studies (Philip & Frey, 2014, 2016) and are shown in Figure 2. Each shape consisted of two to four parts (lines or semicircles 45 mm long). To minimize cognitive demands, each shape was preceded by a short animation indicating the order in which the parts should be completed, and during drawing the shape parts were illuminated in that order. Participants completed 30 trials (15 shapes at two difficulty levels: 5 mm or 6 mm tolerance), plus two practice trials (8 mm tolerance; data not recorded). STEGA collected raw data at 50 Hz, including pen position (0.5 mm precision), position and angular errors (deviation from ideal line, in millimeters and degrees, respectively), and time (in milliseconds). On the basis of these data, five primary measures were used or calculated: four measures at 50 Hz (position error, angular error, pen velocity, and time since session start) and velocity smoothness, which had one value per 45-mm shape part and was defined as the number of submovements per unit distance (−1 × velocity peaks per shape part). No time limit was enforced, but time was incorporated into the analyses through the velocity and time since session start measures.

OPEN IN VIEWER

Figure 1.

STEGA iPad app: (A) The user draws on the iPad screen with the Apple Pencil; (B) the user draws a line (green), with gray dots and shading indicating the upcoming path; and (C) screenshot of a completed shape.

OPEN IN VIEWER

Figure 2.

Schematic of STEGA shapes, each of which could contain two parts (A), three parts (B), or four parts (C).

Handwriting data were collected via the ETCH–C. Legibility was assessed by two raters (Elizabeth Hawkins-Chernof and a nonauthor research coordinator) according to the ETCH–C manual (Amundson, 1995). The two raters first scored legibility independently, and then came to a consensus on their judgments; all analyses were performed on this final consensus. The primary measure was ETCH–C’s letter legibility (converted to a 0–100 score) because of its known reliability, validity, clinical cutoffs (Duff & Goyen, 2010), and high interrater reliability in scoring (Dennis & Swinth, 2001).

The feedback survey assessed participant reactions to STEGA’s user interface with two app development questions, which are not reported here because they were not designed for scientific relevance. The feedback survey also included an open-ended question (“Would you like to share any other feedback about the app?”), the results of which are summarized later to quantify the fraction of participants who had negative feedback on or comments about the app.

Data Analysis

For the current proof-of-concept validation study, the goal was to identify whether a relationship exists between STEGA data and handwriting performance, so an exploratory approach was used with the goal of finding at least one standard analytic approach that could identify such a relationship. Specifically, five analytic approaches were used to predict ETCH–C letter legibility from STEGA: linear regression (Yan & Su, 2009) and four machine learning methods. In brief, the four machine learning methods were (1) deep belief networks, the use of artificial neural networks to detect patterns in data (Hinton et al., 2006); (2) random forest, an ensemble learning model using the bagging technique, which performs regression and classification by averaging a set of parallel decision-tree models based on random subsets of the training data (Breiman, 2001); (3) XGBoost, an ensemble learning model using the gradient boosting technique, in which the decision trees are trained sequentially to reduce the prediction error from prior trees (Chen & Guestrin, 2016); and (4) support vector regression (SVR), which predicts variables by identifying a hyperplane in a high-dimensional space that fits a subset of data points (Drucker et al., 1997).

Statistical significance was detected at α = .01 to provide Bonferroni correction for five comparisons (0.05/5). Each participant’s data were modeled as 812 features: age + sex + (15 shapes × average of 3 parts/shape × 2 difficulty levels × 9 variables). An example of a feature would be “position error in the 5-mm version of Shape 9’s first part.” The nine variables were time (since session start) and the mean and standard deviation of angular error, position error, velocity, and velocity smoothness. For each analytic approach, the number of features was chosen via the following method: Up to 50 features (except for deep learning, which could use all 812) were ranked by univariate correlation with letter legibility (highest to lowest absolute Pearson correlation coefficient) and were then added to the model in that order. We selected the number of features that maximized model performance, separately for each approach. SAS 9.4 was used for the analyses.

STEGA Successfully Predicted Handwriting Performance

Our most effective analytic approach was SVR, which used STEGA’s precision drawing data to predict more than 40% of the interindividual variation in letter legibility (r ² = .437, p < .001), as illustrated in Figure 3. Mean real letter legibility was 73.1 (SD = 14.9, range = 38.9–95.3), and mean predicted letter legibility was 74.1 (SD = 7.6, range = 59.2–88.6), on 0–100 scales.

OPEN IN VIEWER

Figure 3.

Partial but successful prediction of handwriting performance from STEGA data.

SVR achieved its successful prediction by using 23 of the 812 features; the 23 most useful features came from five variables, as shown in Table 1. Notably, mean angular error was the most important variable, accounting for 48% of the features in our predictive model.

The 23 features in our prediction came from 10 of 15 possible shapes (67%), which means that 33% of our collected data did not contribute to the prediction. Age and sex were not among the features selected for the model. Overall, these data show that STEGA’s precision drawing data were sufficient to predict handwriting performance.

Our other analytic approaches were less successful than SVR, although some achieved statistical significance. Our baseline method, linear regression, did not reach statistical significance after multiple comparison correction (r ² = .100, p = .024, vs. Bonferroni corrected α = .01; best performance with five features), which indicates that the relationship between STEGA variables and handwriting is not well captured by a simple linear relationship. Our attempts to predict letter legibility via other machine learning methods also led to predictions that were worse than SVR; these weaker predictions achieved statistical significance for the random forest (r ² = .227, p < .001; best with 49 features) and XGBoost (r ² = .385, p < .001; best with 9 features) methods, but not for the deep belief networks method (r ² = .072, p = .044; with all 812 features). Deep belief networks’ relative inferiority was likely due to insufficient data points for training. The success of XGBoost (which met our a priori criterion of r ² ≥ .25) indicates that SVR’s successful prediction cannot be explained by method-specific artifacts. Moreover, SVR’s use of support vectors (a subset of all training data points) makes SVR less prone to overfitting than the other methods.

STEGA Provided Rapid, Well-Accepted Measurement

Children completed STEGA in an average 6.7 min (SD = 1.3, range = 4.2–10.6), whereas the ETCH–C took an average of 19.7 min (SD = 5.2, range = 11.5–33.1), reflecting a significant administration speed advantage for STEGA over traditional handwriting assessment, paired t test, t(112) = −18.8, p < .001. Of the 57 participants, the feedback from 4 (7%) included negative comments or criticisms of any aspect of STEGA. Therefore, in our current sample, STEGA provided a rapid, feasible, and well-accepted measurement of precision drawing performance.

STEGA Successfully Predicted Cursive Handwriting Performance

Fine motor control data from STEGA predicted almost half of the individual variability in cursive handwriting performance. A prediction of this magnitude is impressive for four reasons: (1) STEGA measures only fine motor control skills, whereas handwriting also includes nonmotor skills, such as letter knowledge and sustained attention; (2) previous motor assessments have achieved only 1% to 20% prediction (Kaiser et al., 2009; Overvelde & Hulstijn, 2011); (3) handwriting benchmarks have limited objectivity and validity (Feder & Majnemer, 2003; Rosenblum et al., 2003; Sudsawad et al., 2001), so a “perfect” measure of handwriting would still not correlate perfectly with existing handwriting tests; and (4) our sample size was insufficient for modern machine learning methods, such as deep learning. STEGA is not designed to perfectly capture handwriting performance: Its purpose is to measure fine motor control skills, and we have demonstrated here that STEGA’s rapid objective measurement of fine motor control also provides a useful predictor of handwriting performance. We anticipate that STEGA’s ability to predict handwriting performance will improve with the collection of additional data to support deep learning approaches to capitalize on hidden relationships between fine motor skills and handwriting.

STEGA’s successful prediction of handwriting performance also reveals the previously unquantified relationship between fine motor control and handwriting. As noted earlier, STEGA uses a precision drawing task to measure the fine motor control skills that are necessary, but not sufficient, for handwriting. Our results place a number on “necessary, but not sufficient” by demonstrating that fine motor skills accounted for almost half of the variation in handwriting performance, at least among our participants. The stronger-than-expected relationship between fine motor skills and handwriting confirms the use of motor assessment as a valid approach to quantifying handwriting difficulties among children and the use of handwriting to assess people with motor disorders such as Parkinson’s disease (Eichhorn et al., 1996; Phillips et al., 1991; Ünlü et al., 2006).

Finally, we identified that the most handwriting-relevant feature of precision drawing was the ability to control angular error; that is, pen direction (see Table 1). This relationship may be intrinsic to the task demands of writing and drawing: Whether a shape has been “correctly” produced is essentially a question of whether the pen tip was moved in the right directions at the right times. This reveals new knowledge about the motor skills that support handwriting among 9- to 12-yr-olds that will allow practitioners to target their motor assessments toward skills with the greatest functional impact.

STEGA Has Potential to Support Research, Assessment, and Screening

This study provided a proof-of-concept demonstration of STEGA’s potential to support research, rehabilitation, and screening via rapid quantitative assessment. STEGA will need additional features before widespread adoption (e.g., cloud-based secure data delivery), but with those features, STEGA’s ability to rapidly characterize drawing behavior could transform rehabilitation practice. For example, STEGA could potentially allow practitioners to identify the cause and nature of motor-related handwriting impairments, and it could provide appropriate intervention: STEGA could distinguish a child with atypical movement smoothness from a child with atypical movement speed and allow the practitioner to provide targeted therapy for each. For example, the practitioner could address movement smoothness deficits via letter formation in multiple media and materials and movement speed deficits via repetitive practice and motor learning (Volman et al., 2006).