Abstract
This study confirms the validity and reliability of two screening scales for handwriting issues: the Handwriting Legibility Scale (HLS) and the Concise Assessment Scale of Children’s Handwriting (BHK), both of which are recommended for occupational therapy practice.
Handwriting is generally described as a complex perceptual–motor human activity that includes motor planning, visual–motor coordination, cognitive and perceptual abilities, and tactile and kinesthetic sensitivity (Feder & Majnemer, 2007; Maeland, 1992; Reisman, 1993; Rosenblum et al., 2003; Šafárová et al., 2020). Therefore, it is necessary to manage several cognitive and fine motor tasks simultaneously to form a written text. These tasks include syntax, grammar, ideation, text production, spelling, and punctuation. Other factors involved in the handwriting process are visual and sensory perception, in-hand manipulation, sustained attention, orthographic–motor integration, self-monitoring, and evaluation (Feder & Majnemer, 2007; Jones & Christensen, 1999; Rosenblum & Dror, 2017; Rosenblum et al., 2003). In this context, the typical symptoms of handwriting issues (HIs) are defined as developmental delays or disruptions of the aforementioned processes.
The prevalence of HI estimates has varied across studies from 7% to 30% (Karlsdottir & Stefansson, 2002; Katusic et al., 2009; Maeland, 1992; Rubin & Henderson, 1982). It depends mostly on the ages of the assessed children and the method of assessment.
The prevalence estimates of specific learning disorders in the Czech Republic vary from 3% to 5% (Kejřová & Krejčová, 2015; Zelinková, 2003). A previous study concerning HIs in the Czech Republic showed that handwriting issues occur among 28.87% of the population (Šafárová et al., 2020). Moreover, prevalence is two or three times higher for boys (Katusic et al., 2009; Snowling, 2005) because the quality of their handwriting (legibility) is worse (Hawke et al., 2009; Medwell & Wray, 2007; Šafárová et al., 2020).
Children with HIs have illegible, irregular, or inconsistent handwriting. Specific symptoms include lack of consistency in letter size (Rosenblum, 2018; Rosenblum et al., 2004; Smits-Engelsman & Van Galen, 1997), slant, and word spacing (Asselborn et al., 2018; Klein & Taub, 2005; Rosenblum, Chevion, & Weiss, 2006; Rosenblum, Dworkin, & Weiss, 2006; Rosenblum et al., 2004; Smits-Engelsman & Van Galen, 1997). Such children have trouble forming correct letter shapes (Rosenblum et al., 2004; Simner & Eidlitz, 2000 ; Smits-Engelsman & Van Galen, 1997). They frequently erase and correct already written text (Engel-Yeger et al., 2009; Rosenblum, 2018; Rosenblum et al., 2004) and are unable to keep the text in a straight line (Rosenblum et al., 2004; Rosenblum & Dror, 2017). Their writing is unsteady, with higher pressure on the surface. They have poor spatial planning abilities; therefore, they cannot stay within the margins (Engel-Yeger et al., 2009; Rosenblum 2018; Rosenblum et al., 2010). Children with dysgraphia have difficulties recalling a letter because of poor orthographic coding in their working memory. Their ability to memorize and produce correct letter shapes is impaired, leading to subsequent difficulties in text recall and copying (Döhla & Heim, 2016; Romani et al., 1999; Rosenblum & Dror, 2017). Another typical symptom is the slow speed of handwriting (Tseng & Chow, 2000) and the disrupted automation process (Tucha et al., 2008). Sometimes, during handwriting, an unusual wrist, body, or paper position is present. This improper posture causes fatigue or pain and, subsequently, an aversion to writing (Engel-Yeger et al., 2009; Pokorná, 2010; Rosenblum et al., 2003; Zelinková, 2015). However, it is generally accepted that an incorrect grip does not influence the development of HIs (Burton & Dancisak, 2000; Graham & Weintraub, 1996; Schwellnus et al., 2012).
HIs in children are not related exclusively to the quality and quantity of handwriting itself (Graham, 1990). These problems are also related to a child’s academic achievement, which is conditioned by handwriting skills. Some studies have addressed the relationship between a child’s school grades and their handwriting neatness (Brackett et al., 2013; Briggs, 1980; Chase, 1986; Graham et al., 2000; Hammerschmidt & Sudsawad, 2004). These studies indicate that children with HIs receive worse grades in school than children without HIs.
Assessment of Handwriting
Two approaches to the assessment of HIs and their symptoms exist: product assessment and process assessment (Rosenblum et al., 2003). The first approach focuses on the final handwritten text, and the second approach uses modern technology (software, digitizers, etc.; e.g., Drotár & Dobeš, 2020; Gargot et al., 2020; Mekyska et al., 2019) to evaluate handwritten text. This new trend—as well as the product approach, which is usually needed as an external validation criterion—is considered important. Methods of evaluating the product could be divided into two groups: global and analytic. Global scales generally assess the overall impression from the text, with the main focus on legibility. Analytical scales assess legibility by compounding different items (i.e., slant, form, and size) with the outcome of a summative score.
Although some robust scales were developed to assess the quality and readability of handwriting (e.g., Bieber et al., 2016; Reisman, 1993), these scales have some limitations and thus could be less practical (Barnett et al., 2018; Falk et al., 2011). Major drawbacks are perceived in four areas (Feder & Majnemer 2003; Roston et al., 2008; Šafárová et al., 2021): (1) Assessment can be robust and time consuming (e.g., the Minnesota Handwriting Test and the Evaluation Tool of Children’s Handwriting; Amundson, 1995; Reisman, 1993); (2) methods have outdated or missing norms (e.g., the Children’s Handwriting Evaluation Scale and the Test of Legible Handwriting; Larsen & Hammill, 1989; Phelps & Stempel, 1988); (3) the scale is focused on a single aspect of handwriting (speed; e.g., the Detailed Assessment of Speed of Handwriting and the Handwriting Speed Test; Barnett et al., 2007; Wallen et al., 1996); and (4) scales can have different language modifications, with a majority in English or other languages (e.g., Hebrew; the Hebrew Handwriting Evaluation; Erez & Parush, 1999).
For this purpose, an alternative method called the Handwriting Legibility Scale (HLS) was designed (Barnett et al., 2018). The HLS is a relatively quick and easy-to-use holistic evaluation of a handwriting product; hence, it was meant to help teachers or other experts identify children with HIs (e.g., developmental dysgraphia or developmental coordination disorder). On the basis of the clinical experiences of the researchers and a verified review of the literature, this scale focuses on the overall impression of the evaluator. According to the authors, the HLS is language independent, and future research beyond the United Kingdom could validate its potential value for other languages (Barnett et al., 2018).
The clinical gold standard method and commonly used handwriting evaluation tool is the Concise Assessment Scale of Children’s Handwriting (BHK), developed by Dutch authors Hamstra-Bletz, de Bie, and De Brinker in 1987. The BHK, with its analytic approach, assesses 13 different aspects of handwriting of elementary school children, on the basis of cursive handwriting. Moreover, research has confirmed that the BHK is sensitive enough to evaluate various populations (Hamstra-Bletz et al., 1987; Jongmans et al., 2003). According to the latest studies, the BHK scale is traditionally used in French-speaking countries (Asselborn et al., 2018, 2020; Lopez & Vaivre-Douret, 2016; Moy et al., 2017; Vaivre-Douret et al., 2021). Designed for teachers, the Handwriting Proficiency Screening Questionnaire (HPSQ) was developed by Rosenblum (2008) and was later adjusted for children (HPSQ–C) to evaluate themselves (Rosenblum & Gafni-Lachter, 2015). Both questionnaires are focused on legibility, performance time, and physical and emotional well-being, covering the main three areas of handwriting difficulties. Previous research has showed that both instruments could be reliable and valid for screening handwriting deficits (Rosenblum, 2008; Rosenblum & Gafni-Lachter, 2015). Later, the HPSQ–C was adapted for the Czech population by Šafárová et al. (2020), who also verified its psychometric properties.
Furthermore, child self-reports are also central to child assessments. The results of child self-reports only slightly correlate with parental reports, because parents do not gain complete information about children’s internal processes or experiences. Therefore, it is vital to include children in the evaluation process, because they are often the best assessors of their own performance in specific activities (Bouman et al., 1999; Fram et al., 2013; Petersson et al., 2013; Sturgess & Ziviani, 1996).
Study Purpose
Many children struggle with handwriting during their school years. Despite this, there is a lack of research studies to define and quantify HIs, as well as a lack of practical assessment tools (Barnett et al., 2018). In the Czech Republic, there is no method for the assessment of HIs. The whole diagnostic process is based on the personal experiences of remedial teachers with no option of a reliable diagnostic method. For this reason, we consider it essential to expand the possibilities of screening tools used to detect children with HIs and to offer practitioners reliable and valid methods.
Another purpose of this study is exploring the relationship between objective diagnostic methods (the HLS and BHK) and children’s self-assessment (the HPSQ–C). This approach has not been used in any previous research concerning HIs. Moreover, the BHK, HLS, and HPSQ–C are language-independent methods, and the outcomes of this study can be used to inform other language adaptations (Barnett et al., 2018; Hamstra-Bletz et al., 1987; Rosenblum & Gafni-Lachter, 2015).
Although some psychometric studies have explored the factor structure of those questionnaires (HLS: Barnett et al., 2018; BHK: Hamstra-Bletz et al., 1987), none have confirmed them. Therefore, in this study, we aim to verify the psychometric properties of the HLS as a new alternative assessment focused on the overall impression of the evaluator and the BHK as the analytical gold standard, which is frequently adapted to other languages to assess HI. The HPSQ–C serves as the last vertex of our figurative triangulation, with the information about the child’s self-assessment independent from the product of the child’s handwriting.
On the basis of the previous drawbacks of the assessment of HIs and lack of evaluation tools in the Czech language, we focus mainly on the following: ▪ Construct validity of handwriting scales: The factor structure of the BHK corresponds to theoretical background; that is, it should have a one-factor structure (Van Waelvelde et al., 2012) or three-factor structure (Blöte & Hamstra-Bletz, 1991; Hamstra-Bletz et al., 1987). The factor structure of the HLS corresponds with theoretical background; that is, it should have a one-factor structure (Barnett et al., 2018). ▪ Reliability analysis of handwriting scales: The internal consistency (McDonald’s ω) of both questionnaires is higher than .7, which is considered an acceptable level. Interrater agreement (Krippendorff’s α) between two independent raters is higher or equal to .67. ▪ Discriminant validity of handwriting scales: The BHK and HLS differentiate between the total scores of girls and boys, who have higher scores. The BHK and HLS differentiate between children with typical handwriting development and children with HIs, who have higher scores. The higher the total scores on the BHK and HLS, the higher the average grade will be. ▪ Exploration of the relationship among the BHK, HLS, and HPSQ–C: Associations among methods are positive and higher than .6, which is considered a strong relationship.
Method
Study Participants
In this study, data from 163 participants were used. Two children were excluded from the sample because their mother tongue was not Czech (Ukrainian and French). Children were enrolled from state elementary schools and through state counseling centers in two districts of the Czech Republic. We collected data on the basis of the convenience sampling method. A total of 161 children from Grades 3 and 4 participated in the study. The range was chosen because the diagnosis of dysgraphia in the Czech Republic is usually given between Grades 3 and 4 when automaticity of writing does not occur. The mean age of the total sample was 9.56 yr (SD = 0.75; minimum, 8.00 yr; maximum, 11.00 yr). The groups of girls (n = 56) and boys (n = 105) were balanced in terms of age (boys: M = 9.57, SD = 0.74; girls: M = 9.53, SD = 0.76). The mean age of the Grade 3 group was 9.28 yr (SD = 0.59; minimum, 8.00 yr; maximum, 10.83 yr). The mean age of the Grade 4 group was 10.32 yr (SD = 0.45; minimum, 9.08 yr; maximum, 11.67 yr).
A total of 124 (82.7%) children with typical handwriting development (THD) were the control group, and 26 children (17.3%) were diagnosed with dysgraphia (i.e., with HIs). The THD-HI variable was missing for 11 children. Thus, for discriminant validity analysis, we used 150 data records from children. The diagnosis was provided by remedial teachers who work in special counseling centers. The sample was constructed to have the same ratio of boys to girls (Hawke et al., 2009; Katusic et al., 2009; Šafárová et al., 2020 ; Snowling, 2005) as in the population of children with HIs (Cermak & Bissell, 2014; Döhla & Heim, 2016; Šafárová et al., 2020).
Furthermore, children’s grades were used as a standard indicator of school achievement. Specifically, grades from the first and second half of the school year were used. These data contain grades for Czech language, mathematics, and English. In the Czech Republic, a grade of 1 is considered excellent achievement and a grade of 5 is considered deficient achievement.
The study was approved by the Masaryk University Ethics Board. Written informed consent to participate in this study was provided by each participant’s legal guardian. We followed the American Psychological Association’s (2019) Ethical Principles of Psychologists and Code of Conduct throughout the whole study.
Procedure
Transcription Task
The transcription task (TT) used in the study is part of a more extensive protocol for assessing HIs. The protocol was designed by a collaborating remedial teacher (the equivalent of an occupational therapist in the Czech Republic) who is a well-known expert in the field of special learning disabilities (Bednářová, 2017; Bednářová & Šmardová, 2006). The entire protocol was described by Šafárová et al. (2021). The text of the TT is three sentences long, contains 18 words composed of 90 letters, and was created for children from Grades 3 and 4. Moreover, it contains letters that are potentially difficult for children with HIs (e.g., cursive letters G, L, or S have similar shapes and loops). The TT is printed in block letters, and the child is asked to copy the text in cursive without any time constraints. The task is performed on a paper with lines, which follows conventions of the workbooks for Grades 3 and 4.
Concise Assessment Scale of Children’s Handwriting
The original language of the BHK is Dutch (Hamstra-Bletz et al., 1987), but other language standardizations exist, for example, Italian (Di Brina & Rossini, 2011), Swiss (Kaiser, 2012), and a shortened English version (Van Waelvelde et al., 2012). We used the French manual (Charles et al., 2003), which was recently standardized. The manual can be used for children from ages 6 to 10 yr. The BHK evaluates both the speed and quality of handwriting using 13 criteria. Each criterion of this scale assesses a different feature of handwriting: ▪ Item 1: Writing is too large ▪ Item 2: Widening of left-hand margin ▪ Item 3: Bad letter or word alignment ▪ Item 4: Insufficient word spacing ▪ Item 5: Acute turns in connecting joins to letters ▪ Item 6a: Irregularities in joins (break in the trace) ▪ Item 6b: Absence of joins ▪ Item 7: Collisions of letters ▪ Item 8: Inconsistent letter size (of x-height letters) ▪ Item 9: Incorrect relative height of the various kinds of letters ▪ Item 10: Letter distortion ▪ Item 11: Ambiguous letterforms ▪ Item 12: Correction of letterforms ▪ Item 13: Unsteady writing trace (Blöte & Hamstra-Bletz, 1991; Charles et al., 2003).
Items 1 and 2 evaluate the quality of the whole text on the 6-point Likert scale ranging from 0 (the best) to 5 (the worst). The remaining 11 items have binary scoring (0 = not present in the sentence; 1 = present in the sentence) for each sentence. Thus, the scores for each item range from 0 to 3 points. Total scores customized to the TT range from 0 to 43 points. A higher total score indicates HIs. Raters scored only the quality of handwriting.
Only three studies have explored the structure of this method: the original Dutch study with a three- factor structure solution (Hamstra-Bletz et al., 1987); a longitudinal study with a different three-factor structure, which explained 51% of the variance (Blöte & Hamstra-Bletz, 1991); and a shortened version (SOS) that explained 65% of the variance (Van Waelvelde et al., 2012). Individual factors with corresponding items with their meanings are displayed in Table 1. Item 6 was split into two different items in the longitudinal study but is assessed as one item in the original version and the SOS. The translated manual for this study works with merged Item 6 (i.e., 6a and 6b) as it is in the original and French manuals. Similarly, the confirmatory factor analysis of models was provided using the merged Item 6.
Items, Their Meanings, and the Factor Names of the Concise Assessment Scale of Children’s Handwriting According to Different Studies
The internal consistency of the BHK varies from .52 (Dutch study; Hamstra-Bletz et al., 1987) to .65 (Lebanese study; Matta Abizeid et al., 2017). The interrater reliability for the total BHK score can be considered high (Dutch: .71–.89; French: .68–.90; Lebanese: .92; SOS: .39–.77; Charles et al., 2003; Hamstra-Bletz et al., 1987 ; Matta Abizeid et al., 2017). The scores of the BHK correlate well with teachers’ evaluations of writing quality, suggesting an acceptable level of criterion validity (r = .78; Hamstra-Bletz et al., 1987). Concurrent validity of the French manual with Ajuriaguerra et al.’s (1964) scale for dysgraphia was significant (r = .68, p < .01; Charles et al., 2003). The SOS correlated with the original version of the BHK (r = .70, p < .01; Van Waelvelde et al., 2012).
Handwriting Legibility Scale
The HLS was designed for children ages 8 to 14 yr, with a major focus on legibility of the final written product. The scale has five evaluation criteria: global legibility, the effort required to read the script, layout on the page, letter formation, and alterations to the writing. Each criterion is assessed on a 5-point Likert scale ranging from 1 (good performance) to 5 (poor performance). The total score is computed as the sum of all items and ranges from 5 to 25 points, with higher scores indicating poorer performance (Barnett et al., 2018; Prunty & Barnett, 2017). During the assessment, handwriting speed and grammar mistakes are registered as well. Handwriting speed is measured in the number of words written per 6 minutes, including shortcuts and unfinished or crossed-out words. The number of grammar mistakes in the text is evaluated in contrast to the results of other children in the class. For this study, only the score for overall impression was used. The evaluated product of handwriting should be a “free-text” writing, with the length across 10 lines (Barnett et al., 2014).
Principal-components analysis determined the occurrence of one factor, which explained 61% of the variance of the five components. The discriminant validity analysis revealed that 89.7% of the typically developing children and 86.2% of the children with a developmental coordination disorder were assigned to the appropriate group according to the total HLS score. The outcomes of reliability analysis suggest a high internal consistency (α = .92) and interrater reliability (intraclass correlation = .92; κ = .67; Barnett et al., 2018).
Handwriting Proficiency Screening Questionnaire for Children
The HPSQ–C is a self-evaluation screening questionnaire with a three-factor structure: legibility, performance time, and well-being. The questionnaire contains 10 items; each is scored on a 5-point Likert scale ranging from 0 (never) to 4 (always). The total score as a sum of items ranges from 0 to 40. Higher scores indicate HIs. Rosenblum and Gafni-Lachter (2015) confirmed the reliability of the HPSQ–C and declared that the Cronbach’s α of the HPSQ–C is .77, indicating moderate reliability. However, the factor analysis of the HPSQ–C detected only two factors that together explain 45% of the variance. The first factor included Items 1, 2, 4, and 10 (legibility), and the second factor comprised Items 3, 5, 6, 7, 8, and 9 (performance time and well-being). The recent study of Šafárová et al. (2020) adapted the HPSQ–C to the Czech language and checked the validity and reliability of this method with a sample of children in Grades 3 and 4. The authors confirmed the construct validity by conducting a confirmatory factor analysis, which proved, unlike the original study, that the measure had a three-factor structure. They assumed that children understood that legibility, performance time, and physical and emotional well-being were independent variables, supporting the theoretical structure. The internal consistency of the HPSQ–C confirmed that the overall reliability could be considered acceptable (ω = .70). The individual subscales had the following McDonald’s ω values: legibility, ω = .67; performance time, ω = .46; and physical and emotional well-being, ω = .57. This study confirmed sex differences in the HPSQ–C total score and its relationship to grades, which proved the discriminant validity of this questionnaire. Girls evaluated themselves better than boys, with a medium effect size (d = 0.32). A significant weak relationship between the total HPSQ–C score and the average grade was found (r = .28).
In the present data set, the original three-factor structure of the HPSQ–C (Rosenblum & Gafni-Lachter, 2015; Šafárová et al., 2020) was confirmed again. The model shows good quality: χ2(32) = 46.71, p = .045; root-mean-square error of approximation (RMSEA) = 0.065; 90% confidence interval (CI) [0.008, 0.085]; standardized root-mean-square residual (SRMR) = 0.063; comparative fit index (CFI) = 0.973; TLI = 0.962. Strict reliability criteria were not met in this case, but values for internal consistency and interrater agreement are higher than in the previous study (legibility: ω = .769, ordinal α = .798; performance time: ω = .587, ordinal α = .568; well-being: ω = .611, ordinal α = .602).
Translation and Rating Process
The primary purpose of this study is to provide screening measures for teachers in schools and remedial teachers in counseling centers in the Czech Republic to assess HIs. Therefore, for the study, we used a forward-backward translation process for both questionnaires. The original language of the HLS is English, and the original language of the BHK manual is French. In the forward-translation step, both scales were translated by one of the authors of this study to the Czech language. The backward translation was carried out to translate the Czech version to English or French. The final versions were compared, with a minimal number of discrepancies. Additionally, during the whole process, we could consult the authors of both questionnaires regarding the meaning of items.
All raters, who worked with both measures in this study, were master’s students of psychology who decided to collaborate on research on HIs. Each rater assessed the reliability of both methods; afterward, one rater assessed construct validity. Scoring materials included (1) one sheet of paper from each child, containing the TT; (2) a uniform scoring table for each questionnaire; and (3) evaluation manuals for the BHK and HLS in the Czech language. Children from the study were identified with special codes, so the raters were unaware of their diagnosis. The evaluations, which were based on the manuals of the BHK and the HLS, were provided independently. In case of any discrepancies, raters were instructed to make notes of them.
Data Analysis
The main analyses were conducted by utilizing structural equation modeling (SEM), which allows incorporating variables as indicators of underlying theoretical constructs (latent variables), modeling measurement errors, and unexplained variances. We used confirmatory factor analysis (CFA) to verify the proposed factor structures of the BHK and HLS and polychoric correlations within SEM to analyze the relationships between grades and the BHK, HLS, and HPSQ–C. Because the data had an ordinal nature and showed multivariate nonnormality, we used the weighted least square mean and variance-adjusted estimator with a pairwise approach to handle missing values in all cases.
Because the number of participants per group was too low to use SEM for a discriminant validity determination, the comparison of groups within SEM (e.g., latent mean comparison with established scalar measurement invariance) was not possible (or feasible). Hence, we analyzed the differences between genders and between children with typical handwriting development (THD) and those with HIs, on the observed level (sums of subscales) using a set of t tests. The specific method was chosen on the basis of the achievement of its assumptions (i.e., the univariate normality and homogeneity of variance), which resulted in the usage of Student’s t test, Welch’s t test, or Mann–Whitney U test. For all t tests, p values were corrected by the Holm–Bonferroni method to reduce the risk of Type 1 error (first-ranked p = .025; second-ranked p = .05).
For reliability evaluation, the internal consistency of the scales’ measurement models was verified using McDonald’s ω. Because the BHK and HLS are not self-report methods but assessments that are based on an evaluation by a third person, three independent raters evaluated the same stimuli (first 40 participants). The interrater agreement of scores for such stimuli was estimated using Krippendorf’s α for ordinal data. To analyze the data, we used jamovi (Şahin & Aybek, 2019) and R (Version 4.1.1; R Core Team, 2021) with the packages semTools (Jorgensen et al., 2021), lavaan (Rosseel, 2012), and irr (Gamer et al., 2019).
Results
Construct Validity
Confirmatory Factor Analysis of the BHK
The CFA was tested on five different models. Model A tested the original three-factor structure (Hamstra-Bletz et al., 1987). Model B tested the structure of the BHK–SOS (Van Waelvelde et al., 2012). Model C tested a modified three-factor structure from the longitudinal study (Blöte & Hamstra-Bletz, 1991). Model D tested a one-factor structure. Model E tested the original three-factor structure as a hierarchical model with one general and two second-order factors. Indexes of how well they fit with other properties are presented in Table 2.
Model Indexes
Note. CFI = confirmatory fit index; CI = confidence interval; RMSEA = root-mean-square error of approximation; SRMR = standardized root-mean-square residual; TLI = Tucker–Lewis index.
However, Models A, C, and E showed high multicollinearity (rs ≥ .84, .90, and .91, respectively), and the last model did not even converge. Furthermore, Models A, D, and E showed shortcomings in their factor structure, especially in high SRMR values. Hence, the only suitable model is Model B. For this reason, Model B, which tested the BHK–SOS, is used in all other analyses.
Confirmatory Factor Analysis of the HLS
The original one-factor model of the HLS was tested (Barnett et al., 2018). The properties of this model are as follows: χ2(5) = 12.00, p = .035; RMSEA = 0.094; 90% CI [0.023, 0.163]; SRMR = 0.029; CFI = 0.998; TLI = 0.996. All factor loadings are statistically significant and higher than .70. Although the RMSEA slightly exceeds the recommended threshold of 0.080, this indicator might be biased in models with a small number of degrees of freedom (Kenny et al., 2015); therefore, this model displayed a good model fit, confirms the original study results, and supports the idea of the one-factor structure of the HLS.
Reliability Analysis
Internal Consistency
First, the internal consistency of the original three- factor structure of the BHK was tested. The results of McDonald’s ω and Krippendorf’s α are not very satisfactory: For the first factor, organization of written work, ω = .41, ordinal α = .53; for the second factor, letter formation, ω = .68, ordinal α = .78; and for the third factor, fine motor ability, ω =.55, ordinal α = .65. Compared with this, the BHK–SOS one-factor structure shows a little better result (ω = .63; ordinal α = .63). Thus, neither the original model nor the shortened model met the strict criterion of McDonald’s ω. The SOS met the ordinal α criterion, at least. Afterward, the consistency of the HLS was verified. The method met both criteria (ω = .88; ordinal α = .92). The indexes of overall internal consistency and values, if items were dropped, for both scales are listed in Table 3.
Item Analysis
To gain detailed information regarding why the internal consistency statistics were not satisfactory, we evaluated the item statistics. The analysis includes the original BHK, the BHK–SOS, and the HLS. All item– rest correlations and “if item were dropped” statistics are presented in Table 3.
Internal Consistency and McDonald’s ω if Items Were Dropped From the Concise Assessment Scale of Children’s Handwriting and the Handwriting Legibility Scale
Note. Overall internal consistency (McDonald´s ω) was .78 for the original BHK, .63 for the shortened BHK, and .89 for the HLS. BHK = Concise Assessment Scale of Children’s Handwriting; HLS = Handwriting Legibility Scale.
In the original BHK, Items 2 and 4 have the weakest correlation with the total score, and the internal consistency would improve if the items were dropped. The BHK–SOS does not include Item 2; therefore, the most problematic item was dropped. Repeatedly, Item 4 had the lowest rest-item correlation. The HLS item analysis showed the best result of all three methods. The internal consistency is high, and the item–rest correlation shows a close relationship. Correlations between items are moderate, which underscores their uniqueness and importance.
Interrater Agreement
We used the ordinal Krippendorff’s α as the index of interrater agreement. Neither the reliability of the original BHK (Krippendorff’s α = .61) nor that for the BHK–SOS (Krippendorff’s α = .57) met the criteria, but reliability for the HLS did meet the criteria (Krippendorff’s α = .72).
Discriminant Validity
Sex Differences
Boys (M = 11.61, SD = 3.41), compared with girls (M = 10.16, SD = 3.78), demonstrated significantly higher total scores on the BHK–SOS, t(159) = 2.47, p < .001; d = 0.41; 95% CI [0.13, infinity (Inf.)]. Welch’s t test shows statistically significant differences between boys (M = 10.90, SD = 3.42) and girls (M = 8.71, SD = 3.66) in HLS total scores, t(106.13) = 3.68, p < .001; d = 0.62; 95% CI [0.34, Inf.]). Results for both questionnaires supported the hypothesis that evaluations for boys would be worse than those for girls.
Differences Between Children With Typical Handwriting Development and Children With Handwriting Issues
Welch’s t test confirmed significant differences between children with HIs (M = 14.50, SD = 2.47) and children with THD (M = 10.44, SD = 3.46) when assessing handwriting with the BHK–SOS, t(48.12) = −7.06, p < .001; d = −1.35; 95% CI [−Inf., −0.93]. The Mann–Whitney U test was computed to find differences in the HLS total scores. Similarly, children with HIs had worse scores (M = 13.62, SD = 3.69) than children with THD (M = 9.37, SD = 3.21), and the Mann–Whitney U test showed significant differences: U(33.40) = 617.50, p < .001; d = −0.62; 95% CI [−Inf., −0.47]. The data support the hypothesis that children with HIs have higher total scores and, therefore, worse quality of handwriting than children with THD. These outcomes suggest that the BHK and HLS can differentiate between children diagnosed with HIs and children with THD.
Relationship Between Grades and BHK and HLS Total Scores
We gathered grades for the Czech language, mathematics, and English at the half-year and at the end of the year for each participant. Because these six items were highly correlated at the observed level, we decided to include them in a structural model as a latent variable representing the underlying construct called school achievement (rather than insert them into a model as separate, independently observed variables). This newly created scale showed high internal consistency (ω = .91; ordinal α = .95). The polychoric correlations between both scales and school achievement were estimated with SEM. This model showed satisfactory fit indices: χ2(116) = 166.175; p = .002; RMSEA = .052; 90% CI [0.033, 0.069]; SRMR = 0.087; CFI = 0.987; TLI = 0.985. The SRMR index reached the limit value, but in the context of other indexes of the model, we do not consider this a major complication in terms of model quality. This approach showed moderate associations between the BHK and school achievement (ρ = .568, p < .001) and a strong association between the HLS and school achievement (ρ = .725, p < .001).
Relationships Between the BHK, HLS, and HPSQ–C
Polychoric correlations were estimated with SEM. The model showed satisfactory fit indices: χ2(179) = 238.93; p = .002; RMSEA = 0.046 90% CI [0.029, 0.060]; SRMR = 0.077; CFI = 0.981; TLI = 0.978. We found strong relationships between the HLS, the BHK (ρ = .805; p < .001), and three factors of the HPSQ–C (see Table 4).
Correlations Between the Shortened BHK, HLS, and HPSQ–C Subscales
Note. BHK–SOS = shortened version of the Concise Assessment Scale of Children’s Handwriting; HLS = Handwriting Legibility Scale; HPSQ–C = Handwriting Proficiency Screening Questionnaire, adjusted for children.
p < .05.
p < .01.
p < .001.
Discussion
Handwriting is a complex activity that includes cognitive, perceptual, kinesthetic, and attention abilities, as well as language knowledge. Developmental delay or disruption of these abilities causes HIs, with prevalence ranging from 10% to 34%, especially among boys (Cermak & Bissell, 2014; Döhla & Heim, 2016; Hawke et al., 2009; Medwell & Wray, 2007; Šafárová et al., 2020). HI symptoms affect legibility of written text and writing speed and cause fatigue among children. Assessment of handwriting quality distinguishes two approaches: product assessment, which focuses on the final handwritten text, and process assessment, which focuses on the process of writing using new technology (e.g., software, digitizers; Drotár & Dobeš, 2020; Gargot et al., 2020; Mekyska et al., 2019). Several methods are used to assess HIs, but they have drawbacks: (1) The method may be time consuming; (2) norms may be outdated or missing; (3) the method may focus on a single aspect of handwriting; or (4) there may be different modifications in languages other than Czech, primarily in English. In the Czech Republic, there is no method for the assessment of HIs, and the diagnostic process is based only on the personal experiences of remedial teachers. A validation of a screening tool for the Czech environment is essential.
To overcome the drawbacks of previous methods, we chose three different scales: the holistic HLS, which was designed as a quick and easy-to-use method (Barnett et al., 2018); the analytic BHK, a widely used and well-accepted method that is considered a gold standard in the assessment of handwriting (Hamstra-Bletz et al., 1987); and the HPSQ–C self-report method, surveying a child’s point of view, as well as their well-being. Thus, it is not a handwritten text evaluation method for teachers or therapists (Rosenblum & Gafni-Lachter, 2015; Šafárová et al., 2020). In the present study, our aim was to confirm the factor structure of the BHK and HLS and verify their psychometric properties.
First, we used two hypotheses to analyze construct validity of both questionnaires by computing the CFA. The original study of the BHK used principal-components analysis to discover the three-factor structure of the method (Hamstra-Bletz et al., 1987). Similarly, a longitudinal study of handwriting structure found a three-factor structure (Blöte & Hamstra-Bletz, 1991). More recent studies simply worked with the original structure, and there has been no effort to confirm the structure with CFA (e.g.: Matta Abizeid et al., 2017). Thus, five different models were tested within the BHK scale. Best properties were obtained in a one-factor model of the shortened BHK called the SOS model (Van Waelvelde et al., 2012). Other models (original longitudinal, one-factor, or hierarchical) do not meet given criteria. Further item analysis showed two problematic items (Item 2 and Item 4). Item 2, which has the worst indexes, is not included in the SOS. This may be the reason why the SOS one-factor model fits better. The HLS has a one-factor structure (Barnett et al., 2018) and showed a good model fit in our data.
Reliability analysis includes the investigation of internal consistency (McDonald’s ω) and interrater agreement (Krippendorff’s α). The results of McDonald’s ω did not exceed the set threshold for the BHK–SOS. If Item 4 were dropped, the internal consistency would improve. Nevertheless, the improvement would be so little that it is better to keep the item (Van Waelvelde et al., 2012). Although the outcomes of the study showed lower internal consistency values, these results correspond with the values reported in the original version (.52–.65; Charles et al., 2003; Hamstra-Bletz et al., 1987) and the SOS (.39–.77; Matta Abizeid et al., 2017).
The problem with Item 2, mentioned in the “Item Analysis” section, could have been caused by poorly designed templates to gauge five different slopes of the text relative to the left margin of the page. Raters reported difficulties in deciding on the correct evaluation of this item, because the layout of the text seemed to suit two different slopes. This may cause some discrepancies in evaluation and therefore distort the results.
The internal consistency of the HLS was verified. The method met the strict criteria and therefore can be considered reliable. Item analysis of the HLS’s third item (layout on the page) shows the worst result. This corresponds to the results reported in the original study (Barnett et al., 2018). Moreover, the authors of the scale referred to some problems with the item-rest correlation and the “if item were dropped” consistency index. They assumed that this problem may be in the evaluation process, not the items themselves, and have suggested that more scoring examples may be needed (Barnett et al., 2018).
The value of Krippendorff’s α between raters is higher than the .667 for the HLS (Krippendorff’s α = .72), which means moderate agreement. The SOS does not meet the criteria. However, not even the original study shows better results (Charles et al., 2003). The French manual mentions the advantage of knowing the method. According to the outcomes of interrater agreement analysis, the authors of the manual reached an almost perfect agreement of .9, and inexperienced experts reached only 68% agreement. However, the official training recommendation is not given here; thus, the HLS could be considered easier to use, and having experience with it is not necessary.
The third set of hypotheses explored the discriminant validity of questionnaires in terms of sex differences and detection of HIs and in relation to school achievement. The results of the study confirm previous findings (Katusic et al., 2009; Snowling, 2005) that demonstrated differences between girls and boys. Boys gained higher total scores on the BHK, SOS, and HLS. On the BHK, we can see only a low effect; on the HLS, it is at least a medium effect.
Both methods differentiate children with typical handwriting development and handwriting issues. These results corroborate the findings of previous works on this topic (Barnett et al., 2018; Hamstra-Bletz et al., 1987; Jongmans et al., 2003). Thus, both methods can be recommended as HI screening tools.
Results showed positive relationships between the BHK, the HLS, and school achievement, represented by grades. Each method was correlated with three school subjects (Czech language, mathematics, and English language) in the middle and at the end of the year. All correlations were significant except one: the English grade at the end of the year. In general, it can be said that this hypothesis was confirmed, but those correlations ranged from weak (BHK–SOS) to intermediate (HLS). This differs from previous findings showing a strong relationship (Brackett et al., 2013; Briggs, 1980; Chase, 1986; Graham et al., 2000; Hammerschmidt & Sudsawad, 2004; Klein & Taub, 2005).
The last hypotheses concern the relationship between objective evaluation of the handwriting product, represented by the BHK and HLS total scores, and children’s self-evaluation, represented by factors of the HPSQ–C. The results indicate a strong relationship between the original BHK, the BHK–SOS, and the HLS, but weak correlations with three factors of the HPSQ–C (legibility, performance time, and physical and emotional well-being). The weak relationships among the HLS, BHK, and HPSQ–C suggest that children’s perception of their handwriting skills and HIs can be independent of parents’ or teachers’ opinions of the quality of children’s handwriting. The ambiguity of results can be caused by two possible scenarios in children’s well-being self-evaluation: ▪ The child has HIs, and their handwriting is objectively poor, but the child is satisfied enough. The process of handwriting is not burdensome, and it does not have a negative effect on the child’s well-being. ▪ The child does not have any HIs, and their handwriting performance is objectively good enough considering their age, but the child is not satisfied with their performance (e.g., because of excessive demands of parents or teachers), which negatively affects the child’s well-being.
Taken together, these results suggest that those questionnaires have different scopes. Evaluation of written text on the basis of the BHK and HLS is made by experts in the field of occupational therapy, whereas the HPSQ–C captures children’s internal experiences. These outcomes correspond with previous research about discrepancies between children’s self-evaluations and parents’ or teachers’ evaluations (Bouman et al., 1999; Fram et al., 2013; Petersson et al., 2013; Šafárová et al., 2020 ; Sturgess & Ziviani, 1996), and they emphasize the fundamental role of children in the diagnostic process.
Limitations and Future Research
The first limitation of this study is that the BHK and HLS were not used strictly according to the instructions given in their manuals. The HLS should be used to evaluate free text written by children, whereas the BHK supplies text, which consists of five short sentences. We used both of these methods to evaluate our TT. In light of our findings regarding construct and discriminant validity, and the reliability of the methods, we consider this to be a minor concession in using these methods.
Another limitation could be seen in the training of the raters or their prior knowledge of the methods. Our raters were not trained in using the scales. The French manual of the BHK claimed that the evident difference between raters depended on prior knowledge or experience with the method in the values of internal consistency coefficients. In spite of this, the authors did not require training of the raters. (Charles et al., 2003). We proceeded in accordance with the instructions. Moreover, it is more meaningful for practice. The internal consistency analysis in our results met the criteria even though the raters were not trained. It is additional evidence that the training of raters is not essential for the BHK scale. Similarly, the instructions of the HLS do not mention prior training or differences between inexperienced and experienced raters. Therefore, we had no information to compare.
An additional weakness of the study was sample composition. The group of children with HIs consisted mostly of boys. Males perform significantly worse than females at all ages in the quality of handwriting (Hawke et al., 2009; Katusic et al., 2009; Medwell & Wray, 2007; Šafárová et al., 2020 ; Snowling, 2005), which means that “normal” quality of handwriting can be different for boys and girls. Despite being consistent with the distribution in the population, it could affect some results. In future standardization studies, genders should be balanced in experimental (HI) and control (THD) groups, allowing the creation of normative data. Gender-specific norms of handwriting quality can prevent the overdiagnosis of handwriting issues in boys, because they will not be compared with girls who are better at handwriting in general.
More validation studies in different language environments are needed to verify the functionality of the BHK and HLS. Our results support the hypothesis of the linguistic independence of these methods (Barnett et al., 2018; Hamstra-Bletz et al., 1987), but it needs to be confirmed by future research. The language independence issue that was not addressed in this study is the depth of the orthographic coding process. The HLS is originally an English measure, and the BHK is a Dutch and French measure. All these languages are orthographically deep, whereas Czech belongs to the orthographically shallow languages. The effect of language depth has not been studied and could be usefully explored in further research.
The relationship among a child’s self-evaluation, parents’ or teachers’ opinions and influence on the child, and objective handwriting performance has not yet been explored and must be the subject of future research. Moreover, our results support previous disparities between the self-evaluations of children and evaluations by their parents or teachers. Therefore, in HI remediation, it is crucial to focus not only on improving the objective quality of the handwriting but also the child’s well-being.
Raters in this study scored only the quality of handwriting, omitting the speed. Handwriting speed depends on many variables such as context; the instruction given; and whether the child is copying, taking dictation, or free writing (Feder & Majnemer, 2007). In future research, we will focus on using modern technology, which allows us more accurate measurement of handwriting speed (Mekyska et al., 2019). With the use of digitizers, even small differences in speed can be recognized, and each factor’s influence can be examined. Thus, this approach could reveal more valid information.
Our aim in this study was to confirm the psychometric values of both questionnaires. For proper use of the BHK and HLS in practice, new standards, sensitivity, scaling, and other properties of examination are needed, for example, to explore whether the BHK or HLS would be useful as an outcome measure to evaluate changes in handwriting performance in the remediation process. The appropriateness and sensitivity of these tools require further analysis of test–retest reliability. In addition, comparing the changes in BHK and HLS scores with the changes of more detailed handwriting assessment tools (e.g., letter-by-letter analysis or temporal/spatial features obtained from a digitizing tablet) could be beneficial.
Implications for Occupational Therapy Practice and Research
This study has the following implications for occupational therapy practice and research: ▪ Both the HLS and BHK are useful screening instruments for the assessment of children’s handwriting skills. Given that the results of the CFA provide a better fit to the data for the HLS than for the BHK and that the correlation between the HLS and the BHK–SOS is very high (ρ = .805), the HLS is a better choice for practitioners. ▪ When addressing handwriting difficulties, practitioners should consider not only the objective quality of the child’s handwriting but also the child’s satisfaction and well-being. ▪ A discrepancy may exist between a child’s self- evaluation and that of parents or teachers, potentially influencing the child’s handwriting performance. Although further research is needed to examine this claim, occupational therapy practitioners should take this possibility into consideration.
Conclusion
In this study, we aimed to evaluate the psychometric properties of two methods used to diagnose handwriting issues—the BHK and the HLS—and their relationship to each other, as well as the HPSQ–C.
The factor structures, reliability, discriminant validity, and relationship between these methods were assessed. The quality of the factor structure was confirmed in the shortened version of the BHK (Van Waelvelde et al., 2012), as well as in the HLS (Barnett et al., 2018). Internal consistency was good in both measures, and interrater agreement of the HLS met the strict criteria. The BHK showed moderate interrater agreement, but these results are consistent with those of the original study. Both measures differentiate between boys and girls, distinguish between children with THD and children with HIs, and have a positive relationship to grades in basic school subjects.
A strong relationship between the BHK and HLS was found. However, both measures have only weak relationships to the HPSQ–C. The main reason may be the different aim of the product assessment scales compared with self-report questionnaires that are aimed more at the handwriting process. On the basis of our results, we can recommend both scales for practice, but more examination is needed. Further research should focus on confirmation factor structure in other languages, developing standards, and providing sensitivity studies. Moreover, a relationship to new computerized approaches is requested.
Footnotes
Acknowledgments
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. The data sets generated for this study are available on request from the corresponding author (Lukáš Čunek).