Abstract
Low working memory (WM) capacity has been linked to poor academic performance and problem behavior. Availability of easy-to-administer screening tests would facilitate early detection of WM deficits. This study investigated the psychometric properties of the Working Memory Power Test for Children (WMPT) in 170 Australian schoolchildren (8½-11 years). Reliability (internal consistency) and validity of WMPT accuracy scores were examined. WMPT accuracy predicted achievement in reading, numeracy, and spelling. The results provide preliminary evidence of reliability and validity that supports interpretation of the WMPT accuracy score. With additional research, the WMPT could be valuable as an easy-to-administer screener for WM deficits.
Keywords
Working memory (WM) is the ability to hold and manipulate information relevant to a current task in the mind for brief periods of time (Baddeley & Hitch, 1974). WM has been described as a “mental workspace” used in everyday activities such as planning, mental arithmetic, and reading comprehension (Gathercole & Alloway, 2008). Although there is general consensus about the function of WM, there is less agreement regarding its structure. Competing cognitive models of WM range from simple, unitary models (e.g., Cowan, 1999) to more complex, multicomponent models (e.g., Baddeley & Hitch, 1974). Whether WM is a separate construct or isomorphic with short-term memory (STM) has also been questioned in psychometric models of cognition (McGrew, LaForte, & Schrank, 2014).
WM problems are relatively common, affecting approximately 10% of children (Alloway, Gathercole, Kirkwood, & Elliott, 2009). Children with poor WM capacity often fail to learn because the WM demands of a given activity exceed their capacity, which can result in poor academic progress (Holmes, 2012) and classroom behavior problems (Alloway et al., 2009). These children may struggle to retain the crucial information needed to guide ongoing activity, such as the sentence they are attempting to write (Gathercole & Alloway, 2008) or the math problem they are attempting to solve (Dumontheil & Klingberg, 2012; Lee, Ning, & Goh, 2014).
Given the links between WM, academic achievement, and problem behavior, it is important children with a WM deficit are identified early so appropriate intervention strategies can be implemented. The newly developed Working Memory Power Test (WMPT) for children has the potential to quickly and easily identify children with a WM impairment. In contrast to test batteries consisting of several subtests designed to assess the individual components assumed in a multicomponent model of WM, the WMPT consists of a single test, compatible with a unitary model of WM. An advantage of the WMPT over measures such as the Automated Working Memory Assessment (AWMA; Alloway, 2007) and the Working Memory Index (WMI; Wechsler, 2003) is that the WMPT is self-administered and automatically scored. The WMPT is unique in that it captures both accuracy and confidence information, allowing indices of children’s metacognitive monitoring ability to be calculated (see Freeman, Karayanidis, & Chalmers, 2017, for details).
The aim of this research was to investigate the psychometric properties of the WMPT in a large sample of Australian schoolchildren. First, we examined changes in accuracy and confidence in performance as the WM demands of the task increase. If the WMPT is a good test of WM that is sensitive to memory load, we would expect both accuracy and confidence in performance to decline as memory load increases (i.e., from Memorize to 1-, 2-, 3- and 4-Swap conditions). Second, we assessed the internal consistency of the WMPT. Third, concurrent validity was assessed using WMPT accuracy to predict academic achievement in reading, numeracy, and spelling. Finally, we examined the relationship between WMPT accuracy and performance in digit and spatial span tasks.
Method
Participants
Participants were 170 Australian schoolchildren (84 males) aged 8 years 8 months to 11 years 1 month (M = 9 years 10 months). 1 All spoke fluent English and came from a broad range of socioeconomic status (SES) backgrounds. Participants were recruited from six schools. School principals were provided with an Information Statement describing the research. If they chose to participate, Information Statements and Consent Forms were distributed to parents of children in Year 4. Children in Year 4 were chosen as they could complete the WMPT with minimal supervision and because testing would not interfere with the national curriculum testing, which takes place in Years 3 and 5 in all Australian schools. 2 All children whose parents gave written informed consent participated in the study. That is, there was no additional screening of children beyond the judgment of the parents as to their suitability to participate. Ethics approval was granted by the University of Newcastle Human Research Ethics Committee.
Measures
WMPT
The WMPT is an online assessment used to measure children’s WM performance. The WMPT uses nonverbal stimuli, and onscreen task instructions are provided in simple English, making it suitable for children from different language backgrounds and abilities. It is designed so children can complete it independently or under supervision. The WMPT has five levels of increasing difficulty: Memorize, 1-, 2-, 3-, 4-Swap. In the Memorize condition, children are presented with three pictures (line drawings of familiar animals) and asked to remember the order of the pictures in the display. They are then shown an answer screen displaying the same three pictures in various orders and instructed to select (by mouse click) the option showing the correct sequence of pictures. Subsequent levels involve mentally swapping the order of the pictures (e.g., “Swap 2 and 3”) and then selecting the option showing the correct sequence after the swap has been made. After each test trial, the children indicate their confidence whether they answered the preceding question correctly, using a 4-point scale. There are 25 trials, five at each level. Testing time is 15 to 20 min.
Wechsler Individual Achievement Test–Second edition, Australian abbreviated (WIAT-II). 3
The WIAT-II is a paper and pencil achievement test assessing word reading, numerical operations, and spelling (Wechsler, 2007). Reliabilities of .96, .91, and .91, for word reading, numerical operations, and spelling, respectively, have been reported. Test scores have been shown to predict a range of achievement measures using an Australian sample (Wechsler, 2007). Testing time is 20 to 30 min.
Span tasks
Digit and spatial span were assessed using the Digit Span and Dot Matrix subtests of the AWMA (Alloway, 2007). In Digit Span, the participant hears a string of digits and repeat them, in the same order, to the experimenter. In Dot Matrix, the participant views a sequence of dots presented in a 4 × 4 matrix and then point to the location in which the dot had appeared, in the order it was presented. Each task commences with one item per trial and increases by one item at each level of difficulty. Progression to the next level requires four out of six correct trials. Test–retest reliability scores of .89 and .85 have been reported for Digit Span and Dot Matrix, respectively (Alloway, 2007). Testing time is 5 to 10 min for each task. 4
Procedure
Children were tested individually in a quiet space at their school. The experimenter (one of four experimenters trained in administration of the tests) introduced each task and remained present throughout testing. The order of presentation of the WMPT, span tasks, and WIAT-II was counterbalanced. Children completed the WMPT on a laptop computer. Memorize, 1-, and 2-Swap conditions started with two practice problems. The span tasks were presented by laptop, with the children’s responses recorded by the experimenter, and computer scored. Each span task began with three practice trials. WIAT-II was administered and scored using standardized procedures. Testing took from 2 to 4 days (spread over 1-2 weeks) per school.
Data Analysis
WMPT confidence and accuracy scores were calculated for each level of difficulty and converted to percentages. Total accuracy was calculated as the percentage correct over all 25 trials. Children (N = 3) who did not complete all levels of the WMPT were given a 0 score for noncompleted levels. Repeated measures ANOVA was used to examine WMPT accuracy and confidence as a function of level of difficulty (Memorize, 1-, 2-, 3-, 4-Swap). Statistical significance was evaluated against an alpha level of .05. The Greenhouse–Geisser correction (Greenhouse & Geisser, 1959) was applied when the assumption of sphericity was violated. Post hoc analyses using Bonferroni-corrected (α = .01) paired-samples t tests were used to determine the significance of differences between adjacent levels of difficulty. Internal consistency (Cronbach’s α) was calculated using WMPT accuracy scores over all trials.
Three separate multiple linear regression analyses were conducted to determine the predictive ability of gender, age, and WMPT accuracy on achievement in word reading, numeracy, and spelling. Pearson correlations were used to examine the relationship between WMPT accuracy and digit and spatial span. Principal axis factor analysis was used to examine whether WMPT accuracy and digit and spatial span scores reflect one or more underlying constructs. All assumptions for these analyses were met.
Results
As shown in Figure 1, WMPT accuracy and confidence decreased with increasing WM load, from Memorize to 4-Swap conditions. Separate one-way repeated-measures ANOVA confirmed the decrease in accuracy, F(3.5, 588) = 169.63, p < .001, η2 = .50, and confidence, F(2.8, 468) = 108.52, p < .001, η2 = .39, was statistically significant, with large effect sizes (η2 > .14) for both analyses (using the Cohen [1988] guidelines). Paired-samples t tests comparing adjacent levels of difficulty were all significant, for both accuracy and confidence (all p < .01). 5 Internal consistency (Cronbach’s α) was .85.
Mean percentage (and standard error) for accuracy and confidence for each level of the Working Memory Power Test for children.
Concurrent validity was examined using WMPT accuracy (averaged over levels of difficulty) to predict academic achievement measured by the WIAT-II. Mean standard score (and standard deviation) for WIAT-II subtests were 101 (10.6), 98 (13.6), and 103 (15), for word reading, numerical operations, and spelling, respectively. Three multiple linear regressions—with gender, age in months, and WMPT accuracy as predictors and WIAT-II word reading, numerical operations, or spelling raw scores as the criterion—were conducted. The results are presented in Table 1. WMPT accuracy explained between 12% and 35% of the variance in achievement scores. The contribution of gender and age was not significant in any of the models.
Regression Results for Gender, Age, and WMPT Accuracy Predicting Word Reading, Numerical Operations, and Spelling Scores.
Note. WIAT-II = Wechsler Individual Achievement Test–Second edition; WMPT = Working Memory Power Test.
p < .001.
Pearson correlation analysis revealed significant correlations between WMPT accuracy and digit span, r = .23, p = .003, and WMPT accuracy and spatial span, r = .39, p < .001, with small and medium effect sizes, respectively (Cohen, 1988). Principal axis factor analysis showed WMPT accuracy, digit span, and spatial span loaded on a single factor (with eigenvalue >1), explaining 56% of the variance (loadings of .48, .48, .81), suggesting these tasks assess a single construct. Descriptive statistics for variables used in the regression and correlation analyses are reported in Supplement Table 1.
Discussion
The results showed both accuracy and confidence decreased as WM load increased from the Memorize (lowest WM load) to 4-Swap (highest WM load) conditions. The decrease was larger for accuracy than confidence. The increasing difference between confidence and accuracy as task difficulty increases suggests children are more likely to overestimate their ability when performing harder tasks.
Children’s self-assessment of their ability is thought to influence their motivation and persistence in learning (Chapman & Tunmer, 2003); however, little is known about the accuracy of children’s self-assessment. The availability of confidence ratings in the WMPT enables the accuracy of children’s self-assessments to be examined. Further research is needed to assess how these indices relate to children’s motivation to learn.
The results provide preliminary evidence for reliability and concurrent validity of WMPT test scores. WMPT accuracy predicted academic achievement, explaining a significant proportion of variance in each of the WIAT-II subtests. These results are consistent with previous research showing WM performance can predict numeracy (e.g., Dumontheil & Klingberg, 2012; Lee et al., 2014) and literacy (e.g., Siegel & Ryan, 1989; Swanson, Kehler, & Jerman, 2010). The importance of the present findings is that we have shown a single test score, WMPT accuracy, can account for a significant proportion of variance across three areas of academic achievement, reading, mathematics, and spelling, in the one study.
WMPT accuracy was correlated with digit and spatial span. The higher correlation with spatial span is likely due to the visuospatial characteristics of both tasks. The weaker correlation between WMPT accuracy and digit span, a verbal task, is consistent with the observation that children often used verbal rehearsal when performing the WMPT swap trials.
Factor analysis showed WMPT accuracy, digit span, and spatial span loaded on a single factor. This finding is consistent with a unitary model of WM (e.g., Cowan, 1999) and with previous research suggesting STM and WM are isomorphic (e.g., McGrew et al., 2014).
The WMPT has a number of advantages over some commonly used measures of WM. The task is primarily nonverbal, with minimal reliance on prior knowledge. It is easy to administer, can be completed online, and requires minimal supervision. In contrast to test batteries such as the AWMA and WMI, which require one-on-one administration, responses are recorded automatically. Together, these qualities suggest the WMPT could be suitable as an initial screener for detecting WM problems in a classroom setting. Follow-up assessment could then be conducted to ensure poor performance was not simply the result of the child not being engaged with the task. However, further research that examines diagnostic accuracy is needed to determine whether the WMPT accuracy score has discriminative value as a screener for WM problems.
Whereas the present results provide support for the validity of the WMPT as an assessment of WM in children, it is important to acknowledge the limitations of the study. Participants were Australian schoolchildren ranging in age from approximately 8½ to 11 years. School principals and the children’s parents elected to participate in the study. The extent to which these results generalize to other study populations needs further investigation. Having established that children from 8 years 8 months can complete the WMPT with minimal supervision, the next step is to establish the lower age at which children can complete the test independently.
The one-on-one administration required for the WIAT-II and span tasks meant testing within a school could not be completed within a single session. As suggested by a reviewer, it is possible that children may have discussed their experiences in completing the tasks with their peers. Although this may have occurred, we found no evidence to suggest this impacted their performance (i.e., there were no order effects).
In conclusion, we have provided preliminary evidence of reliability and validity that supports the interpretation of the WMPT accuracy score. With further research to establish the generality of these findings, the WMPT could be suitable as a preliminary screening test suitable for use in a classroom setting.
Footnotes
Acknowledgements
The authors thank the children, parents, and schools whose participation made this research possible.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was partially supported by the NSW (Australia) Government Trade and Investment TechVoucher program.
Notes
Supplemental Material
Supplementary material is available for this article online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
