Development and Evaluation of an Online,Multicomponent Working Memory Battery

Abstract

Research has demonstrated strong connections among working memory (WM), higher-level cognition, and academic achievement. Despite the importance of WM, currently available WM tests have practical limitations and lack comprehensive coverage of multiple WM components. The Working Memory Battery (WOMBAT) includes nine subtests measuring multiple content domains and processing demands, in accordance with contemporary WM theoretical frameworks. The current study evaluated the WOMBAT factor structure and identified misfitting items using confirmatory factor analysis and Rasch modeling with scores from 125 adolescents and 177 adults (N = 302). Overall, results indicated the WOMBAT measures separate Verbal, Static Visual-Spatial, and Dynamic Visual-Spatial dimensions, and that more than 98% of items contribute to measurement of those dimensions. This provides support for the theoretical organization of WM into three distinct content domains in the WOMBAT. Misfitting items were identified using infit and outfit indices for further review to improve reliability and stability. Results also demonstrated adequate person separation and Rasch person reliability and item reliability. Test–retest reliability and internal consistency coefficients suggest adequate reliability for early-stage research, but further refinement is needed before the WOMBAT can be used for individual decision making. Implications for future test development and research on the WM construct are provided.

Keywords

Rasch model working memory cognitive assessment factor analysis psychometrics

Research has demonstrated working memory (WM) is one of the most important constructs in understanding higher-level cognition (Daneman & Carpenter, 1980; Engle, Kane, & Tuholski, 1999) and academic achievement (Alloway & Alloway, 2010; Dehn, 2008; Evans, Floyd, McGrew, & Leforgee, 2002; Gathercole, Alloway, Willis, & Adam, 2006; Gathercole, Tiffany, Briscoe, Thorn, the ALSPAC Team, 2005; Raghubar, Barnes, & Hecht, 2010; Swanson & Siegel, 2001; Swanson, Zheng, & Jerman, 2009). WM deficits have been implicated in numerous clinical conditions, including attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorders, learning disabilities (LD), Down’s syndrome, Williams syndrome, specific language impairment, and developmental coordination disorder (Alloway, 2011; Gathercole et al., 2006; Swanson, 1993). Swanson (1993) has presented evidence for the importance of understanding WM deficits in children with LD. Most concerning is that WM places constraints on children’s learning and acquisition of academic skills (Gathercole et al., 2005). This suggests that WM deficits in individuals with learning difficulties, if gone undetected, will continue to limit skill acquisition and achievement well past the early grades. Consequently, there is a strong need for the development of theoretically driven cognitive measures that can identify children who need help early in their schooling, before cognitive difficulties begin to affect academic performance (Dehn, 2008).

Fortunately, contemporary research demonstrates WM is malleable. Children identified as a -risk for learning difficulties in reading and/or mathematics due to WM deficits can be provided interventions specific to WM (e.g., Dehn, 2008; Holmes, Gathercole, & Dunning, 2009; Klingberg et al., 2005; Olesen, Westerberg, & Klingberg, 2007; Swanson & O’Connor, 2009; Thorell, Lindqvist, Nutley, Bohlin, & Klingberg, 2009; Westerberg & Klingberg, 2007). There is some evidence that traditional interventions that aim to teach the child WM strategies—such as chunking (Parente & Herrman, 1996), organizational devices, rehearsal training (Turley-Ames & Whitfield, 2003), and mnemonic devices (Mastropieri, Sweda, & Scruggs, 2000)—can improve memory and learning in children with LD (Mastropieri et al., 2000).

In addition, new approaches for directly intervening on WM are gaining in popularity and have begun to show evidence of effectiveness for improving WM—at least in the short term. These programs target increased WM capacity through computerized WM training (Jaeggi, Buschkuehl, Jonides, & Shah, 2011; Klingberg, 2010). Computerized WM interventions such as COGMED (Klingberg, 2010) have been shown to improve WM in children with ADHD (Holmes, Gathercole, Place, et al., 2009; Klingberg et al., 2005), to improve WM and reading comprehension in children with special needs (Dahlin, 2011), and to improve mathematical instruction memory and problem solving 6 months after training (Holmes, Gathercole, & Dunning, 2009). In addition, several studies have shown near transfer effects to WM tasks not included in the COGMED program itself (Holmes, Gathercole, & Dunning, 2009; Holmes, Gathercole, Place, et al., 2009; Jaeggi et al., 2011; Klingberg et al., 2005; Klingberg, Forssberg, & Westerberg, 2002; Thorell et al., 2009; Westerberg et al., 2007). More evidence is needed to support the generalizability and sustainability of treatment gains from these programs, but this area of research is promising.

Despite the demonstrated importance of the WM construct, there are limitations in currently available measures used to estimate WM ability. Most WM measures require significant time, training, and financial resources for individual administration. For example, the Swanson Cognitive Processing Test (Swanson, 1995) and the Working Memory Test Battery for Children (Pickering & Gathercole, 2001) must be individually administered by highly trained examiners and take an hour or more to administer to each examinee. The price of test kits and protocols for these paper-and-pencil batteries may also be prohibitive for some schools and educators.

In addition to practical limitations, currently available WM measures have theoretical limitations—most important, construct underrepresentation. For example, WM tests from major cognitive batteries such as the Wechsler Intelligence Scales for Children–Fourth Edition (Wechsler, 2003) and the Woodcock–Johnson III Tests of Cognitive Abilities (Woodcock, McGrew, & Mather, 2001) are largely verbal, with little representation of visual-spatial WM. However, major theories (e.g., Baddeley, 2000) and psychometric studies of WM have consistently suggested WM includes several related but distinct processing demands and content domains, including at the least a verbal and a visual-spatial component (Buehner, Mangels, Krumm, & Ziegler, 2005; Cornoldi & Vecchi, 2000, 2003; Ferreira, Almeida, & Prieto, 2011; Oberauer, Schulze, Wilhelm, & Süß, 2005; Oberauer, Süß, Schulze, Wilhelm, & Wittmann, 2000; Oberauer, Süß, Wilhelm, & Wittmann, 2003). While verbal WM has particular importance for reading (Daneman & Carpenter, 1980; Gathercole & Pickering, 2000; Leather & Henry, 1994), visual-spatial WM is predictive of mathematics achievement (Bull, Espy, & Wiebe, 2008; Swanson, 2006). Thus, comprehensive measures of WM should include different tests for verbal and visual-spatial WM.

As an additional limitation related to construct representation, few WM tests adequately measure visual-spatial WM components. Research suggests there are at least two types of visual-spatial WM: static or simultaneous visual-spatial WM (remembering static, simultaneously presented spatial locations of static stimuli) and dynamic or sequential visual-spatial WM (remembering dynamic sequences of spatial locations; Cornoldi & Vecchi, 2000, 2003; Kesner, Hopkind, & Fineman, 1994; Lecerf & de Ribaupierre, 2005; Logie, 1995; Mammarella, Borella, Pastore, & Pazzaglia, 2013; Mammarella et al., 2006; Pazzaglia & Cornoldi, 1999; Pickering, 2001; van der Sluis, van der Leij, & de Jong, 2005). Studies examining visual-spatial WM and achievement have not included a distinction for static or simultaneous versus dynamic or sequential visual-spatial WM tasks, but the empirical evidence from cognitive investigations (that did not include achievement) suggests such a distinction may be warranted. Inclusion of separate static and dynamic visual-spatial tasks within a comprehensive WM battery would allow for examination in future studies of possible links between each specific WM domain and different academic achievement areas. In addition, including more specific measures of all the WM components demonstrated by research provides a more comprehensive approach to WM assessment.

Finally, currently available WM tests, in addition to content domains, lack adequate measurement of different processing demands represented in theories of WM. Prominent WM theories (Baddeley & Hitch, 1974; Cornoldi & Vecchi, 2000, 2003; Engle, 2002), and empirical evidence suggest different storage (STM) and processing (WM) demands compose WM (Buehner et al., 2005; Ferreira et al., 2011; Oberauer et al., 2000; Oberauer et al., 2003). In addition to traditional “storage plus manipulation” tasks, such as Digits Backward, Engle (2002) has suggested that tasks including an “interference” component are necessary to comprehensively assess the WM construct. Both STM and WM functions—including those with an interference component (e.g., Daneman & Carpenter, 1980)—are important for reading and mathematics achievement (Swanson et al., 2009; Swanson & Sachse-Lee, 2001).

A multicomponent, theoretically organized measure of WM ability that includes tasks tapping all three domains (verbal, static visual-spatial, and dynamic visual-spatial) and demands (STM or storage, WM, and WM with an interference component) and can be efficiently administered and interpreted by school staff would allow more children with WM deficits to be identified for intervention before these deficits negatively affect academic achievement. In addition, such a measure would aid in intervention planning for children with specific WM problems (e.g., verbal WM deficits or static visual-spatial STM/storage deficits) that may differentially affect specific academic achievement areas.

The WOMBAT

The Working Memory Battery (WOMBAT) was developed to address the major limitations of currently available WM tests. The WOMBAT is a multicomponent, online measure of WM designed for use with school-aged children and adolescents, as well as adults. It combines the theoretical foundation of experimental WM measures used in laboratory settings with practical aspects suitable for school-based applications. The ultimate goals in the design of the WOMBAT are to help identify children with WM deficits and to guide intervention for individuals with these deficits, from elementary to adult levels. In addition, researchers can use the WOMBAT with children, adolescents, and adults to obtain a profile of participants’ WM strengths and weaknesses.

The WOMBAT was designed based on a theoretical model that comprehensively covers three WM content domains and three processing demands. In addition, it overcomes practical limitations of requiring intensive time, training, and financial resources for one-on-one administration through online administration and scoring. The WOMBAT (a) includes 180 items in nine subtests organized by a guiding conceptual model (see Figure 1), sampling across three domains and three demands from the WM literature and including tasks suggested by multiple WM theories (Baddeley & Hitch, 1974; Cornoldi & Vecchi, 2000, 2003; Engle, 2002) and (b) is administered completely online, with less reliance on highly trained examiners and accessible anywhere with an Internet connection. The current study focuses on psychometric evaluation of the WOMBAT using confirmatory factor analysis (CFA) and the Rasch model. CFA can help determine the dimensionality or factor structure—and therefore the appropriate level of interpretation—of the WOMBAT, while the Rasch model can help identify misfitting items to target during measure refinement.

Figure 1.

Organization of WOMBAT subtests along domain and demand dimensions. Each domain subscale (Verbal, Static Visual-Spatial, and Dynamic Visual-Spatial) includes three subtests that cover different demands from WM research.

Item Response Theory: The Rasch Model

Item response models are major advances over classical test theory approaches (Embretson & Reise, 2000). Researchers have stressed the need for applying Rasch models to measure development and validation to achieve psychometric support beyond what is available from traditional analyses, such as factor analysis (Smith Conrad, Chang, & Piazza, 2002). Previous studies of computerized and traditional paper-and-pencil WM tasks have examined item difficulty and other measurement characteristics using Rasch modeling (e.g., Ferreira et al., 2011; Pomplun & Custer, 2005). As described by DiStefano and Morgan (2010), Rasch modeling is a one-parameter item response theory model that uses scores obtained on an instrument to calculate the probability that an individual will correctly respond to a particular item, given the person’s ability and the difficulty of the item. The Rasch model is based on the premise that an instrument measures a single underlying dimension, and that items and persons can be arranged in order of difficulty and ability, respectively, along this dimension. The item and person rankings are converted from an ordinal to an interval scale using logarithmic transformation, such that item difficulty values and person ability values can be obtained. This allows for comparisons to be made across individuals (e.g., Person A has twice the ability of Person B on this latent trait), across items (e.g., Item B is twice as hard as Item A), and across time points—which is necessary for calculating meaningful change scores (e.g., Person A did twice as well after 2 weeks of intervention than she did before the intervention). In addition, the Rasch model provides fit indices that aid in identifying items that do not contribute to measurement of the underlying dimension or latent trait measured by the instrument.

The Current Study

The current study focuses on psychometric evaluation of the WOMBAT using CFA and the Rasch model. We address two research questions:

Does the factor structure of the WOMBAT reflect the theoretical organization of WM into three separate domains?

Which items contribute to measurement of the identified WM dimensions, and which items should be flagged for revision due to misfit?

The general dimension or latent trait hypothesized to underlie all items on the WOMBAT is WM; more specifically, however, we hypothesize that the WOMBAT measures three distinct dimensions of WM, as supported by previous literature and theory: verbal, static visual-spatial, and dynamic visual-spatial WM. Alternative models will also be tested to determine the appropriate level of interpretation for the WOMBAT. The best-fitting model will be used to structure Rasch analyses aimed at identifying misfitting items, which can be targeted for refinement to improve reliability and stability of the WOMBAT. This information will guide clinical interpretation of performance on the WOMBAT and inform future test development.

Method

Participants

Because the WOMBAT was designed for clinical, school, and research use with school-aged children and adolescents, as well as for research use with adults, both adolescents and adults were recruited from the University of South Carolina and local public schools. Adults from the University of South Carolina were recruited through the Department of Psychology Participant Pool and received course credit for participation, while participating high schools were recruited by phone and received monetary compensation to be used at the discretion of the principal for anything directly benefitting the school. Adolescents were selected instead of younger children because the WOMBAT is in the early stages of development and the online interface was pilot tested only with adults before the current study. In addition, adolescents and adults are closer to one another in terms of age and cognitive developmental level than are adults and elementary or middle school children. This allows for analysis of adolescents and adults as one sample.

Adults 18 and older gave online consent, and parental consent and online assent were required for minors. Inclusion criteria included the following: normal or corrected-to-normal hearing and vision, fluent English speaker, and not receiving special services for a cognitive disability at the time of test administration.

The total sample included 302 total individuals: 177 adults (age M = 22.97, SD = 4.73, Mdn = 21, mode = 21) and 125 adolescents (age M = 15.88, SD = 1.22, Mdn = 16, mode = 15). Adolescent participants all attended a midsize suburban high school in the southeastern United States. Adult participants were undergraduate or graduate students at the University of South Carolina, a large public university in the southeastern United States. Demographic characteristics for adolescents, adults, and the total sample are provided in Table 1. The total sample consisted of 55% (n = 166) women and 45% (n = 136) men and was 63% White, 29% Black, 4% Asian, 3% Other, and 2% Latino. According to 2012 U.S. Census Bureau data, the United States includes 50.8% women, 77.9% White only, 13.1% Black only, 5.1% Asian only, 1.2% American Indian/Alaskan Native only, 0.2% Native Hawaiian/Pacific Islander only, 2.4% two or more races, and 16.9% Hispanic or Latino (U.S. Census Bureau, 2013). Thus, in terms of demographics, women and Black participants were overrepresented, while men and Hispanic or Latino participants were underrepresented.

Table 1.

Demographic Characteristics of the Sample.

		Age in Years		Females		Males		White		Black		Latino		Asian		Other
	n	M	SD	n	%	n	%	n	%	n	%	n	%	n	%	n	%
Adolescents	125	15.88	1.22	47	37.60	78	62.4	56	44.80	65	52.00	1	0.80	1	0.80	2	1.60
Adults	177	22.97	4.73	117	66.10	60	33.90	133	75.14	22	12.43	5	2.82	10	5.65	7	3.95
Total sample	302	20.04	5.09	166	54.97	136	45.03	189	62.59	87	28.81	6	1.99	11	3.64	9	2.98

Data from adult and adolescent samples were analyzed together because of the proximity in age of adolescent to adult participants and to provide as diverse an ability range as possible to improve estimation capability for item calibrations with the Rasch model. Table 2 shows sample sizes and other descriptive statistics for individual subtests and domains. Depending on the subtest involved, some statistical analyses had sample sizes below the total sample size of 302. The online platform for the WOMBAT does not allow for skipping individual items, but it is possible that attempts to refresh the browser or log out during a subtest (despite instructions warning against this) may cause participants to show missing data for entire subtests. In the adult data, 28 of the total 177 participants had at least one missing subtest (M = 0.68, SD = 1.88, range = 0-8). In the adolescent data, 7 of the total 125 participants had at least one missing subtest (M = 0.22, SD = 0.91, range = 0-6). Thus, complete data for all subtests were available for 267 total participants. Participants’ number of missing subtests was not correlated with their average subtest raw score, r(302) = –.02, p = .0774, suggesting that missing data were unrelated to WM performance. Number of missing subtests was, however, correlated with administration order, r(302) = .99, p < .001. Data were most likely to be missing from the Interference demand subtests, which were administered last (seventh, eighth, and ninth) in the sequence of nine subtests for all participants.

Table 2.

Descriptive Statistics for WOMBAT Domain Subscales and Individual Subtests.

					Raw Score
Domain/Subtest	No. of Items	Item Numbers	Administration Order	n	M	SD
Verbal Domain	60	1-60		302	9.14	3.00
Digits Forward	20	1-20	1	302	10.68	2.55
Digits Backward	20	21-40	4	288	8.45	2.96
Digits Forward–Interference	20	41-60	7	281	8.18	2.85
SVS Domain	60	61-120		297	13.51	3.08
Dots	20	61-80	2	297	13.75	2.54
Dots Up	20	81-100	5	287	11.94	3.34
Dots–Interference	20	101-120	8	275	14.89	2.54
DVS Domain	60	121-180		292	8.53	2.17
Dots Sequence	20	121-140	3	292	9.11	1.97
Dots Sequence Backward	20	141-160	6	282	8.23	2.16
Dots Sequence–Interference	20	161-180	9	271	8.21	2.28

Note. DVS = Dynamic Visual-Spatial; SVS = Static Visual-Spatial; WOMBAT = Working Memory Battery.

The Rasch model (and WINSTEPS software; Linacre, 2005) is capable of estimating item difficulty values for each item with missing data from some persons. Thus, for item difficulty calibration and dimensionality analyses using the Rasch model, only participants with recorded responses for at least one subtest in the particular domain being analyzed were included in calibration for that domain. This resulted in sample sizes of 302 for Verbal, 297 for Visual, and 292 for Spatial domain analyses.

Instrument: The WOMBAT

The WOMBAT includes nine separate subtests, each its own WM task, organized by two parameters: WM demand and domain (see Figure 1). The demand parameter includes three types of tasks for different WM functions or processes: storage, storage + manipulation, and storage + interference. The domain parameter includes three types of content to be remembered and manipulated: verbal, static (or simultaneous) visual-spatial (SVS), and dynamic (or sequential) visual-spatial (DVS). There are three subtests for each domain, each a different type of task (a different WM demand). Subtests include the following: Digits Forward (Verbal storage), Digits Backward (Verbal storage + manipulation), Digits Forward–Interference (Verbal storage + interference), Dots (SVS storage), Dots Up (SVS storage + manipulation), Dots–Interference (SVS storage + interference), Dots Sequence (DVS storage), Dots Sequence Backward (DVS storage + manipulation), and Dots Sequence–Interference (DVS storage + interference). Table 3 provides task characteristics for each subtest.

Table 3.

Task Characteristics for WOMBAT Subtests.

Subtest Name	Domain	Demand	Stimulus	Response	String Length Range
Verbal Domain
Digits Forward	Verbal	Storage (STM)	Auditory (digits 1-9)	Motor (type digits 1-9)	2-11
Digits Backward	Verbal	Storage + Manipulation (WM)	Auditory (digits 1-9)	Motor (type digits 1-9)	2-11
Digits Forward–Interference	Verbal	Storage + Interference (WM-Executive Attention)	Auditory (digits 1-9, word and nonword sounds)^a	Motor (type digits 1-9, type letters Y or N)^b	2-11
SVS Domain
Dots	Visual (static)	Storage	Visual (nonmoving black dots in a square grid)	Motor (click boxes on screen)	1-10
Dots Up	Visual	Storage + Manipulation	Visual (nonmoving black dots in a square grid)	Motor (click boxes on screen)	1-10
Dots–Interference	Visual	Storage + Interference	Visual (nonmoving black dots in a square grid, 2 blue nonmoving dots)^a	Motor (click boxes on screen, click bigger dot)^b	1-10
DVS Domain
Dots Sequence	Spatial (dynamic)	Storage	Visual (moving black dots in a square grid)	Motor (click boxes on screen in specific sequence)	1-10
Dots Sequence Backward	Spatial	Storage + Manipulation	Visual (moving black dots in a square grid)	Motor (click boxes on screen in specific sequence)	1-10
Dots Sequence–Interference	Spatial	Storage + Interference	Visual (moving black dots in a square grid, 2 nonmoving blue dots)^a	Motor (click boxes on screen in specific sequence, click bigger dot)^b	1-10

Note. DVS = Dynamic Visual-Spatial; STM = Short-term memory; SVS = Static Visual-Spatial; WM = Working memory; WOMBAT = Working Memory Battery. String length = number of pieces of information (digits, dots, or dot locations) to be remembered. All subtests have 20 items and terminate when 4 consecutive items are answered incorrectly. All responses are scored as correct (1) or incorrect (0).

These stimulus types are presented only as part of interference questions and do not have to be remembered to respond to the question correctly.

These responses are part of the interference questions and are not scored.

Storage only tasks, which test STM, require the participant to recall auditorily or visually presented information in the same order as it is given. Storage + manipulation tasks require maintaining and transforming the provided information, such as reordering a digit sequence. Storage + interference tasks require maintenance of target information while processing unrelated distracter information in the same modality (verbal or visual-spatial) as the target information. All Verbal tasks involve information presented in auditory form—either digits or words. SVS tasks involve information presented simultaneously in static visual form, as an array of large black dots in a grid of square boxes. DVS tasks present arrays of dots similar to SVS tasks, but the dots appear and disappear, one by one, in a dynamic sequence rather than being presented as a single static array.

On each subtest, participants begin with items hypothesized to be the easiest (requiring the smallest amount of WM resources) and progress toward (hypothetically) more difficult items. A total of 20 items is included on each subtest; this can be organized as 60 items for each demand and each domain dimension, or 180 items on the entire instrument. Within subtests, item difficulty is operationalized as string length, or the number of pieces of information (digits, dots, or dot locations) that must be remembered. Participants begin with a string length of two digits (Verbal subtests), one dot (SVS subtests), or one dot location (DVS subtests), and the string length systematically increases as the items progress (see Table 3). Items are numbered beginning with number 1 as the first item in the Digits Forward subtest in the Verbal domain, and the numbers continue according to subtest order in the domain, rather than administration order (which can be changed for different studies). Table 3 shows item numbers for each subtest and domain.

Scoring

Responses to each item in all subtests are dichotomously scored. Participants receive an item score of 1 if the response is completely correct. For example, if the correct answer on a Digits Forward item is 3759, and the participant types 3759, the item is scored 1. If there are any errors—commission, omission, or sequencing errors—the item is scored 0. For example, a commission error on a Digits Forward item means the participant has typed an additional number that was not in the presented sequence (e.g., 37592 when the correct answer is 3759). An omission error means the participant has typed too few numbers (e.g., 375 when the answer is 3759). Sequencing errors mean the participant has typed the numbers in the incorrect sequence (e.g., 3579 when the answer is 3759). Any single error or combination of these errors is considered incorrect and scored 0.

On all subtests, administration automatically terminates when a ceiling of four consecutive incorrect responses is recorded. Thus, after four consecutive item scores of 0 on a particular subtest, the remaining item scores for that subtest are recorded as 0. There are also two unscored practice items on each subtest. Responses to interference questions on storage + interference subtests are recorded but not scored. Each subtest includes 20 total items, so the maximum raw score for a single subtest is 20 and the minimum is 0.

Test–Retest Reliability

A small subset of adults (n = 21, age M = 21.48 years, 8 men) was administered all nine subtests two different times, with an average test–retest interval of 2 weeks. For average WOMBAT subtest scores (mean of each participant’s subtest scores), the test–retest correlation was r = .83. Test–retest correlations for individual subtests ranged from .49 (Dots–Interference) to .88 (Dots) and are shown in Table 4. Nunnally and Bernstein (1994) have recommended reliability coefficients of .70 or higher for instruments in the early stages of research, but levels of .90 or higher as necessary for decision-making and individual assessment. The current results suggest the Digits Backward and Dots subtests (20 items per subtest) and the test as a whole have acceptable levels of test–retest reliability for early-stage instrument development, but that subtest-level test–retest reliability will need to be improved before the WOMBAT can be used for individual decision making. In the current study, misfitting items are identified as targets for revision to improve reliability of the WOMBAT.

Table 4.

WOMBAT Test–Retest Reliability and Internal Consistency.

Subtest	Test–Retest r^a	Cronbach’s Alpha^b
WOMBAT (test as a whole)	.83	.90
Digits Forward	.57	.75
Digits Backward	.74	.85
Digits Forward–Interference	.61	.83
Dots	.88	.76
Dots Up	.53	.81
Dots–Interference	.49	.75
Dots Sequence	.52	.66
Dots Sequence Backward	.59	.68
Dots Sequence–Interference	.54	.71

Note. DVS = Dynamic Visual-Spatial; SVS = Static Visual-Spatial; WOMBAT = Working Memory Battery.

n = 21. Average test–retest interval = 2 weeks.

n = 302.

Internal Consistency

Cronbach’s alpha was calculated for the test as a whole and for the nine subtests using item scores (0 or 1) to evaluate internal consistency (n = 302). The Cronbach’s alpha value for the test as a whole (all nine subtests) was .90. Cronbach’s alpha values for each subtest ranged from .66 (Dots Sequence) to .85 (Digits Backward) and are shown in Table 4. These values suggest acceptable levels of reliability for individual assessment and decision making for the test as a whole; however, more refinement is needed to increase reliability for the WOMBAT subtests for individual assessment (Nunnally & Bernstein, 1994). Again, misfitting items identified in the current study will be targeted for revision to improve internal consistency for the WOMBAT.

Procedure

The WOMBAT is administered and scored completely online without the need for an administrator. It can be administered in a school computer lab, under laboratory conditions with a research assistant supervising the participant, or on adult participants’ personal computers outside the laboratory. The current study included all three types of administrations. Only adults (older than 18) participated in administration outside a supervised school or lab setting. A total of 30 adults participated in supervised lab administration, and 147 adults used personal computers (home administration). Results of a one-way analysis of variance (ANOVA) showed no significant difference in average subtest scores between adults who participated in lab versus those who participated in personal computer administration, F(1, 175) = 0.60, p = .441. All 125 adolescent participants were administered the WOMBAT at their school under supervision by teachers and study staff in classrooms of 10 to 30 students simultaneously.

The WOMBAT includes an automated introduction and tutorial, plus nine separate subtests. The introduction includes general instructions and a brief keyboard practice session requiring participants to press designated letter and number keys, with error feedback. Participants were instructed during the test introduction to ensure access to headphones or speakers, a mouse, and a keyboard. They were also instructed to maximize their browser and adjust volume to a comfortable level. Then, the online system guided them through all nine subtests in the same sequence for every participant: Digits Forward, Dots, Dots Sequence, Digits Backward, Dots Up, Dots Sequence Backward, Digits Forward–Interference, Dots–Interference, and Dots Sequence–Interference. The entire battery of nine subtests takes approximately 30 to 50 minutes to complete.

Data Analyses

Factor Structure

Our first research question aims to determine whether the factor structure of the WOMBAT reflects the theoretical organization of WM into three separate domains. To address the hypothesis that the WOMBAT measures three distinct dimensions, we used CFA. Based on theory, we first tested a three-factor measurement model that included Verbal, SVS, and DVS factors (Model 1). In Model 1, each domain factor was modeled to directly influence three subtests: the Verbal factor influenced Digits Forward, Digits Backward, and Digits Forward–Interference subtests; the SVS factor influenced Dots, Dots Up, and Dots–Interferences subtests; and the DVS factor influenced Dots Sequence, Dots Sequence Backward, and Dots Sequence–Interference subtests. Two- and one-factor models were also tested for comparison: Model 2 included only a Verbal factor and a combined Visual-Spatial factor (SVS and DVS) and Model 3 included only one overall WM factor. Previous studies of the WM construct have also modeled verbal and visual-spatial content domains as latent factors influencing subtest performance (e.g., Buehner et al., 2005; Ferreira et al., 2011; Oberauer et al., 2000; Oberauer et al., 2003).

As previously described, each of the three hypothesized factors was measured by three types of subtests: storage, manipulation, and interference. To account for any variance due to common subtest types, we used a multitrait-multimethod (MTMM) model (Model 1b), as described in Kenny and Kashy (1992). To test the three-factor (Model 1) versus two- (Model 2) and one-factor (Model 3) models, we used correlated uniqueness models, where the subtests with common methods have correlated errors (or uniquenesses). This is a step down from a full MTMM, but is necessary to compare model fit, as the MTMM requires a minimum of three method and three trait factors.

Instead of using the sums of raw subtest scores, as in Oberauer et al. (2000, 2003), in the current investigation Rasch person ability estimates for each subtest were used as measures of performance. Ferreira and colleagues (2011) used person ability estimates to evaluate the structure of the WM construct using CFA. As argued by Ferreira and colleagues, using person ability estimates provides a linear scale for measuring performance (due to Rasch scaling) that is well matched for linear CFA approaches. The person ability estimates used in the current study were calibrated within each subtest for each participant. Each participant, then, had one person ability score for each subtest—a total of nine person ability scores for each participant (unless the participant had missing subtest scores). Missing data handling was needed because some participants had missing data at the subscale (subtest), rather than the item level. Although Rasch person ability scores can be calculated when the sample includes missing data at the item level, in the current study Rasch person ability was calibrated within subtests. This means that participants would have neither a raw score nor a Rasch person ability score for any missing subtest. Thus, each observed variable (subtest) had a different sample size, depending on the number of participants missing data for that subtest (see Table 2 for sample sizes for each subtest).

Linear CFAs were conducted using the maximum likelihood (ML) estimation method in Mplus 7.11 (Muthén & Muthén, 2013). Missing data were handled using the full information maximum likelihood (FIML) method. Parameter estimates obtained using FIML are generally superior to those yielded through listwise deletion or other ad hoc approaches (Schafer & Graham, 2002). To ensure that using Rasch person ability scores did not affect our choice of models, we also compared model fit using the raw data scores. Domain factors were allowed to covary, and one factor-loading for each factor was set to 1. Multiple indicators were examined to assess model fit: chi-square (χ²), Tucker–Lewis index (TLI; also called the nonnormed fit index or NNFI), comparative fit index (CFI), root mean square error of approximation (RMSEA), and standardized root mean square residual (SRMR). The chi-square test assesses the fit between observed and model covariance matrices, with smaller values indicating better fit (Macmann & Barnett, 1994); however, there is no consensus on an accepted standard for an acceptable chi-square value. Thus, chi-square statistics are reported here but not examined as primary indicators of model fit. According to Hu and Bentler (1999), good model fit is better indicated by the following values for continuous data: TLI greater than .95, CFI greater than .95, RMSEA less than .06, and SRMR less than .08. The Akaike information criterion (AIC) index was also examined to compare models, with lower values indicating better fit. The AIC accounts for degrees of freedom in the model, providing some statistical control for the possibility of finding good fit simply because a model is sufficiently complex.

Item Fit

Our second research question requires determining which WOMBAT items contribute to measurement of the WM dimensions identified through CFA, and which items should be flagged for revision due to misfit. To address this, we used the dichotomous Rasch model to calculate item fit. Item fit indices indicate the extent to which the behavior of an individual item fits the prediction of the Rasch model—namely, the expectation that all items will contribute to measurement of a single underlying dimension. Data were analyzed using WINSTEPS software (Linacre, 2005).

Item fit indices are expressed in unstandardized mean squares, which represent the average value of squared residuals for each item, calculated from the difference between Rasch-predicted item performance (i.e., difficulty or placement on the scale) and actual item performance in the observed data (Bond & Fox, 2001). Thus, larger mean square values represent poorer item fit to the Rasch model. Mean square fit values have an expected value of 1. Values less than 1 indicate item redundancy or model overfit, whereas values greater than 1 indicate unpredictability or model underfit. Two item fit indices were examined: outfit and infit. Typically, more attention is paid to infit than to outfit scores because infit is a weighted index while outfit is unweighted. The scores of persons with person ability values closer to the item difficulty value for a particular item are weighted more heavily than the scores of persons far from the item difficulty value when calculating mean squares for the infit index. For the current study, misfitting items were defined as those for which outfit mean squares were greater than 1.4 or for which infit mean squares fell outside the acceptable range of 0.60 to 1.40 (Wuang & Su, 2009).

Person Separation and Item Reliability

As an additional evaluation of WOMBAT consistency, we evaluated person separation and item reliability using the Rasch model. The person separation reliability statistic (PSI) is a Rasch measure of internal consistency similar to Cronbach’s alpha (Andrich, 1982). The PSI value indicates a test’s ability to separate persons into different levels of ability (Duncan, Bode, Min Lai, & Perera, 2003), in this case verbal, SVS, or DVS WM ability. A PSI of at least 2.00 means the test can distinguish three levels of person ability (high, average, and low). The PSI is used to calculate a person reliability index (as a percentage). The person reliability index provides a measure of how replicable the placement of persons along the logit scale would be if the same persons were administered another group of items measuring the same construct (Bond & Fox, 2001). Similarly, the item reliability index provides a measure of how replicable the placements of item difficulty values along the logit scale would be if the same items were administered to another sample with similar levels of person ability (Bond & Fox, 2001). Item and person reliability values range from 0 to 1 and are considered acceptable at values of .80 and greater (Fox & Jones, 1998). These indices provide evidence of how generalizable item parameters calculated from the current data are for similar person and item samples.

Results

Factor Structure

CFA was conducted using Mplus 7.11 to evaluate the factor structure of the WOMBAT. Table 5 provides zero-order correlations among subtests, which ranged from .10 to .58. Regarding independence of observations, it is possible that practice effects and/or strategy use or cheating occurred, but no systematic pattern of dependency was observed. Scatterplots of bivariate relationships among variables indicated the data approached multivariate normality. Regarding a positive definite covariance matrix, all eigenvalues were real, nonzero numbers, and no perfect correlations were observed. Thus, the data supported the assumptions needed for ML estimation in CFA.

Table 5.

WOMBAT Subtest Zero-order Correlations.

Subtest	1	2	3	4	5	6	7	8	9
1. Digits Forward	1.00
2. Digits Backward	.58**	1.00
3. Digits Forward–Interference	.45**	.56**	1.00
4. Dots	.20**	.26**	.15**	1.00
5. Dots Up	.27**	.27**	.18**	.54**	1.00
6. Dots–Interference	.10	.20**	.21**	.51**	.48**	1.00
7. Dots Sequence	.14*	.24**	.22**	.47**	.39**	.42**	1.00
8. Dots Sequence Backward	.21**	.36**	.27**	.44**	.49**	.49**	.45**	1.00
9. Dots Sequence–Interference	.14*	.25**	.25**	.21**	.29**	.27**	.38**	.37**	1.00

Note. N = 302. WOMBAT = Working Memory Battery. Correlations were calculated using Rasch-derived person ability estimates (not summed raw scores) for each subtest.

p < .05.

p < .01.

Model 1, the three-factor correlated uniqueness model, showed good fit to the data, χ²(15, n = 302) = 29.80, p = .013, TLI = .959, CFI = .983, RMSEA = .057, SRMR = .031, AIC = 10505.80 (see Figure 2). We also tested a two-factor correlated uniqueness model, Model 2, containing only two domain factors—verbal and a combined Visual-Spatial domain factor. Model 2 was tested using Mplus 7.11 with the same parameters used for Model 2 (domain factors allowed to correlate, one path coefficient for each factor set to 1). Model 2 also showed good fit to the data, χ²(17, n = 302) = 40.58, p = .001, TLI = .942, CFI = .973, RMSEA = .068, SRMR = .034, AIC = 10512.55 (see Figure 3). The AIC for Model 2 (10512.55) was higher than that for Model 1 (AIC = 10505.80), the TLI and CFI were both lower for Model 2 (CFI = .942, TLI = .973) than for Model 1 (CFI = .959, TLI = .983), and the RMSEA and SRMR were both higher for Model 2 (RMSEA = .068, SRMR = .034) than for Model 1 (RMSEA = .057, SRMR = 0.031), suggesting that the three-factor model is the better fitting model of the two. We compared the same models (1, 2, and 3) using the raw data and again found the three-factor model to provide a better fit to the data than the two-factor model.

Figure 2.

Model 1: Three-factor correlated uniqueness model tested with linear confirmatory factor analysis for Working Memory Battery subtests. This model predicted three domain factors (Verbal, Static Visual-Spatial [SVS], and Dynamic Visual-Spatial [DVS]) directly influence subtest performance. Rasch person ability estimates were used as measures of subtest performance. Model 1 showed good fit to the data, χ²(15, n = 302) = 29.80, p = .013, TLI = .959, CFI = .983, RMSEA = .057, SRMR = .031, AIC = 10505.80.

Figure 3.

Model 2: Two-factor correlated uniqueness model tested with linear confirmatory factor analysis for Working Memory Battery subtests. This model predicted two domain factors (Verbal and a combined Static Visual-Spatial and Dynamic Visual-Spatial factor [SVS + DVS]) directly influence subtest performance. Rasch person ability estimates were used as measures of subtest performance. Standardized estimates are reported here. Model 2 showed good fit to the data, χ²(17, n = 302) = 40.58, p = .001, TLI = .942, CFI = .973, RMSEA = .068, SRMR = .034, AIC = 10512.55.

Because the three-factor model fit the data better than the two-factor model, we were then able to test a full MTMM model (Model 1b), which also included three method factors, one for each of the three types of subtest: storage, manipulation, and interference. The MTMM model is generally preferred over the correlated uniqueness model (Lance, Noble, & Scullen, 2002). This model showed superior fit to the three-factor correlated uniqueness model on every fit statistic, χ²(13, n = 302) = 22.90, p = .043, TLI = .968, CFI = .989, RMSEA = .050, SRMR = .021, AIC = 10499.362 (see Figure 4).

Figure 4.

Model 1b: Full multitrait-multimethod model. This model showed excellent fit, superior to the three-factor correlated uniqueness model on every fit statistic, χ²(13, n = 302) = 22.90, p = .043, TLI = .968, CFI = .989, RMSEA = .050, SRMR = .021, AIC = 10499.362. Rasch person ability estimates were used as measures of subtest performance. Standardized estimates are reported here.

Because all items on the WOMBAT are designed to measure the WM construct, which could represent a single underlying dimension, a one-factor model, also with correlated uniquenesses (one general WM factor modeled to directly influence all nine subtests), was also tested. However, this model demonstrated poor fit to the data, χ²(18, n = 302) = 195.661, p < .001, TLI = .588, CFI = .794, RMSEA = .181, SRMR = .101, AIC = 10665.66. Therefore, results suggest that, as hypothesized, the three-factor model with Verbal, SVS, and DVS factors provides a better fit to the data in comparison to a one- or two-factor model.

Item Fit

An assumption of the Rasch model is that all items measure a single, underlying dimension or trait. The best-fitting model from the CFA included three factors, Verbal, SVS, and DVS, indicating the WOMBAT measures three distinct dimensions. Data were therefore analyzed for these three domains separately (i.e., treated as three separate instruments, each measuring a single underlying dimension) to calibrate item difficulty and person ability values, fit indices, and person and item reliability. Tables 6, 7, and 8 show item difficulty, error, and fit index parameters for the Verbal, SVS, and DVS domains, respectively. Rasch analysis of the 60 items within each domain revealed three misfitting items, according to the infit index (see Tables 6, 7, and 8). Two items were from the Verbal domain in the Digits Backward subtest (11, 16) and one (61) was from the SVS domain in the Dots subtest. All three items showed model underfit, or unpredictability (values > 1.4). This suggests that two items are not contributing to the measurement of the latent Verbal WM dimension and one item is not contributing to the measurement of the SVS WM dimension.

Table 6.

Rasch Model Item Parameters for Verbal Domain.

Subtest	Item	Measure	Error	Infit MNSQ	Outfit MNSQ
Digits Forward	1	−6.46	0.73	1.07	9.02
	2	−7.18	1.01	1.01	0.15
	3	−5.71	0.52	1.12	2.06
	4	−6.46	0.73	1.07	9.90
	5	−5.71	0.52	1.04	0.21
	6	−4.78	0.36	0.85	0.96
	7	−3.15	0.21	1.00	9.90
	8	−2.64	0.19	0.92	1.18
	9	−1.73	0.16	1.20	5.52
	10	−1.29	0.15	1.07	2.53
	11	0.95	0.15	1.41	2.16
	12	0.14	0.14	0.97	1.08
	13	1.89	0.18	1.19	1.30
	14	1.65	0.17	0.97	0.71
	15	3.73	0.28	1.22	0.98
	16	2.50	0.20	1.46	1.91
	17	3.90	0.29	0.89	0.60
	18	4.28	0.33	1.18	0.62
	19	4.08	0.31	0.69	0.24
Digits Backward	20	4.39	0.34	0.75	0.38
	21	−7.02	1.01	1.04	5.16
	22	−5.34	0.47	1.17	9.90
	23	−4.16	0.29	1.02	5.81
	24	−5.89	0.59	0.95	9.90
	25	−5.14	0.43	1.01	9.90
	26	−4.16	0.29	1.03	1.63
	27	−2.13	0.17	0.97	5.41
	28	0.74	0.15	0.93	0.87
	29	0.81	0.15	0.82	0.67
	30	1.07	0.16	0.84	0.68
	31	2.15	0.19	0.97	0.78
	32	2.34	0.20	0.93	0.72
	33	3.16	0.24	0.88	0.56
	34	3.29	0.25	0.89	0.36
	35	3.89	0.30	0.74	0.21
	36	3.56	0.27	0.63	0.46
	37	4.08	0.32	0.84	0.51
	38	3.98	0.31	0.64	0.16
	39	4.29	0.34	0.69	0.16
Digits Forward–Interference	40	4.66	0.38	0.74	0.17
	41	−4.82	0.38	0.99	3.59
	42	−5.35	0.47	0.93	6.02
	43	−4.68	0.36	1.06	4.10
	44	−4.34	0.32	0.93	1.90
	45	−3.57	0.25	0.96	2.68
	46	−2.77	0.20	1.07	1.71
	47	−1.31	0.15	1.07	1.70
	48	−0.79	0.15	1.04	1.01
	49	1.08	0.16	0.95	0.87
	50	1.32	0.16	0.89	1.09
	51	1.95	0.19	0.92	0.99
	52	2.24	0.20	0.81	0.70
	53	3.70	0.29	0.84	0.37
	54	3.54	0.28	0.82	0.32
	55	3.98	0.32	0.85	0.46
	56	4.46	0.37	0.75	0.22
	57	4.61	0.39	0.67	0.10
	58	4.21	0.34	0.83	0.20
	59	5.15	0.47	1.01	0.27
	60	4.77	0.41	0.75	0.20

Note. N = 302. Measure = item difficulty in logits; MNSQ = mean square. Items showing misfit to the Rasch model appear in bold.

Table 7.

Rasch Model Item Parameters for Static Visual-Spatial Domain.

Subtest	Item	Measure	Error	Infit MNSQ	Outfit MNSQ
Dots	61	−5.82	1.20	1.85	0.49
	62	−3.25	0.47	1.22	1.96
	63	−3.80	0.59	1.08	1.60
	64	−3.04	0.44	1.17	2.31
	65	−3.50	0.52	1.13	0.83
	66	−1.85	0.28	0.98	1.84
	67	−4.21	0.69	0.71	0.07
	68	−3.25	0.47	0.93	0.42
	69	−2.02	0.30	1.09	2.04
	70	−2.02	0.30	0.95	2.89
	71	1.33	0.14	1.14	1.45
	72	−0.55	0.19	1.10	2.15
	73	2.25	0.14	1.01	0.99
	74	−0.92	0.21	1.01	0.95
	75	1.95	0.14	0.93	0.87
	76	3.40	0.15	0.94	0.90
	77	4.91	0.22	0.94	1.19
	78	5.11	0.23	0.88	0.53
	79	4.28	0.18	1.07	1.10
	80	4.35	0.19	0.72	0.41
Dots Up	81	−2.51	0.38	0.96	1.33
	82	−2.66	0.40	0.84	1.26
	83	−3.28	0.52	1.13	1.90
	84	−0.74	0.20	0.98	1.00
	85	−1.61	0.27	0.94	1.73
	86	−2.25	0.34	0.87	0.32
	87	−1.47	0.26	0.92	1.14
	88	−1.23	0.24	0.90	0.51
	89	0.21	0.16	0.92	0.85
	90	1.27	0.14	0.90	0.96
	91	1.80	0.14	0.96	1.04
	92	1.73	0.14	1.12	1.26
	93	1.73	0.14	0.96	0.92
	94	2.59	0.14	1.07	1.05
	95	3.38	0.16	0.99	0.86
	96	4.25	0.18	1.01	1.04
	97	3.79	0.17	0.87	0.79
	98	3.26	0.15	0.89	0.75
	99	5.38	0.26	0.89	0.61
	100	9.91	1.84	1.00	1.00
Dots–Interference	101	−3.44	0.59	0.92	4.83
	102	−3.86	0.72	0.94	0.16
	103	−3.14	0.51	0.98	0.95
	104	−2.90	0.46	1.01	1.62
	105	−2.90	0.46	1.02	0.92
	106	−2.70	0.43	1.02	1.89
	107	−1.51	0.27	0.97	0.67
	108	−2.14	0.34	1.00	2.38
	109	−2.26	0.35	0.99	0.69
	110	−1.66	0.28	0.92	0.62
	111	−0.70	0.21	1.22	3.33
	112	0.27	0.16	0.99	0.89
	113	0.10	0.17	1.14	1.20
	114	−0.18	0.18	1.00	0.90
	115	1.13	0.15	0.96	1.01
	116	4.39	0.19	1.00	0.83
	117	1.57	0.14	0.97	0.96
	118	3.25	0.16	1.10	1.03
	119	4.21	0.19	0.91	0.69
	120	5.49	0.27	1.01	0.77

Note. N = 297. Measure = item difficulty in logits; MNSQ = mean square. Items showing misfit to the Rasch model appear in bold.

Table 8.

Rasch Model Item Parameters for Dynamic Visual-Spatial Domain.

Subtest	Item	Measure	Error	Infit MNSQ	Outfit MNSQ
Dots Sequence	121	−4.08	0.28	0.97	0.99
	122	−3.36	0.22	1.08	1.22
	123	−5.10	0.42	0.85	0.46
	124	−3.56	0.23	1.10	1.28
	125	−2.64	0.18	1.19	1.77
	126	−4.00	0.27	1.01	1.22
	127	−3.06	0.20	0.87	1.04
	128	−3.51	0.23	0.91	0.76
	129	−1.62	0.15	0.92	1.04
	130	−1.27	0.14	0.96	0.92
	131	1.24	0.16	1.04	1.08
	132	1.53	0.17	1.02	0.98
	133	3.49	0.31	0.95	0.45
	134	4.48	0.46	1.08	0.50
	135	7.38	1.82	1.00	1.00
	136	7.38	1.82	1.00	1.00
	137	6.16	1.01	0.99	0.19
	138	7.38	1.82	1.00	1.00
	139	7.38	1.82	1.00	1.00
	140	7.38	1.82	1.00	1.00
Dots Sequence Backward	141	−2.71	0.18	1.01	0.81
	142	−2.52	0.17	0.98	0.78
	143	−4.58	0.35	1.02	0.79
	144	−3.66	0.25	1.03	1.43
	145	−3.93	0.27	0.93	0.98
	146	−3.03	0.20	1.02	0.99
	147	−1.59	0.15	1.08	1.70
	148	−2.61	0.18	0.96	1.18
	149	−0.82	0.14	0.97	1.27
	150	−0.06	0.14	1.02	0.99
	151	2.47	0.23	1.02	0.68
	152	1.80	0.19	0.97	0.91
	153	4.43	0.47	1.00	0.39
	154	3.26	0.29	0.91	0.40
	155	7.33	1.82	1.00	1.00
	156	6.11	1.01	1.00	0.26
	157	7.33	1.82	1.00	1.00
	158	7.33	1.82	1.00	1.00
	159	7.33	1.82	1.00	1.00
	160	7.33	1.82	1.00	1.00
Dots Sequence–Interference	161	−3.43	0.23	1.10	1.07
	162	−3.22	0.22	1.02	1.05
	163	−4.11	0.30	1.06	1.64
	164	−2.88	0.20	1.05	1.82
	165	−2.77	0.19	1.07	1.14
	166	−2.96	0.20	1.03	0.80
	167	−2.09	0.16	0.94	0.89
	168	−0.68	0.14	1.02	1.02
	169	−0.90	0.14	0.99	0.97
	170	−1.33	0.15	0.91	0.78
	171	2.36	0.22	1.08	1.79
	172	2.96	0.27	0.87	0.51
	173	3.03	0.28	0.91	0.58
	174	5.32	0.72	1.04	1.24
	175	5.32	0.72	1.03	0.85
	176	3.99	0.40	0.79	0.16
	177	6.03	1.01	0.88	0.05
	178	6.03	1.01	0.88	0.05
	179	6.03	1.01	0.88	0.05
	180	6.03	1.01	0.88	0.05

Note. N = 292. Measure = item difficulty in logits; MNSQ = mean square. Items showing misfit to the Rasch model appear in bold.

According to the outfit index, which is influenced by extreme or outlying scores, the following misfitting items were identified: 22 in the Verbal domain, 15 in the SVS domain, and 6 in the DVS domain. All of these items showed model underfit, or unpredictability (values > 1.4). Because two misfitting items from the Verbal domain were problematic in terms of both infit and outfit, this resulted in a total of 44 misfitting items identified, according to either infit or outfit, out of 180 total (24%). For the Verbal domain, infit mean squares ranged from 0.63 to 1.46 (M = 0.95, SD = 0.17) and outfit mean squares ranged from 0.10 to 9.9 (M = 2.23, SD = 2.95; see Table 6). For the SVS domain, infit mean squares ranged from 0.71 to 1.85 (M = 1.00, SD = 0.15) and outfit mean squares ranged from 0.07 to 4.83 (M = 1.19, SD = 0.79; see Table 7). For the DVS domain, infit mean squares ranged from 0.79 to 1.19 (M = 0.99, SD = 0.07) and outfit mean squares ranged from 0.05 to 1.82 (M = 0.90, SD = 0.43; see Table 8). Despite a high number of misfitting items according to outfit, when only items that misfit according to the infit index are considered, only 1.67% of all items showed misfit to the Rasch model (3 out of 180). Therefore, Items 11, 16, and 61 have been flagged for revision. The additional items showing misfit according to the outfit index will also need to be reviewed during further refinement of the WOMBAT.

Person Separation and Item Reliability

Results of Rasch analysis showed PSI values for all three domains exceeded 2.0: Verbal PSI = 2.85, SVS PSI = 2.83, and DVS PSI = 2.08. This means scores can be used to distinguish among three levels of person ability (low, average, and high). In addition, both person and item reliability index values for all three domains were within the acceptable range (>.80; Fox & Jones, 1998): .89 person/.99 item reliability for the Verbal domain, .89/.98 for the SVS domain, and .81/.96 for the DVS domain (see Table 5). This suggests measurement properties and item parameters (item difficulty, person ability, etc.) derived from Rasch analysis of the WOMBAT using the current sample and pool of items can be reasonably generalized to other samples and similar item pools.

Discussion

Because of the importance of different components of WM for learning and achievement, there is a growing need for theoretically defensible measures of WM. The current study used CFA and the Rasch model to evaluate the WOMBAT, a multicomponent, online WM measure. The current study addressed two research questions:

Does the factor structure of the WOMBAT reflect the theoretical organization of WM into three separate domains?

Which items contribute to measurement of the identified WM dimensions, and which items should be flagged for revision due to misfit?

CFA results indicated the best-fitting model to the WOMBAT data is an MTMM model with three domain factors: Verbal, SVS, and DVS . This supported our hypothesis, as well as the organizational structure of WOMBAT tasks, that the WOMBAT measures three distinct WM dimensions corresponding to three content domains. The division of visual-spatial WM into a separate static and dynamic factor is consistent with previous research and theoretical models that suggest visual-spatial WM includes both static and dynamic components (Cornoldi & Vecchi, 2000, 2003; Kesner et al., 1994; Lecerf & de Ribaupierre, 2005; Logie, 1995; Mammarella et al., 2006; Mammarella et al., 2013; Pazzaglia & Cornoldi, 1999; Pickering, 2001; van der Sluis et al., 2005). The broader conclusion that WM can be divided into distinct content domain dimensions is also consistent with previous research demonstrating separate verbal and visual-spatial WM factors (Buehner et al., 2005; Cornoldi & Vecchi, 2000, 2003; Ferreira et al., 2011; Oberauer et al., 2000; Oberauer et al., 2003; Oberauer et al., 2005). Previous studies have approached the task of dividing WM into distinct components differently, using both content domains and processing demands (e.g., STM, WM, and executive processing demands). It is possible that we identified a third domain factor (i.e., two separate factors within visual-spatial WM) because other studies did not include tasks targeted at these separate visual-spatial content domains. Our results suggest future research should include tasks that specifically target static and dynamic visual-spatial WM to extend the psychometric and neuroimaging evidence for multiple visual-spatial WM components.

CFA results suggested that the WOMBAT measures three underlying dimensions. Thus, for the purposes of Rasch modeling, the three Verbal subtests, three SVS subtest, and three DVS subtests should be treated as separate measures. This informs test development in supporting our conceptualization of three distinct domains of WM subtests that each measure different types of information to be remembered. In addition, since a one-factor correlated uniqueness model was not supported, the implication is that interpretation of WOMBAT scores should focus at the factor or domain level (i.e., on separate Verbal, SVS, and DVS scores) rather than at the overall level (i.e., one overall WOMBAT score). These results also contribute to research on the structure of the WM construct by providing additional evidence for multiple WM components.

The CFA results also supported the Rasch assumption of unidimensionality for the Verbal, SVS, and DVS domains. Thus, calibration of item difficulty and person ability values in the Rasch portion of the study within these three domains—as well as interindividual, interitem, and intraindividual comparisons—can be said to be meaningful. The unidimensionality of the three separate WOMBAT domains is further supported by PSI values falling in the acceptable range for Verbal, SVS, and DVS item sets.

Despite overall unidimensionality for the three domains, item fit indices revealed several individual items that fell outside Rasch model expectations and may not contribute to measurement of the Verbal, SVS, or DVS underlying dimensions. These results address the second research question regarding which items do and do not contribute to measurement of the dimensions identified through CFA. According to outfit, about 24% of WOMBAT items (43 out of 180 total) showed misfit. The direction of these statistics indicates considerable model underfit (unpredictability in the model) for these items, suggesting further test development should include careful examination and revision of these items. Three items were also identified as underfitting according to infit indices, and these items will need to be revised or replaced. Replacing misfitting items may improve reliability of the WOMBAT. While the current results demonstrated acceptable levels of test–retest reliability and internal consistency for early-stage instrument research at the level of the whole test, more refinement is needed to improve subtest and overall reliability to the level needed for individual assessment and decision making (Nunnally & Bernstein, 1994).

Although further refinement is needed, when only items that showed misfit according to the infit index are considered, 1.67% of the total 180 WOMBAT items showed misfit to Rasch expectations (3 out of 180 total). This is likely because outfit is an unweighted indicator and was influenced by outlying scores in each subtest, while infit is weighted more heavily for persons whose scores that fall near the item difficulty value for a particular item. As noted by Bond and Fox (2001), because infit is a weighted index, it may provide more insight into the performance of an item than outfit. Thus, when focusing on infit indices, the majority (more than 98%) of WOMBAT items appear to meet Rasch expectations and contribute to the measurement of the respective domain’s single underlying dimension.

High person and item reliability indices for each subscale offer support for the consistency of WOMBAT domains, and suggest these results and measurement properties may be generalized beyond the current samples of persons and items. All person reliability index values exceeded .80 and all item reliability index values exceeded .90, showing the WOMBAT domain measurement properties demonstrated in the current study would be similar with those for another sample of persons with similar abilities or of similar items. This can be tested in future research with different samples of both persons and of new items, for example, those generated to replace problem items identified in the current study. Future studies should also evaluate traditional reliability indices with larger samples. In the current study, only 21 individuals participated in the test–retest reliability study, which may have attenuated test–retest correlations.

Moreover, similar analyses to those conducted in the current study can be conducted with younger school-aged children to improve Rasch estimation stability. Differential item functioning (DIF) analyses should also be conducted to determine whether WOMBAT items function differently (i.e., are more or less difficult for certain age groups, taking person ability into account) for school-aged children, adolescents, and adults. The current sample size limited our ability to conduct such analyses, but they are necessary for further validating the WOMBAT for use with different age groups.

Collecting data from school-aged children will also aid in the generation of norms during future standardization of the test. For increased generalizability, future studies and norming efforts should include participant samples that more broadly represent the demographic composition of the United States. The PSI values for the Verbal, SVS, and DVS domains exceeded 2.0, suggesting these WOMBAT domains are capable of distinguishing among at least three groups of persons—low, average, and high ability. This will aid in determining descriptive ranges for WOMBAT subscale scores once the test has been standardized. Unidimensionality and overall fit to the Rasch model for the three domains also means that scores from the WOMBAT Verbal, SVS, and DVS domains can be calibrated to create meaningful ability and difficulty scores, which will also aid in standardization by providing interval-level scores for comparisons between students, between items, or from pre- to postintervention (like W scores on the Woodcock–Johnson tests; Woodcock et al., 2001).

Overall, the results indicate the WOMBAT is a three-factor measure, with Verbal, SVS, and DVS factors each measuring a single dimension, and that most items contribute to measurement of these respective dimensions. The results of the current study provide preliminary psychometric support for the WOMBAT, as well as empirical support for the theoretical organization of WM into three distinct dimensions. Further research and test refinement are needed. However, indices such as test–retest reliability, subtest internal consistency, and item outfit need improvement before the instrument can be used for individual assessment. The current analysis identified misfitting items that can be targeted for revision to increase reliability. With further refinement, the WOMBAT may help schools and clinicians identify and intervene with children with specific WM deficits before their school achievement is negatively affected.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported in part by the American Psychological Foundation’s 2013 Elizabeth Munsterberg Koppitz Scholarship, the South Carolina Association of School Psychologists Ralph Tindall Research Award, the American Academy of School Psychology Hyman and Lambert Memorial Scholarship, and the Society for the Study of School Psychology Dissertation Grant Award, awarded to Julia A. Englund.

References

Alloway

T. P.

(2011). A comparison of working memory profiles in children with ADHD and DCD. Child Neuropsychology, 17(5), 483-494.

Alloway

T. P.

Alloway

R. G.

(2010). Investigating the predictive roles of working memory and IQ in academic attainment. Journal of Experimental Child Psychology, 106(1), 20-29. doi:10.1016/j.jecp.2009.11.003

Andrich

(1982). An extension of the Rasch model for ratings providing both location and dispersion parameters. Psychometrika, 47(1), 105-113.

Baddeley

A. D.

(2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4, 417-423.

Baddeley

A. D.

Hitch

G. J.

(1974). Working memory. In Bower

G. A.

(Ed.), Recent advances in learning and motivation (Vol. 8, pp. 47-89). New York, NY: Academic Press.

Bond

T. G.

Fox

C. M.

(2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.

Buehner

Mangels

Krumm

Ziegler

(2005). Are working memory and attention related constructs? Journal of Individual Differences, 26(3), 121-131. doi:10.1027/1614-0001.26.3.121

Bull

Espy

Wiebe

S. A.

(2008). Short-term memory, working memory, and executive functioning in preschoolers: Longitudinal predictors of mathematical achievement at age 7 years. Developmental Neuropsychology, 33(3), 205-228. doi:10.1080/87565640801982312

Cornoldi

Vecchi

(2000). Mental Imagery in blind people: The role of passive and active visuospatial processes. In Morton

A. H.

(Ed.), Touch representation and blindness (pp. 29-58). Oxford, UK: Oxford University Press.

10.

Cornoldi

Vecchi

(2003). Visuo-spatial working memory and individual differences. Hove, UK: Psychology Press.

11.

Dahlin

K. I. E.

(2011). Effects of working memory training on reading in children with special needs. Reading and Writing, 24, 479-491. doi:10.1007/s11145-010-9238-y

12.

Daneman

Carpenter

P. A.

(1980). Individual differences in working memory and reading. Journal of Verbal Learning and Verbal Behaviour, 19, 450-466.

13.

Dehn

(2008). Working memory and academic learning: Assessment and intervention. Hoboken, NJ: John Wiley.

14.

DiStefano

Morgan

G. B.

(2010). Evaluation of the BESS TRS-CA using the Rasch rating scale model. School Psychology Quarterly, 25(4), 202-212.

15.

Duncan

P. W.

Bode

R. K.

Min Lai

Perera

(2003). Rasch analysis of a new stroke-specific outcome scale: The Stroke Impact Scale. Archives of Physical Medicine and Rehabilitation, 84(7), 950-963.

16.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.

17.

Engle

R. W.

(2002). Working memory capacity as executive attention. Current Directions in Psychological Science, 11, 19-23.

18.

Engle

R. W.

Kane

M. J.

Tuholski

S. W.

(1999). Individual differences in working memory capacity and what they tell us about controlled attention, general fluid intelligence, and functions of the prefrontal cortex. In Miyake

Shah

(Eds.), Models of working memory: Mechanisms of active maintenance and executive control (pp. 102-134). New York, NY: Cambridge University Press.

19.

Evans

J. J.

Floyd

R. G.

McGrew

K. S.

Leforgee

M. H.

(2002). The relations between measures of Cattell-Horn-Carroll (CHC) cognitive abilities and reading achievement during childhood and adolescence. School Psychology Review, 31(2), 246-262.

20.

Ferreira

Almeida

L. S.

Prieto

(2011). The role of processes and contents in human memory: An item response theory approach. Journal of Cognitive Psychology, 23(7), 873-885. doi:10.1080/20445911.2011.584692

21.

Fox

C. M.

Jones

J. A.

(1998). Uses of Rasch modeling in counseling psychology research. Journal of Counseling Psychology, 45, 30-45.

22.

Gathercole

S. E.

Alloway

T. P.

Willis

Adam

A.-M.

(2006).Working memory in children with reading disabilities. Journal of Experimental Child Psychology, 93, 265-281.

23.

Gathercole

S. E.

Pickering

S. J.

(2000). Working memory deficits in children with low achievements in the national curriculum at 7 years of age. British Journal of Educational Psychology, 70, 177-194.

24.

Gathercole

S. E.

Tiffany

Briscoe

Thorn

, & the ALSPAC Team. (2005). Developmental consequences of poor phonological short-term memory function in childhood: A longitudinal study. Journal of Child Psychology and Psychiatry, 46, 598-611.

25.

Holmes

Gathercole

S. E.

Dunning

D. L.

(2009). Adaptive training leads to sustained enhancement of poor working memory in children. Developmental Science, 12, F9-F15.

26.

Holmes

Gathercole

S. E.

Place

Dunning

D. L.

Hilton

Elliott

J. G.

(2009). Working memory deficits can be overcome: Impacts of training and medication on working memory in children with ADHD. Applied Cognitive Psychology, 12, 9-15.

27.

Hugh

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

28.

Jaeggi

S. M.

Buschkuehl

Jonides

Shah

(2011). Short- and long-term benefits of cognitive training. Proceedings of the National Academy of Sciences of the United States of America, 108(25), 10081-10086. doi:10.1073/pnas.1103228108

29.

Kenny

D. A.

Kashy

D. A.

(1992). Analysis of multitrait-multimethod matrix by confirmatory factor analysis. Psychological Bulletin, 112, 165-172.

30.

Kesner

R. P.

Hopkind

R. O.

Fineman

(1994). Item and order dissociation in humans with prefrontal cortex damage. Neuropsychologia, 32, 881-891.

31.

Klingberg

(2010). Training and plasticity of working memory. Trends in Cognitive Sciences, 14, 317-324.

32.

Klingberg

Fernell

Olesen

P. J.

Johnson

Gustafsson

Dahlström

. . . Westerberg

(2005). Computerized training of working memory in children with ADHD—A randomized, controlled trial. Journal of the American Academy of Child Psychiatry, 44, 177-186.

33.

Klingberg

Forssberg

Westerberg

(2002). Training of working memory in children with ADHD. Journal of Clinical Experimental Neuropsychology, 24, 781-791.

34.

Lance

C. E.

Noble

C. L.

Scullen

S. E.

(2002). A critique of the correlated trait-correlated method and correlated uniqueness models for multitrait-multimethod data. Psychological Methods, 7(2), 228-244.

35.

Leather

C. V.

Henry

L. A.

(1994). Working memory span and phonological awareness tasks as predictors of early reading ability. Journal of Experimental Child Psychology, 58, 88-111.

36.

Lecerf

de Ribaupierre

(2005). Recognition in a visuospatial memory task: The effect of presentation. European Journal of Cognitive Psychology, 17, 47-75.

37.

Linacre

J. M.

(2005). A user’s guide to WINSTEPS. Chicago, IL: Mesa Press.

38.

Logie

R. H.

(1995). Visuo-spatial working memory. Hove, UK: Lawrence Erlbaum.

39.

Macmann

G. M.

Barnett

D. W.

(1994). Structural analysis of correlated factors: Lessons from the verbal-performance dichotomy of the Wechsler scales. School Psychology Quarterly, 9, 161-197.

40.

Mammarella

I. C.

Borella

Pastore

Pazzaglia

(2013). The structure of visuospatial memory in adulthood. Learning and Individual Differences, 25, 99-110. doi:10.1016/j.lindif.2013.01.014

41.

Mammarella

I. C.

Cornoldi

Pazzaglia

Toso

Grimoldi

Vio

(2006). Evidence for a double dissociation between spatial-simultaneous and spatial-sequential working memory in visuospatial (nonverbal) learning disabled children. Brain and Cognition, 62, 58-67.

42.

Mastropieri

M. A.

Sweda

Scruggs

T. E.

(2000). Putting mnemonic strategies to work in an inclusive classroom. Learning Disabilities Research & Practice, 15, 69-74.

43.

Muthén

L. K.

Muthén

B. O.

(2013). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

44.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

45.

Oberauer

Schulze

Wilhelm

Süß

H.-M.

(2005). Working memory and intelligence—Their correlation and their relation: A comment on Ackerman, Beier, and Boyle (2005). Psychological Bulletin, 131, 61-65.

46.

Oberauer

Süß

H.-M.

Schulze

Wilhelm

Wittmann

W. W.

(2000). Working memory capacity—Facets of a cognitive ability construct. Personality and Individual Differences, 29, 1017-1045.

47.

Oberauer

Süß

H.-M.

Wilhelm

Wittmann

W. W.

(2003). The multiple faces of working memory: Storage, processing, supervision, and coordination. Intelligence, 31, 167-193.

48.

Olesen

P. J.

Westerberg

Klingberg

(2004). Increased prefrontal and parietal activity after training of working memory. Nature Neuroscience, 7, 75-79.

49.

Parente

Herrman

(1996). Retraining memory strategies. Topics in Language Disorders, 17, 45-57.

50.

Pazzaglia

Cornoldi

(1999). The role of distinct components of visuo-spatial working memory in the processing of texts. Memory, 7, 19-41.

51.

Pickering

S. J.

(2001). Cognitive approaches to the fractionation of visuo-spatial working memory. Cortex, 37, 470-473.

52.

Pickering

S. J.

Gathercole

S. E.

(2001). Working Memory Test Battery for Children. London, UK: Psychological Corporation Europe.

53.

Pomplun

Custer

(2005). The construct validity of the Stanford–Binet 5 measures of working memory. Assessment, 12(3), 338-346. doi:10.1177/1073191105276796

54.

Raghubar

K. P.

Barnes

M. A.

Hecht

S. A.

(2010). Working memory and mathematics: A review of developmental, individual difference, and cognitive approaches. Learning and Individual Differences, 20, 110-122.

55.

Schafer

J. L.

Graham

J. W.

(2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147-177.

56.

Smith

E. V.

Conrad

K. M.

Chang

Piazza

(2002). An introduction to Rasch measurement for scale development and person assessment. Journal of Nursing Measurement, 10(3), 189-206.

57.

Swanson

H. L.

(1993). Working memory in learning disability subgroups. Journal of Experimental Child Psychology, 56(1), 87-114. doi:10.1006/jecp.1993.1027

58.

Swanson

H. L.

(1995). Swanson Cognitive Processing Test (S-CPT): A dynamic assessment measure. Austin, TX:PRO-ED.

59.

Swanson

H. L.

(2006). Cross-sectional and incremental changes in working memory and mathematical problem solving. Journal of Educational Psychology, 98, 265-281.

60.

Swanson

H. L.

O’Connor

(2009). The role of working memory and fluency practice on the reading comprehension of students who are dysfluent readers. Learning Disabilities, 42(6), 548-575. doi:10.1177/0022219409338742J

61.

Swanson

H. L.

Sachse-Lee

(2001). Mathematical problem solving and working memory in children with learning disabilities: Both executive and phonological processes are important. Journal of Experimental Child Psychology, 79, 294-321.

62.

Swanson

H. L.

Siegel

(2001). Learning disabilities as a working memory deficit. Issues in Education, 7, 1-48.

63.

Swanson

Zheng

Jerman

(2009). Working memory, short-term memory, and reading disabilities: A selective meta-analysis of the literature. Journal of Learning Disabilities, 42(3), 260-287. doi:10.1177/0022219409331958

64.

Thorell

L. B.

Lindqvist

Nutley

S. B.

Bohlin

Klingberg

(2009). Training and transfer effects of executive functions in preschool children. Developmental Science, 12, 106-113.

65.

Turley-Ames

K. J.

Whitfield

M. M.

(2003). Strategy training and working memory task performance. Journal of Memory and Language, 49, 446-468.

66.

U.S. Census Bureau. (2013, June 27). State & county quickfacts: USA. Retrieved from http://quickfacts.census.gov/qfd/states/00000.html

67.

van der Sluis

van der Leij

de Jong

P. F.

(2005). Working memory in Dutch children with reading- and arithmetic-related LD. Journal of Learning Disabilities, 38, 207-221.

68.

Wechsler

(2003). Wechsler Intelligence Scale for Children—Fourth Edition. San Antonio, TX: Psychological Corporation.

69.

Westerberg

Jacobaeus

Hirvikoski

Clevberger

M.-L.

Ostensson

Bartfai

Klingberg

(2007). Computerized working memory training after stroke—A pilot study. Brain Injury, 21, 21-29.

70.

Westerberg

Klingberg

(2007). Changes in cortical activity after training of working memory—A single-subject analysis. Physiology & Behavior, 92, 186-192. doi:10.1016/j.physbeh.2007.05.041

71.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock–Johnson III Tests of Cognitive Abilities. Itasca, IL: Riverside.

72.

Wuang

Y. P.

C. Y.

(2009). Rasch analysis of the Developmental Test of Visual-Motor Integration in children with intellectual disabilities. Research in Developmental Disabilities, 30(5), 1044-1053.