Reliability and Validity of the Spanish-Language Version of the NIH Toolbox

Abstract

The psychometric properties of the English-language NIH Toolbox for Assessment of Neurological and Behavioral Function (NIH Toolbox) have been examined in numerous populations. This study evaluated the reliability and validity of the Spanish-language NIH Toolbox. Participants were children aged 3 to 7 years and adults aged 18 to 85 years who took part in the NIH Toolbox norming study in Spanish. Results supported the internal consistency reliability of included measures. Test–retest reliability was strong for most tests, though it was weaker for the test of olfaction among children and the test of locomotion among adults. Spearman’s correlations and general linear models showed Spanish tests were often associated with age, sex, and education. Convergent validity for the two language measures that underwent more intensive development, evaluated via Spearman’s correlations with legacy measures, was strong. Results support using the Spanish-language NIH Toolbox to measure neurological and behavioral functioning among Spanish-speaking individuals in the United States.

Keywords

NIH Toolbox reliability validity Spanish Latinx

Over the previous half century, the Latinx population in the United States has increased nine fold, currently representing approximately 18% of the overall population (U.S. Census Bureau, 2018a). Future projections indicate that by 2060, Latinxs will represent 27.5% of the population, reflecting a near doubling of the current number of Latinxs living in the United States (U.S. Census Bureau, 2018b). While English is the dominant language of the United States, as of 2011 only approximately half of Latinxs living in the United States reported being able to speak English “very well” (Ryan, 2013). Moreover, of the more than 60 million individuals aged 5 years and older living in the United States that reported speaking a language other than English at home, nearly two thirds (62%) spoke Spanish (Ryan, 2013). Given these increases, consideration of the linguistic needs of the Latinx population in the United States is of particular importance for clinicians and researchers who wish to evaluate different components of neurological and behavioral functioning in multicultural contexts.

In recognition of this, the NIH Toolbox for Assessment of Neurological and Behavioral Function^® (NIH Toolbox^®) was developed as a collection of neurobehavioral measures designed to quickly assess sensory, motor, emotional, and cognitive functioning across the life span (aged 3 to 85 years) in both English and Spanish. It was created to be appropriate for use in longitudinal studies by measuring the same constructs at different stages of human development, and can be used to track individual performance over time. Additionally, it was designed to provide a “common currency” in epidemiologic research, enabling the pooling and sharing of large data sets across studies. The full NIH Toolbox measurement system can be administered in 2 hours or less and, although it was not originally developed for clinical use, a growing body of evidence demonstrates its applicability for evaluating functioning in various normative and clinical populations; for example, cancer (Sinha et al., 2018), neurologic conditions (Tulsky & Heinemann, 2017), and social anxiety (Troller-Renfree et al., 2015).

The psychometric properties of the English-language version of the NIH Toolbox have been examined in the general U.S. population and numerous clinical populations. For example, prior studies have demonstrated support for the reliability and validity of the majority of the NIH Toolbox measures in healthy samples in English (Coldwell et al., 2013; Dalton et al., 2013; Reuben et al., 2013; Rine et al., 2013; Salsman et al., 2013; Varma et al., 2013; Weintraub et al., 2013). Additionally, the clinical utility and psychometric strength of the tests of cognitive functioning have been demonstrated among individuals with disabilities (Tulsky & Heinemann, 2017), and among children aged 3 to 15 years (Bauer & Zelazo, 2013). Although normative standards for the Spanish-language tests of cognitive (Casaletto et al., 2016) and emotional (Babakhanyan et al., 2018) functioning have been published, a comprehensive psychometric evaluation of the Spanish NIH Toolbox measures has not. In the present study, we explore the reliability and validity of the Spanish-language version of the NIH Toolbox for children and adults across four domains of functioning: sensory, motor, emotional, and cognitive.

Method

Participants and Procedures

Data from the NIH Toolbox norming study were used. These data are publicly available and can be accessed via the HealthMeasures Dataverse data repository (https://dataverse.harvard.edu/dataverse/HealthMeasures). Participants in the NIH Toolbox norming study were community-dwelling children and adults who were neurologically healthy and capable of following instructions. A sampling strategy stratified by age, sex, and primary language was followed in accordance with the published NIH Toolbox norming plans (Beaumont et al., 2013). Participants were recruited by the market research company Delve, Inc. (now known as Focus Pointe Global) from 10 locations throughout the United States (Atlanta, Chicago—Oak Brook, Cincinnati, Columbus, Dallas, Los Angeles, Minneapolis, Philadelphia, Phoenix, and St. Louis) that were selected, among other reasons, to maximize access to subjects from varied Spanish-speaking communities. To be eligible for inclusion, potential participants had to have adequate visual, auditory, vestibular, and motor functioning, either independently or with support from assistive devices, to enable completion of all items included in the NIH Toolbox testing battery. The present analysis included those participants aged 3 to 7 years (n = 496) and aged 18 to 85 years (n = 408) who elected to participate in the NIH Toolbox norming study in Spanish. Spanish-speaking children between age 8 and 17 years were not included in the NIH Toolbox norming study, as census data indicated that less than 2% of children in this age range living in the United States at the time of study enrollment used Spanish as their primary language (Beaumont et al., 2013).

Potential participants met with trained research personnel who administered structured interviews and standardized questionnaires to ensure eligibility prior to enrollment. Informed consent was obtained from all adult participants. Parental informed consent was obtained from children aged 3 to 7 years, and assent was obtained from children aged 7 years. A subset of the individuals who participated in the NIH Toolbox norming study repeated the NIH Toolbox measures 5 to 14 days following initial test administration to enable evaluation of test–retest reliability. The NIH Toolbox norming study was approved by the institutional review board at Northwestern University through a protocol that covered all testing sites and was completed in accordance with the Helsinki Declaration.

Measures

The NIH Toolbox for Assessment of Neurological and Behavioral Functioning (Gershon et al., 2013)

As outlined above, the NIH Toolbox is composed of assessments targeting four primary domains and contributing to four batteries: sensation, motor, emotion, and cognition. The full battery includes all assessments, while an early childhood battery includes a subset thereof, as described next. Most measures were originally developed in English and then translated into Spanish using the Functional Assessment of Chronic Illness Therapy translation methodology (Bonomi et al., 1996; Cella et al., 1998; Eremenco et al., 2005; Lent et al., 1999), although the Cognition Battery language measures were effectively developed from scratch in Spanish. Detailed information regarding the cultural adaptation, linguistic translation, and overall development of the Spanish-language version of the NIH Toolbox is available elsewhere (Gershon et al., 2020).

NIH Toolbox Sensation Battery

The Sensation Battery (Coldwell et al., 2013; Cook et al., 2013; Dalton et al., 2013; Varma et al., 2013; Zecker et al., 2013) is composed of six measures assessing five domains: (1) audition, (2) gustation, (3) vision, (4) olfaction, and (5) pain. Audition is assessed using the NIH Toolbox Words-in-Noise Test (Zecker et al., 2013), which tests hearing in a noisy environment and is appropriate for individuals aged 6 to 85 years. Assessment of speech perception in noise is considered an ecologically valid measure of auditory functioning, as real-world communication often occurs in noisy environments. A Spanish-language version of the Words-in-Noise test was developed separately and was adapted for use in the NIH Toolbox (McArdle et al., 2009). Scores on this measure are presented as decibels of signal-to-noise ratio (dB S/N), reflecting the quietest signal correctly perceived amid noise. Lower scores reflect better hearing. Gustation is assessed with the NIH Toolbox Regional Taste Intensity Test (Coldwell et al., 2013), which measures the perceived intensity of quinine (i.e., bitter taste) and salt as administered in liquid. This test is appropriate for individuals aged 12 to 85 years, and scores are presented on a generalized labeled magnitude scale. Higher scores reflect greater perceived taste intensity. Vision is assessed with the NIH Toolbox Visual Acuity Test (Varma et al., 2013), which measures distance vision at three meters and is appropriate for individuals aged 3 to 85 years. Scores are presented as LogMAR units and in Snellen format; higher scores reflect worse vision. Olfaction is assessed with the NIH Toolbox Odor Identification Test (Dalton et al., 2013), which assesses a person’s ability to identify different odors using scratch-and-sniff cards, and is appropriate for individuals aged 3 to 85 years. Nine odorants are presented to individuals aged 10+ years, and five are presented to individuals aged 3 to 9 years. Scores reflect the number of odorants correctly identified, with higher scores reflecting better smell. Pain is assessed with the Pain Intensity Survey and the Pain Interference Survey (Cook et al., 2013), which are appropriate for individuals aged 18 to 85 years. The Pain Intensity Survey is a single-item numerical rating scale assessing pain severity over the prior week; higher scores indicate more severe pain. The Pain Interference Survey is administered as a computer adaptive test and assesses the degree to which pain interferes with engagement in normal activities. Scores are presented as T-scores, with higher scores indicating more pain interference.

NIH Toolbox Motor Battery (Reuben et al., 2013)

The Motor Battery is composed of five measures assessing five domains: (1) dexterity, (2) strength, (3) balance, (4) locomotion, and (5) endurance. The NIH Toolbox Motor Battery for individuals aged 7 to 85 years includes all five of these measures. The NIH Toolbox Early Childhood Motor Battery for individuals aged 3 to 6 years includes all of these measures except the test of locomotion. Dexterity is assessed using the 9-hole Pegboard Test. Scores reflect the number of seconds needed to accurately place and remove nine plastic pegs into a plastic pegboard using the dominant hand. Higher scores reflect worse manual dexterity. Strength is assessed using the Grip Strength Test, which evaluates the number of pounds of force generated using the dominant hand on a hand dynamometer. Higher scores indicate greater strength. Balance is assessed using the Standing Balance Test, which evaluates anterior–posterior postural sway. Raw normalized path length scores are calculated based on time and acceleration, and are then converted to theta values using an item response theory (IRT) model. Higher scores reflect better balance. Locomotion is assessed using the 4-Meter Walk Gait Speed Test. Scores are based on the number of seconds it takes to walk 4 meters at one’s usual pace, using the better of two trials, which are then converted to meters/second. Higher scores are indicative of better locomotion. Endurance is assessed using the 2-Minute Walk Test. Scores reflect the number of feet walked in 2 minutes, with higher scores reflecting better endurance.

NIH Toolbox Emotion Battery (Salsman et al., 2013)

The Emotion Battery broadly assesses four domains: (1) negative affect, (2) social relationships, (3) psychological well-being, and (4) stress and self-efficacy. The Spanish-language battery includes 17 self-report scales for adults aged 18 to 85 years, and 10 parent-report scales for children aged 3 to 7 years. In addition to these 27 measures, the English-language battery also includes 15 self-report scales for children aged 8 to 17 years and 11 parent-report scales for children aged 8 to 12 years. Spanish translations of these additional measures are available; however, no norms exist given that children aged 8 to 17 years were not included in the Spanish-language norming sample. Higher scores indicate more of the construct being assessed across all measures included in the Emotion Battery. The negative affect subdomain includes six assessments for adults (Anger-Affect, Anger-Hostility, Anger-Physical Aggression, Sadness, Fear-Affect, and Fear-Somatic Arousal) and four assessments for children (parent-reported assessments of Anger, Fear-Over Anxious, Fear-Separation Anxiety, and Sadness). The psychological well-being subdomain includes three assessments for adults (Positive Affect, Life Satisfaction, and Meaning) and two assessments for children (parent-reported assessments of Positive Affect and Life Satisfaction). The parent-reported assessment of Positive Affect was not administered in Spanish during the norming study and was therefore not included in the present analysis; however, a Spanish-language version of this measure is available without norms. The social relationships subdomain includes six assessments for adults (Friendship, Loneliness, Emotional Support, Perceived Hostility, Instrumental Support, and Perceived Rejection) and four assessments for children (parent-reported assessments of Social Withdrawal, Positive Peer Interactions, Peer Rejection, and Empathic Behaviors). The stress and self-efficacy subdomain includes two assessments for adults (Perceived Stress and Self-Efficacy). There are no assessments for children included in the stress and self-efficacy subdomain. All measures excepting the parent-reported assessments of Life Satisfaction (five items), Social Withdrawal (four items), Positive Peer Interactions (four items), and Peer Rejection (nine items) are scored using IRT methods to yield a theta value. For these four parent-reported measures, scores reflect a raw sum of the included items.

NIH Toolbox Cognition Battery (Weintraub et al., 2013)

The Cognition Battery is composed of seven measures assessing five domains: (1) executive function and attention, (2) episodic memory, (3) working memory, (4) processing speed, and 5) language. Executive function and attention are assessed using the Dimensional Change Card Sort Test and the Flanker Inhibitory Control and Attention Test, which are appropriate for individuals aged 3 to 85 years. Scores on these measures reflect a combination of accuracy and reaction time, where each of these components receives a score between 0 and 5. These scores are then summed to yield a computed score ranging from 0 to 10, with higher scores reflecting better performance. Episodic memory is assessed using the Picture Sequence Memory Test, which is appropriate for individuals aged 3 to 85 years. Scores are computed as IRT-estimated thetas, and reflect the ability to accurately recall an increasingly lengthy series of illustrations. Higher scores reflect better episodic memory. Working memory is assessed using the List Sort Memory Test, which is appropriate for individuals aged 7 to 85 years, with a parallel supplemental test available for children aged 3 to 6 years. Scores are computed as the total number of foods and animals correctly recalled and reordered from smallest to largest, by category. Higher scores reflect better working memory. Processing speed is assessed using the Pattern Comparison Processing Speed Test, which is appropriate for individuals aged 7 to 85 years. Scores reflect the number of times in an 85-second window that an individual can accurately determine if two side-by-side images are identical, with higher scores indicating better processing speed. Language ability is assessed using the Picture Vocabulary Test, which is appropriate for individuals aged 3 to 85 years, and the Oral Reading Recognition Test, which is appropriate for individuals aged 7 to 85 years. Both language measures are administered as computer adaptive tests and scored using IRT methodology to yield a theta score. The score for the Picture Vocabulary Test reflects an individual’s ability to correctly select one of four images to match the meaning of an audio recorded word, and the score for the Oral Reading Recognition Test reflects the ability to read and correctly pronounce letters and words, shown one at a time on a screen. Higher scores reflect better language abilities. The NIH Toolbox Cognition Battery for individuals aged 7 to 85 years includes all seven core measures. The NIH Toolbox Early Childhood Cognition Battery for individuals aged 3 to 6 years includes only the Dimensional Change Card Sort Test, Flanker Inhibitory Control and Attention Test, Picture Sequence Memory Test, and Picture Vocabulary Test.

Sociodemographic Variables

In addition to completing the NIH Toolbox measures, participants self-reported sociodemographic information including age, sex, and years of education, as well as race/ethnicity and Latinx status as measured by the 2010 Census (Humes et al., 2011).

Validity Measures

Convergent validity was not evaluated for most of the Spanish-language versions of the NIH Toolbox measures. However, the two language measures included in the Cognition Battery involved significantly more development than the other measures. Therefore, convergent validity for these two measures was evaluated by correlating scores on these measures with scores on two well-established legacy Spanish-language instruments, as described below.

The Batería-III Woodcock-Muñoz Vocabulario Sobre Dibujos Test (Muñoz-Sandoval et al., 2005)

To enable assessment of the convergent validity of the Picture Vocabulary Test, the Batería-III Woodcock-Muñoz Vocabulario Sobre Dibujos Test, the parallel Spanish-language version of the Woodcock-Johnson^® III Picture Vocabulary test (Woodcock et al., 2001), was administered to a subset of participants. The Vocabulario Sobre Dibujos test measures oral language development and word knowledge.

The Word Accentuation Test

To enable assessment of the convergent validity of the Oral Reading Recognition Test, a 48-item version of the Word Accentuation Test, a Spanish-language word recognition task, was administered to a subset of participants. The Word Accentuation Test requires respondents to read words aloud, which are then scored as correct or incorrect based on the accentuation of the word. The 48-item version used in this analysis was created by supplementing the 40 items included in the Word Accentuation Test—Chicago (Krueger et al., 2006), which was developed for use with Spanish speakers in the United States, with eight additional items included in the original Word Accentuation Test (Del Ser et al., 1997), which was developed in Madrid, that are not in the Chicago version.

Analytic Plan

For the present analysis, z scores were first computed for all measures based on the mean and standard deviation for the full sample (e.g., children and adults) combined. This placed scores on a common metric, thus facilitating analysis and interpretation of results. Normality of the data was evaluated via a combination of visual inspection and statistical evaluation (i.e., Kolmogorov–Smirnov and Shapiro–Wilk tests). Nonparametric analytic approaches were implemented as numerous variables were nonnormally distributed.

Reliability

Different types of reliability were calculated for different measures within the NIH Toolbox, as appropriate. Cronbach’s coefficient alphas were calculated to assess internal consistency reliability for those self-report and proxy-report measures within the Emotion Battery that were administered as fixed forms. Marginal reliability coefficients were estimated with graded response models (IRT) for measures within the Emotion Battery that were administered as computer adaptive tests. Spearman’s correlations with associated p values were calculated to evaluate test–retest reliability for measures within the Sensation, Motor, and Cognition Batteries. intraclass correlations (ICCs) were initially calculated as well; however, results were unreliable due to low power and nonnormality of the data. The strength of Spearman’s correlations was evaluated with cutoffs of .10, .30, and .50 indicating small, medium, and large effects, respectively (Cohen, 1992). Because most of the NIH Toolbox Emotion Battery measures are structured to assess a 7-day recall period, the average time span between administrations for the Emotion Battery was greater than 8 days, and emotions would be expected to fluctuate during that interval, test–retest reliability was not evaluated for this battery. Additionally, to reduce overall burden, measures from the Motor Battery assessing dexterity, strength, and balance were not readministered to the Spanish-speaking participants. Accordingly, the test–retest reliability of these measures was not evaluated. Within the Sensation Battery, the gustation and pain assessments were only administered to Spanish-speaking participants aged 18 years and older, per NIH Toolbox administration guidelines, and only two participants aged 3 to 7 years repeated the Words-in-Noise audition measure. Therefore, test–retest reliability was not evaluated for these tests in children.

Validity

Convergent validity: Language measures

Scores on the gold-standard language measures were converted to z scores using the same approach outlined above, so scores on the validity measures and the NIH Toolbox language measures would be on the same metric. Spearman’s correlations between these gold-standard measures and the NIH Toolbox language measures were then calculated to evaluate convergent validity. These comparisons were cross-sectional.

Comparison of scores with demographic characteristics

In addition to the convergent validity analyses conducted for the two NIH Toolbox language measures, the validity of scores on the Spanish-language versions of all measures was evaluated with Spearman’s correlations and general linear models relating scores to demographic subgroup membership, controlling for age and sex as appropriate. These comparisons were cross-sectional. For Cognition Battery measures, these analyses also controlled for education level for adults and mother’s education level for children. For general linear models, effect sizes were reported as Cohen’s ds, and cutoffs of .20, .50, and .80 indicated small, medium, and large effects, respectively (Cohen, 1992). To account for multiple testing, a Bonferroni correction (α = .001) was used in this analysis.

Results

Participant Characteristics

The NIH Toolbox Spanish-language norming sample was composed of 496 children aged 3 to 7 years and 408 adults aged 18 to 85 years. Details regarding participant characteristics are presented in Table 1. Slightly more than 20% of the adult sample was born in the United States, including 2.9% reporting that they were born in Puerto Rico. The majority of adult respondents (77.5%) were born outside of the United States, similar to the sample included in the Pew Research Center’s 2013 National Survey of Latinos and Religion, in which 88.5% of Spanish-dominant respondents reported having been born outside of the United States (Pew Research Center, 2013). Foreign-born participants in the present sample reported having been born in Mexico (43.4%), Central America (i.e., Costa Rica, El Salvador, Guatemala, Honduras, Nicaragua, Panama; 16.4%), the Caribbean (i.e., Cuba, Dominican Republic; 4.2%), South America (i.e., Brazil, Chile, Colombia, Ecuador, Peru, Uruguay, Venezuela; 12.7%), and Europe (i.e., Spain; 0.2%). Information regarding country of origin was not available for the 3.0% of child respondents who were not born in the United States.

Table 1.

Sample Characteristics.

	Ages 3 to 7 years (n = 496)	Ages 18 to 85 years (n = 408)
Age^a	4.92 (1.42)	44.10 (16.72)
Education/mother’s education^a	9.73 (3.91)	10.71 (4.33)
Sex: Female^b	52.4% (260)	65.0% (265)
Latinx^b	99.8% (495)	100% (408)
Background group^b,c
Mexican	76.2% (378)	55.1% (225)
Puerto Rican	1.8% (9)	4.7% (19)
Cuban	—	2.0% (8)
Other	20.6% (102)	37.3% (152)
Race^b
American Indian/Alaskan Native	13.1% (65)	9.6% (39)
Asian	—	0.5% (2)
Black	1.0% (5)	5.4% (22)
Native Hawaiian/Pacific Islander	—	0.5% (2)
White	78.6% (390)	77.0% (314)
Multiracial	1.6% (8)	1.7% (7)
Other/not specified	5.6% (28)	5.4% (22)
Country of birth^b
The United States	95.2% (472)	21.3% (87)
Foreign-born	3.0% (15)	77.5% (316)

Mean (Standard Deviation). ^bPercentage (n). ^cResponse options reflective of question regarding Latinx origin as asked in the 2010 Census.

Reliability

Internal Consistency Reliability

As shown in Table 2, Cronbach’s alphas demonstrated acceptable to strong internal consistency reliability for all measures within the Emotion Battery that were administered as fixed forms among both adults (αs ranged from .77 to .95) and children (αs ranged from .66 to .82).

Table 2.

Internal Consistency and IRT Reliability for NIH Toolbox Emotion Battery Scores.

	Ages 3 to 7 years (n = 496)			Ages 18 to 85 years (n = 408)
	No. of items	α	IRT reliability	No. of items	α	IRT reliability
Negative Affect
Anger-Affect	—	—	—	CAT	—	.95
Anger-Hostility	—	—	—	5	.84	—
Anger-Physical Aggression	—	—	—	5	.79	—
Sadness	—	—	—	CAT	—	.94
Fear-Affect	—	—	—	CAT	—	.95
Fear-Somatic Arousal	—	—	—	6	.77	—
Anger, PR	9	.82	—	—	—	—
Fear-Over Anxious, PR	6	.66	—	—	—	—
Fear-Separation Anxiety, PR	7	.71	—	—	—	—
Sadness, PR	7	.70	—	—	—	—
Psychological Well-Being
Positive Affect	—	—	—	CAT	—	.97
Life Satisfaction	—	—	—	CAT	—	.95
Meaning	—	—	—	CAT	—	.87
Life Satisfaction, PR	5	.69	—	—	—	—
Social Relationships
Friendship	—	—	—	8	.92	—
Loneliness	—	—	—	5	.88	—
Emotional Support	—	—	—	8	.95	—
Perceived Hostility	—	—	—	8	.90	—
Instrumental Support	—	—	—	8	.94	—
Perceived Rejection	—	—	—	8	.92	—
Social Withdrawal, PR	4	.76	—	—	—	—
Positive Peer Interactions, PR	4	.79	—	—	—	—
Peer Rejection, PR	9	.81	—	—	—	—
Empathic Behaviors, PR	CAT	—	.90	—	—	—
Stress and Self-Efficacy
Perceived Stress	—	—	—	10	.78	—
Self-Efficacy	—	—	—	CAT	—	.91

Note. IRT = item response theory; CAT = computer adaptive test; PR = parent-report.

IRT Reliability

All Emotion Battery measures administered as computer adaptive tests had excellent reliability exceeding .90, with the exception of the Psychological Well-being—Meaning measure, which had an estimated reliability of .87.

Test–Retest Reliability

Of the 904 individuals who participated in the NIH Toolbox norming study and completed questionnaires in Spanish, 73 to 86 participants—depending on the domain—repeated these measures 5 to 14 days following initial test administration (sensation domain: n = 85, M = 8.12 days, SD = 2.63; motor domain: n = 85, M = 8.12 days, SD = 2.63; emotion domain: n = 73, M = 8.45 days, SD = 2.37; cognition domain: n = 86, M = 8.08 days, SD = 2.64. Of note, not all participants completed all measures included in each NIH Toolbox domain, as indicated in Table 3, Table 4, and Table 5.

Table 3.

Spearman’s Test–Retest Correlations for NIH Toolbox Sensation Battery Measures.

	Ages 3 to 7 years (n = 38)		Ages 18 to 85 years (n = 47)
	n	ρ (p)	n	ρ (p)
Audition
WIN Left Ear	—	—	9	.49 (.181)
WIN Right Ear	—	—	10	.55 (.100)
Gustation
Quinine, tongue	—	—	38	.49 (.002)
Quinine, mouth	—	—	38	.48 (.003)
Salt, tongue	—	—	38	.53 (.001)
Salt, mouth	—	—	38	.53 (.001)
Vision
Visual Acuity	31	.69 (<.001)	37	.87 (<.001)
Olfaction
Odor ID	38	.20 (.223)	41	.52 (.001)
Pain
Pain Intensity	—	—	44	.78 (<.001)
Pain Interference	—	—	47	.63 (<.001)

Note. WIN = Words-In-Noise Test; Odor ID = Odor Identification; tongue = tip of tongue; mouth = whole mouth. ns reflect number of participants included in test–retest analysis for each NIH Toolbox measure.

Table 4.

Spearman’s Test–Retest Correlations for NIH Toolbox Motor Battery Measures.

	Ages 3 to 7 years (n = 38)		Ages 18 to 85 years (n = 47)
	n	ρ (p)	n	ρ (p)
Locomotion
4-Meter walk time	—	—	40	.26 (.104)
Endurance
2-Minute walk distance	37	.62 (<.001)	37	.71 (<.001)

Note. Assessments of strength, dexterity, and balance were only administered to Spanish-speaking participants once. ns reflect number of participants included in test–retest analysis for each NIH Toolbox measure.

Table 5.

Spearman’s Test–Retest Correlations for NIH Toolbox Cognition Battery Measures.

	Ages 3 to 7 years (n = 38)		Ages 18 to 85 years (n = 48)
	n	ρ (p)	n	ρ (p)
Executive Function and Attention
DCCS	27	.59 (.001)	47	.63 (<.001)
Flanker	30	.69 (<.001)	46	.65 (<.001)
Memory
List Sort	—	—	47	.69 (<.001)
PSM	14	.73 (.003)	20	.77 (<.001)
Processing Speed
Pattern Comp	—	—	47	.75 (<.001)
Language
PVT	38	.75 (<.001)	48	.87 (<.001)
ORRT	—	—	47	.88 (<.001)

Note. DCCS = Dimensional Change Card Sort Test; Flanker = Flanker Inhibitory Control and Attention Test; List Sort = List Sorting Working Memory Test; PSM = Picture Sequence Memory Test; Pattern Comp = Pattern Comparison Processing Speed Test; PVT = Picture Vocabulary Test; ORRT = Oral Reading Recognition Test. ns reflect number of participants included in test–retest analysis for each NIH Toolbox measure.

Sensation Battery

Among adults, analyses supported test–retest reliability based on large effects for the assessments of visual acuity and pain intensity (ρs ranged from .78 to .87). Reliability was lower for assessments of audition, gustation, olfaction, and pain interference, though effects were still medium to large (ρs ranged from .48 to .63). Among children, test–retest reliability was supported for vision based on a large effect (ρ = .69), but was poor for olfaction based on a small effect (ρ = .20; see Table 3).

Motor Battery

Among both children and adults, test–retest reliability was supported for the assessment of endurance based on large effects (ρs ranged from .62 to .71), although it was poor for the assessment of locomotion among adults based on a small effect (ρ = .26; see Table 4).

Cognition Battery

Although all seven measures are appropriate for individuals aged 7 years and older, only one 7-year-old respondent repeated the Cognition Battery. Therefore, test–retest reliability was not evaluated for the measures not included in the Early Childhood Battery (i.e., List Sort Working Memory Test, Pattern Comparison Test, and Oral Reading Recognition Test) among children. Among adults, test–retest reliability was supported by large effects for all tests (ρs ranged from .63 to .88). Similarly, among children, test–retest reliability was demonstrated with large effects for tests of executive function and attention, language, and episodic memory (ρs ranged from .59 to .75; see Table 5).

Convergent Validity: Language Measures

Of the 904 Spanish-language respondents, 385 (ages 3 to 7 years: n = 263; ages 18 to 85 years: n = 122) completed the Batería-III Woodcock-Muñoz Vocabulario Sobre Dibujos test (Muñoz-Sandoval et al., 2005), the legacy measure for the Spanish Picture Vocabulary Test, and 195 (ages 3 to 7 years: n = 103, although only those aged 7 years included in analysis [n = 56]; ages 18 to 85 years: n = 92) completed the 48-item version of the Word Accentuation Test (Del Ser et al., 1997; Krueger et al., 2006), the legacy measure for the Spanish Oral Reading Recognition Test. Among adults, Spearman’s correlations demonstrated good convergent validity based on large effects between the NIH Toolbox Spanish language measures and legacy measures for both the Picture Vocabulary Test (ρ = .76, p < .001) and the Oral Reading Recognition Test (ρ = .65, p < .001). Among children, good convergent validity based on a large effect was found for the Picture Vocabulary Test (ρ = .60, p < .001), although convergent validity was lower with a medium effect for the Oral Reading Recognition Test (ρ = .26, p = .053).

Comparison of Scores With Demographic Characteristics

Sensation Battery

Among adults, significant but small to medium age effects (ρs ranged from −.22 to .44) were found for the Words-in-Noise Test, Visual Acuity Test, Odor Identification Test, and Pain Intensity Survey after controlling for sex. As expected, older age was associated with worse hearing, worse vision, worse olfaction, and greater pain. Among children, after controlling for sex, older age was related to better olfaction based on a medium effect (ρ = .37), and better vision based on a large effect (ρ = −.52; see Table 6). No sex effects were found among adults or children. Although the Words-in-Noise Test of audition is appropriate for respondents age 6 and older, analyses were not conducted for this test among children due to low sample size in the appropriate age range (n = 10).

Table 6.

Effect Sizes for Comparisons of NIH Toolbox Sensation Battery Among demographic Groups.

	Ages 3 to 7 years (n = 496)			Ages 18 to 85 years (n = 408)
	n	Age^a (ρ)	Sex^b (d)	n	Age^a (ρ)	Sex^b (d)
Audition
WIN Left Ear	—	—	—	63	.38	.25
WIN Right Ear	—	—	—	63	.44*	.32
Gustation
Quinine, tongue	—	—	—	340	−.17	.13
Quinine, mouth	—	—	—	340	−.15	.25
Salt, tongue	—	—	—	341	−.10	.04
Salt, mouth	—	—	—	335	−.03	.10
Vision
Visual Acuity	454	−.52*	.01	381	.33*	.25
Olfaction
Odor ID	483	.37*	.11	399	−.22*	.40
Pain
Pain Intensity	—	—	—	388	.23*	.17
Pain Interference	—	—	—	408	.17	.12

Note. WIN = Words-In-Noise Test; Odor ID = Odor Identification; tongue = tip of tongue; mouth = whole mouth. ns reflect number of participants included in demographic comparison analyses for each NIH Toolbox measure.

Adjusted for sex. ^bAdjusted for age.

p < .001.

Motor Battery

Among adults, both small to medium age effects (ρs ranged from .32 to −.42) and small to large sex effects (ds ranged from .24 to .99) were found for nearly all motor domain measures. Younger and male participants performed better. Among children, medium to large age effects (ρs ranged from .48 to −.77) were found for all measures, with older children performing better. Additionally, one small sex effect was found (d = .23), with female children demonstrating better dexterity than male children (see Table 7).

Table 7.

Effect Sizes for Comparisons of NIH Toolbox Motor Battery Among Demographic Groups.

	Ages 3 to 7 years (n = 496)			Ages 18 to 85 years (n = 408)
	n	Age^a (ρ)	Sex^b (d)	n	Age^a (ρ)	Sex^b (d)
Dexterity
9-Hole pegboard, D	465	−.76*	.23* (M > F)	386	.34*	.41
9-Hole pegboard, ND	464	−.77*	.23	385	.32*	.32
Strength
Grip, D	461	.68*	.11	391	−.41*	.91* (M > F)
Grip, ND	461	.71*	.14	390	−.42*	.99* (M > F)
Balance
Standing balance test	375	.48*	.21	251	−.33*	.24* (M > F)
Locomotion
4-Meter walk time	—	—	—	376	−.18	.15
Endurance
2-Minute walk distance	473	.66*	.08	372	−.35*	.42* (M > F)

Note. D = dominant hand; ND = nondominant hand; M = male; F = female. ns reflect number of participants included in demographic comparison analyses for each NIH Toolbox measure.

Adjusted for sex. ^bAdjusted for age.

p < .001.

Emotion Battery

No significant age or sex effects were found among adults. Significantly higher scores were found for older children as compared with younger children on the parent-report measure of Fear-Over Anxious and parent-report measures reflecting positive social interactions based on small effects (ρs ranges from .16 to .22). No other age or sex effects were found among children (see Table 8).

Table 8.

Effect Sizes for Comparisons of NIH Toolbox Emotion Battery Scores Among Demographic Groups.

	Ages 3 to 7 years (n = 496)			Ages 18 to 85 years (n = 408)
	n	Age^a (ρ)	Sex^b (d)	n	Age^a (ρ)	Sex^b (d)
Negative Affect
Anger-Affect	—	—	—	386	−.04	.21
Anger-Hostility	—	—	—	386	−.04	.10
Anger-Physical Aggression	—	—	—	385	−.02	.35
Sadness	—	—	—	388	.13	.14
Fear-Affect	—	—	—	386	.12	.11
Fear-Somatic Arousal	—	—	—	386	.13	<.01
Anger, PR	493	−.06	.11	—	—	—
Fear-Over Anxious, PR	491	.22*	.12	—	—	—
Fear-Separation Anxiety, PR	491	.10	.01	—	—	—
Sadness, PR	493	.02	.02	—	—	—
Psychological Well-Being
Positive Affect	—	—	—	387	−.03	.02
Life Satisfaction	—	—	—	388	.03	.02
Meaning	—	—	—	278	−.03	.10
Life Satisfaction, PR	478	−.09	.12	—	—	—
Social Relationships
Friendship	—	—	—	388	−.08	.13
Loneliness	—	—	—	387	.03	.05
Emotional Support	—	—	—	387	−.11	.18
Perceived Hostility	—	—	—	386	.09	.34
Instrumental Support	—	—	—	387	−.02	.05
Perceived Rejection	—	—	—	386	.16	.30
Social Withdrawal, PR	484	−.04	.09	—	—	—
Positive Peer Interactions, PR	483	.16*	.01	—	—	—
Peer Rejection, PR	487	−.02	.05	—	—	—
Empathic Behaviors, PR	491	.16*	.24	—	—	—
Stress and Self-Efficacy
Perceived Stress	—	—	—	384	.17	.18
Self-Efficacy	—	—	—	383	−.16	.02

Note. PR = Parent-report. ns reflect number of participants included in demographic comparison analyses for each NIH Toolbox measure.

Adjusted for sex. ^bAdjusted for age.

p < .001.

Cognition Battery

Among adults, significant small to medium age effects (ρs ranged from −.36 to −.47) were observed on all measures except the language tests, on which no significant relationships were found. Additionally, significant medium to large associations (ρs ranged from .30 to .59) were found between all cognition measures and education after controlling for age and sex. Younger age and greater educational attainment were associated with better performance. No significant sex effects were found among adults after adjusting for age and education. Among children, significant and large age effects (ρs ranged from .55 to .72) were observed for all Cognition Battery measures after controlling for sex and mother’s education, with older children performing better. Additionally, children demonstrated a positive, although weaker, small adjusted relationship (ρ = .24) between scores on the Dimensional Change Card Sort Test and mother’s education after controlling for age and sex. Children of mothers with greater educational attainment performed better. Finally, females performed better than males on the Picture Vocabulary Test based on a small effect (d = .28; see Table 9).

Table 9.

Effect Sizes for Comparisons of NIH Toolbox Cognition Battery Scores Among Demographic Groups.

	Ages 3 to 7 years (n = 496)				Ages 18 to 85 years (n = 408)
	n	Age^a (ρ)	Mother’s Educ^b (ρ)	Sex^c (d)	n	Age^d (ρ)	Educ^b (ρ)	Sex^e (d)
Executive Function and Attention
DCCS	355	.55*	.24*	.13	337	−.45*	.59*	.04
Flanker	358	.68*	.15	.05	339	−.47*	.59*	.09
Memory
List Sort	—	—	—	—	333	−.36*	.37*	.19
PSM	290	.72*	.18	.16	264	−.36*	.37*	.30
Processing Speed
Pattern Comp	—	—	—	—	338	−.44*	.37*	.06
Language
PVT	409	.64*	.09	.28* (F > M)	338	−.01	.38*	.03
ORRT	—	—	—	—	337	−.18	.30*	.09

Note. Educ = Education; DCCS = Dimensional Change Card Sort Test; Flanker = Flanker Inhibitory Control and Attention Test; List Sort = List Sorting Working Memory Test; PSM = Picture Sequence Memory Test; Pattern Comp = Pattern Comparison Processing Speed Test; PVT = Picture Vocabulary Test; ORRT = Oral Reading Recognition Test; F = female; M = male. ns reflect number of participants included in demographic comparison analyses for each NIH Toolbox measure.

Adjusted for sex and mother’s education. ^bAdjusted for age and sex. ^cAdjusted for age and mother’s education. ^dAdjusted for sex and education. ^eAdjusted for age and education.

p < .001.

Discussion

This study evaluated the reliability and validity of the Spanish-language version of the NIH Toolbox among children aged 3 to 7 years and adults aged 18 to 85 years who participated in the initial norming study. Such information is crucial to the success of researchers working with and attempting to interpret NIH Toolbox scores in this population. Relative to English-language tests, there are relatively few well-validated Spanish-language testing batteries available. Given the ongoing and rapid expansion of the Latinx population in the United States (U.S. Census Bureau, 2018a, 2018b), the Spanish-language version of the NIH Toolbox is poised to be a particularly valuable tool for assessing neurological and behavioral functioning among primary Spanish speakers.

Sensation Battery

For the Sensation Battery, test–retest reliability was strong for vision among both children and adults, and moderate for both olfaction and taste among adults. This is consistent with the English-language versions of the measures (see Supplemental Table 1, available online; Dalton et al., 2013; Rawal et al., 2015; Varma et al., 2013). English-language normative data are not available for the NIH Toolbox assessment of pain intensity because this measure was derived from the Patient-Reported Outcomes Measurement Information System^® (PROMIS^®), which underwent its own separate norming process. However, the test–retest reliability for this measure was somewhat lower for the present Spanish-language sample as compared with a general population sample of 100 individuals who completed the measure in English (Broderick et al., 2013). This may be because the single-item pain assessment is structured to assess a 7-day recall period, and the average time span between administrations for the Sensation Battery was greater than 8 days. Differences in reliability were also found for the Words-in-Noise Test, with stronger support having been found for the English-language version of the measure (Wilson & Burks, 2005), although this may be a function of low sample size in the present study. Finally, measures in the Sensation Battery generally related to age and sex in a manner consistent with prior effects found in English-speaking samples. For example, Friedman et al. (2004) found that older age was associated with increased prevalence of age-related macular degeneration in the U.S. population, consistent with the present finding that performance on the Visual Acuity Test decreased with age. Similarly, Kaneda et al. (2000) found that odor discrimination abilities decreased with age, and both Kaneda et al. and Mojet et al. (2001) found that taste decreased with age. Wilson (2011) found an inverse relationship between age and performance on the Words-in-Noise Test in English. Finally, reported pain has also been shown to increase with age (Johannes et al., 2010). Thus, the present results provide support for the research-related use of the Sensation Battery to assess sensory functioning among Spanish-speaking adults residing in the United States. Additional research with larger sample sizes is needed to effectively evaluate the utility of this battery among Spanish-speaking children.

Motor Battery

For the Motor Battery, test–retest reliability was poor for the 4-meter walk test of locomotion. This measure also performed the worst of all measures in this domain among English-speaking participants in the normative sample (see Supplemental Table 1, available online; Reuben et al., 2013). A recently published analysis of this test conducted with the combined language adult sample from the NIH Toolbox norming study also identified disconcertingly low test–retest reliability for this measure (ICC = .41; Bohannon & Wang, 2019). It may be that the poor reliability is at least in part a function of human error, as the test is hand timed by an administrator and trial length is generally relatively quick. Automated timing may improve these results, although it is generally more burdensome and less practical than hand timing. Additionally, as was observed with the Sensation Battery, measures in the Motor Battery related to age and sex in a manner consistent with prior effects found in other samples. Similar to the present findings, published work conducted with children has reported greater grip strength and dexterity with increased age, and greater grip strength but worse dexterity in boys as compared with girls (Ervin et al., 2014; Omar et al., 2018). Greater grip strength has also been reported in adult men as opposed to adult women (Yorke et al., 2015). Also in adults, decreased dexterity has been found with increasing age (Ruff & Parker, 1993). Deficits in balance have also been observed for adult women relative to men (Wolfson et al., 1994), and have been found to worsen with increasing age in adults (Kalisch et al., 2011) and improve with age in children (Butz et al., 2015). Finally, endurance has been shown to be greater for men as compared with women, and to decrease with age among adults (Gibbons et al., 2001) but increase with age among children (Bohannon et al., 2018). Interestingly, Öberg et al. (1993) reported decreases in gait speed and step length with increasing age and for women as opposed to men; however, no relationships between sociodemographic variables and 4-meter walk time were found in the present study. These associations with demographic variables provide preliminary support for the validity of the Motor Battery for use in research with Spanish-speaking children and adults living in the United States; however, additional research with larger samples is needed to more confidently assess the psychometric strength of this battery for use with this population.

Emotion Battery

The internal consistency reliability of the fixed form measures included in the Emotion Battery was supported by both Cronbach’s alpha values and IRT reliability statistics. The computed values were similar to the Cronbach’s alpha values reported for the English-language version of the NIH Toolbox Emotion Battery, for which alpha values ranged from .83 to .97 among adults and .73 to .92 among children (Salsman et al., 2013). Among adults, no relationships between measures in the Emotion Battery and sociodemographic variables were found. This is generally consistent with Babakhanyan et al. (2018), who evaluated the Emotion Battery data from the NIH Toolbox norming study in both English and Spanish. These authors explored the relationships of scores to the combined effects of numerous sociodemographic variables (i.e., age, education, gender, ethnicity, and household income), and found few significant relationships with only small effect sizes in both languages (English R² ranged from .005 to .048, Spanish R² ranged from .017 to .033). Of note, these authors used a less stringent alpha value of .01, while the present analysis imposed a limit of .001, which may explain the presence of significant relationships in their results but not the present results. In the present study, only three measures in the Emotion Battery were related to age and none to sex among children. However, notably more associations with age and sex were found among children in the English-language normative sample (Paolillo et al., 2018). This may be a function of larger sample size in the English study, thus yielding greater power to identify relationships. Additionally, Paolillo et al. (2018) did not impose any corrections to control for family-wise error, and they included a broader age range for many of the parent-report measures included in the Emotion Battery (i.e., ages 3 to 12 years). That is to say, scores of children aged 8 to 12 years, who were only represented in the English study, may have driven the relationships of scores on measures within the Emotion Battery to sociodemographic variables.

In aggregate, although the Spanish-language Emotion Battery self-report and parent-report measures do not appear to be robust to commonly encountered challenges in the assessment of emotion (e.g., reliable parent-report assessment), these results provide preliminary support for the use of the Emotion Battery in research with Spanish-speaking individuals in the United States.

Cognition Battery

For the Cognition Battery, test–retest reliability was strong across both adults and children, as was observed with the English language subset of the normative sample (see Supplemental Table 1, available online; Akshoomoff et al., 2013; Bauer & Zelazo, 2013; Heaton et al., 2014; Weintraub et al., 2013). Convergent validity of the language measures included in the NIH Toolbox Cognition Battery was strong among adults, further supporting the utility of these measures in Spanish-language adult respondents. Among children, good convergent validity was found for the Picture Vocabulary Test, although it was slightly lower for the Oral Reading Recognition Test. This may be a function of lower sample size, as the Oral Reading Recognition Test is only appropriate for respondents aged 7 years and older. Moreover, as was observed among English speakers, and consistent with expectations, performance on the Cognition Battery was generally associated with younger age and higher education among adults (Heaton et al., 2014; Weintraub et al., 2013), and with older age among children (Akshoomoff et al., 2013; Bauer & Zelazo, 2013). Unlike in the English-language comparison sample, mother’s education level was generally not significantly related to performance on the Cognition Battery among Spanish-speaking children; however, this may be due to the fact that children aged 8 to 15 years were included in the analysis in English (Akshoomoff et al., 2013). Additionally, in the English-language normative sample of children, sex was not significantly related to scores on the Picture Vocabulary Test (Akshoomoff et al., 2013), while it was significantly related, albeit weakly, to scores on this measure in Spanish. Although the causes of sociodemographic differences in cognitive test performance remain poorly understood (Heaton et al., 2014), it is important to note that these relationships are not equivalent for the English- and Spanish-language versions of these tests. This highlights the importance of considering language groups differently when establishing normative standards and evaluating changes in cognitive functioning over time, as the impact of these variables on scoring is likely to differ across language groups. Despite these differences, in total, the present results provide support for the use of the Cognition Battery in research among both children and adults living in the United States who speak Spanish as their primary language.

Limitations and Future Directions

The present study has limitations. Nonnormality of the data required nonparametric analytic approaches, which complicates comparison of findings with prior published studies. Additionally, while the present sample is representative of the areas in the United States where recruitment facilities were located, it is not necessarily representative of all Spanish-speaking individuals living in the United States.

Despite these limitations, the present results provide important support for the use of the Spanish-language version of the NIH Toolbox in research to measure sensory, motor, emotional, and cognitive functioning among Spanish-speaking individuals living in the United States. While the original goal of the NIH Toolbox was to develop measures for use in research studies, there are now numerous published articles providing clinical validation evidence for the English-language version. Most commercially available tests would use this type of evidence to also impute the clinical use of their tests in translations. It is likely that the clinical appropriateness of the NIH Toolbox measures extends to the Spanish version as well. Individual clinicians and clinical researchers are encouraged to determine the appropriateness of the Spanish-language version of the NIH Toolbox for their own needs, and where possible to conduct research to directly clinically validate the Spanish-language measures. Pending appropriate identification of such support, the Spanish-language version of the NIH Toolbox is likely to be useful in tracking outcomes in clinical and epidemiological research across the life span, and in diverse samples including mixed-language participants.

Supplemental Material

Spanish_NIHTB_Reliability_and_Validity_RR2_Supplemental_Table – Supplemental material for Reliability and Validity of the Spanish-Language Version of the NIH Toolbox

Supplemental material, Spanish_NIHTB_Reliability_and_Validity_RR2_Supplemental_Table for Reliability and Validity of the Spanish-Language Version of the NIH Toolbox by Rina S. Fox, Jennifer J. Manly, Jerry Slotkin, John Devin Peipert and Richard C. Gershon in Assessment

Footnotes

Acknowledgements

The authors would like to thank Jennifer Beaumont for providing additional details to facilitate the development of this article. Additionally, the authors would like to thank the participants of the NIH Toolbox norming study for their important contributions.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is funded in whole or in part with Federal funds from the Blueprint for Neuroscience Research, NIH, under contract number HHS-N-260-2006-00007-C, and by the Environmental Influences on Child Health Outcomes (ECHO) program, Office of the Director, NIH, under award number U24OD023319. PROMIS, Patient-Reported Outcomes Measurement Information System, and NIH Toolbox for Assessment of Neurological and Behavioral Function are marks owned by the U. S. Department of Health and Human Services.

ORCID iD

Richard C. Gershon

Supplemental Material

Supplemental material for this article is available online.

References

Akshoomoff

Beaumont

J. L.

Bauer

P. J.

Dikmen

S. S.

Gershon

R. C.

Mungas

Slotkin

Tulsky

Weintraub

Zelazo

P. D.

Heaton

R. K.

(2013). VIII. NIH Toolbox Cognition Battery (CB): Composite scores of crystallized, fluid, and overall cognition. Monographs of the Society for Research in Child Development, 78(4), 119-132. https://doi.org/10.1111/mono.12038

Babakhanyan

McKenna

B. S.

Casaletto

K. B.

Nowinski

C. J.

Heaton

R. K.

(2018). National Institutes of Health Toolbox Emotion Battery for English- and Spanish-speaking adults: Normative data and factor-based summary scores. Patient Related Outcome Measures, 2018(9), 115-127. https://doi.org/10.2147/prom.S151658

Bauer

P. J.

Zelazo

P. D.

(2013). IX. NIH Toolbox Cognition Battery (CB): Summary, conclusions, and implications for cognitive development. Monographs of the Society for Research in Child Development, 78(4), 133-146. https://doi.org/10.1111/mono.12039

Beaumont

J. L.

Havlik

Cook

K. F.

Hays

R. D.

Wallner-Allen

Korper

S. P.

Lai

J.-S.

Nord

Zill

Choi

Yost

K. J.

Ustsinovich

Brouwers

Hoffman

H. J.

Gershon

(2013). Norming plans for the NIH Toolbox. Neurology, 80(11 Suppl. 3), S87-S92. https://doi.org/10.1212/WNL.0b013e3182872e70

Bohannon

R. W.

Wang

Y. C.

(2019). Four-meter gait speed: Normative values and reliability determined for adults participating in the NIH Toolbox study. Archives of Physical Medicine and Rehabilitation, 100(3), 509-513. https://doi.org/10.1016/j.apmr.2018.06.031

Bohannon

R. W.

Wang

Y. C.

Bubela

Gershon

R. C.

(2018). Normative two-minute walk test distances for boys and girls 3 to 17 years of age. Physical & Occupational Therapy in Pediatrics, 38(1), 39-45. https://doi.org/10.1080/01942638.2016.1261981

Bonomi

A. E.

Cella

D. F.

Hahn

E. A.

Bjordal

Sperner-Unterweger

Gangeri

Bergman

Willems-Groot

Hanquet

Zittoun

(1996). Multilingual translation of the Functional Assessment of Cancer Therapy (FACT) quality of life measurement system. Quality of Life Research, 5(3), 309-320. https://doi.org/10.1007/BF00433915

Broderick

J. E.

Schneider

Junghaenel

D. U.

Schwartz

J. E.

Stone

A. A.

(2013). Validity and reliability of Patient-Reported Outcomes Measurement Information System instruments in osteoarthritis. Arthritis Care & Research, 65(10), 1625-1633. https://doi.org/10.1002/acr.22025

Butz

S. M.

Sweeney

J. K.

Roberts

P. L.

Rauh

M. J.

(2015). Relationships among age, gender, anthropometric characteristics, and dynamic balance in children 5 to 12 years old. Pediatric Physical Therapy, 27(2), 126-133. https://doi.org/10.1097/PEP.0000000000000128

10.

Casaletto

K. B.

Umlauf

Marquine

Beaumont

J. L.

Mungas

Gershon

Slotkin

Akshoomoff

Heaton

R. K.

(2016). Demographically corrected normative standards for the Spanish language version of the NIH Toolbox Cognition Battery. Journal of the International Neuropsychological Society, 22(3), 364-374. https://doi.org/10.1017/s135561771500137x

11.

Cella

Hernandez

Bonomi

A. E.

Corona

Vaquero

Shiomoto

Baez

(1998). Spanish language translation and initial validation of the Functional Assessment of Cancer Therapy quality-of-life instrument. Medical Care, 36(9), 1407-1418. https://doi.org/10.1097/00005650-199809000-00012

12.

Cohen

(1992). A power primer. Psychological Bulletin, 112(1), 155-159. https://doi.org/10.1037/0033-2909.112.1.155

13.

Coldwell

S. E.

Mennella

J. A.

Duffy

V. B.

Pelchat

M. L.

Griffith

J. W.

Smutzer

Cowart

B. J.

Breslin

P. A. S.

Bartoshuk

L. M.

Hastings

Victorson

Hoffman

H. J.

(2013). Gustation assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S20-S24. https://doi.org/10.1212/WNL.0b013e3182872e38

14.

Cook

K. F.

Dunn

Griffith

J. W.

Morrison

M. T.

Tanquary

Sabata

Victorson

Carey

L. M.

MacDermid

J. C.

Dudgeon

B. J.

Gershon

R. C.

(2013). Pain assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S49-S53. https://doi.org/10.1212/WNL.0b013e3182872e80

15.

Dalton

Doty

R. L.

Murphy

Frank

Hoffman

H. J.

Maute

Kallen

M. A.

Slotkin

(2013). Olfactory assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S32-S36. https://doi.org/10.1212/WNL.0b013e3182872eb4

16.

Del Ser

Gonzalez-Montalvo

J. I.

Martinez-Espinosa

Delgado-Villapalos

Bermejo

. (1997). Estimation of premorbid intelligence in Spanish people with the Word Accentuation Test and its application to the diagnosis of dementia. Brain and Cognition, 33(3), 343-356. https://doi.org/10.1006/brcg.1997.0877

17.

Eremenco

S. L.

Cella

Arnold

B. J.

(2005). A comprehensive method for the translation and cross-cultural validation of health status questionnaires. Evaluation & the Health Professions, 28(2), 212-232. https://doi.org/10.1177/0163278705275342

18.

Ervin

R. B.

Fryar

C. D.

Wang

C.-Y.

Miller

I. M.

Ogden

C. L.

(2014). Strength and body weight in US children and adolescents. Pediatrics, 134(3), e782-e789. https://doi.org/10.1542/peds.2014-0794

19.

Friedman

D. S.

O’Colmain

B. J.

Munoz

Tomany

S. C.

McCarty

de Jong

Nemesure

Mitchell

Kempen

Congdon

(2004). Prevalence of age-related macular degeneration in the United States. Archives of Ophthalmology, 122(4), 564-572. https://doi.org/10.1001/archopht.122.4.564

20.

Gershon

R. C.

Fox

R. S.

Manly

J. J.

Mungas

D. M.

Nowinski

C. J.

Roney

E. M.

Slotkin

(2020). The NIH Toolbox: Overview, development, and use with Hispanics. Journal of the International Neuropsychological Society, 26(6), 567. https://doi.org/10.1017/S1355617720000028

21.

Gershon

R. C.

Wagster

M. V.

Hendrie

H. C.

Fox

N. A.

Cook

K. F.

Nowinski

C. J.

(2013). NIH Toolbox for assessment of neurological and behavioral function. Neurology, 80(11 Suppl. 3), S2-S6. https://doi.org/10.1212/WNL.0b013e3182872e5f

22.

Gibbons

W. J.

Fruchter

Sloan

Levy

R. D.

(2001). Reference values for a multiple repetition 6-minute walk test in healthy adults older than 20 years. Journal of Cardiopulmonary Rehabilitation and Prevention, 21(2), 87-93. https://doi.org/10.1097/00008483-200103000-00005

23.

Heaton

R. K.

Akshoomoff

Tulsky

Mungas

Weintraub

Dikmen

Jennifer Beaumont

Casaletto

K. B.

Conway

Slotkin

Gershon

(2014). Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults. Journal of the International Neuropsychological Society, 20(6), 588-598. https://doi.org/10.1017/s1355617714000241

24.

Humes

K. R.

Jones

N. A.

Ramirez

R. R.

(2011). Overview of race and Hispanic origin: 2010. U.S. Census Bureau.

25.

Johannes

C. B.

T. K.

Zhou

Johnston

J. A.

Dworkin

R. H.

(2010). The prevalence of chronic pain in United States adults: Results of an Internet-based survey. Journal of Pain, 11(11), 1230-1239. https://doi.org/10.1016/j.jpain.2010.07.002

26.

Kalisch

Kattenstroth

J.-C.

Noth

Tegenthoff

Dinse

H. R.

(2011). Rapid assessment of age-related differences in standing balance. Journal of Aging Research, 2011, Article 160490. https://doi.org/10.4061/2011/160490

27.

Kaneda

Maeshima

Goto

Kobayakawa

Ayabe-Kanamura

Saito

(2000). Decline in taste and odor discrimination abilities with age, and relationship between gustation and olfaction. Chemical Senses, 25(3), 331-337. https://doi.org/10.1093/chemse/25.3.331

28.

Krueger

K. R.

Lam

C. S.

Wilson

R. S.

(2006). The Word Accentuation Test–Chicago. Journal of Clinical and Experimental Neuropsychology, 28(7), 1201-1207. https://doi.org/10.1080/13803390500346603

29.

Lent

Hahn

Eremenco

Webster

Cella

(1999). Using cross-cultural input to adapt the Functional Assessment of Chronic Illness Therapy (FACIT) scales. Acta Oncologica, 38(6), 695-702. https://doi.org/10.1080/028418699432842

30.

McArdle

Carlo

Wilson

(2009). Words-in-Noise-Test: English and Spanish. https://www.asha.org/Events/convention/handouts/2009/0150_McArdle_Rachel/

31.

Mojet

Christ-Hazelhof

Heidema

(2001). Taste perception with age: Generic or specific losses in threshold sensitivity to the five basic tastes? Chemical Senses, 26(7), 845-860. https://doi.org/10.1093/chemse/26.7.845

32.

Muñoz-Sandoval

A. F.

Woodcock

R. W.

McGrew

K. S.

Mather

(2005). Batería III Woodcock-Muñoz. Riverside.

33.

Öberg

Karsznia

Öberg

(1993). Basic gait parameters: Reference data for normal subjects, 10-79 years of age. Journal of Rehabilitation Research & Development, 30(2), 210-223. https://pdfs.semanticscholar.org/019d/91af53b19a72030726b2bfae06ddc1686218.pdf

34.

Omar

M. T.

Alghadir

A. H.

Zafar

Al Baker

(2018). Hand grip strength and dexterity function in children aged 6-12 years: A cross-sectional study. Journal of Hand Therapy, 31(1), 93-101. https://doi.org/10.1016/j.jht.2017.02.004

35.

Paolillo

E. W.

McKenna

B. S.

Nowinski

C. J.

Thomas

M. L.

Malcarne

V. L.

Heaton

R. K.

(2018). NIH Toolbox® emotion batteries for children: Factor-based composites and norms. Assessment, 27(3), 607-620. https://doi.org/10.1177/1073191118766396

36.

Pew Research Center. (2013). National Survey of Latinos and Religion. https://www.pewhispanic.org/datasets/

37.

Rawal

Hoffman

H. J.

Honda

Huedo-Medina

T. B.

Duffy

V. B.

(2015). The taste and smell protocol in the 2011–2014 US National Health and Nutrition Examination Survey (NHANES): Test–retest reliability and validity testing. Chemosensory Perception, 8(3), 138-148. https://doi.org/10.1007/s12078-015-9194-7

38.

Reuben

D. B.

Magasi

McCreath

H. E.

Bohannon

R. W.

Wang

Y. C.

Bubela

D. J.

Rymer

W. Z.

Beaumont

Rine

R. M.

Lai

J.-S.

Gershon

R. C.

(2013). Motor assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S65-S75. https://doi.org/10.1212/WNL.0b013e3182872e01

39.

Rine

R. M.

Schubert

M. C.

Whitney

S. L.

Roberts

Redfern

M. S.

Musolino

M. C.

Roche

J. L.

Steed

D. P.

Corbin

Lin

C.-C.

Marchetti

G. F.

Beaumont

Carey

J. P.

Shepard

N. P.

Jacobson

G. P.

Wrisley

D. M.

Hoffman

H. J.

Furman

Slotkin

(2013). Vestibular function assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S25-S31. https://doi.org/10.1212/WNL.0b013e3182872c6a

40.

Ruff

R. M.

Parker

S. B.

(1993). Gender-and age-specific changes in motor speed and eye-hand coordination in adults: Normative values for the Finger Tapping and Grooved Pegboard Tests. Perceptual and Motor Skills, 76(3 Suppl.), 1219-1230. https://doi.org/10.2466/pms.1993.76.3c.1219

41.

Ryan

(2013). Language use in the United States: 2011: American Community Survey Reports. https://www2.census.gov/library/publications/2013/acs/acs-22/acs-22.pdf

42.

Salsman

J. M.

Butt

Pilkonis

P. A.

Cyranowski

J. M.

Zill

Hendrie

H. C.

Kupst

M. J.

Kelly

M. A. R.

Bode

R. K.

Choi

S. W.

Lai

J.-S.

Griffith

J. W.

Stoney

C. M.

Brouwers

Knox

S. S.

Cella

(2013). Emotion assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S76-86. https://doi.org/10.1212/WNL.0b013e3182872e11

43.

Sinha

Wong

A. W. K.

Kallogjeri

Piccirillo

J. F.

(2018). Baseline cognition assessment among patients with oropharyngeal cancer using PROMIS and NIH Toolbox. JAMA Otolaryngology—Head & Neck Surgery, 144(11), 978-987. https://doi.org/10.1001/jamaoto.2018.0283

44.

Troller-Renfree

S. V.

Barker

T. V.

Pine

D. S.

Fox

N. A.

(2015). Cognitive functioning in socially anxious adults: Insights from the NIH Toolbox Cognition Battery. Frontiers in Psychology, 6, Article 764. https://doi.org/10.3389/fpsyg.2015.00764

45.

Tulsky

D. S.

Heinemann

A. W.

(2017). The clinical utility and construct validity of the NIH Toolbox Cognition Battery (NIHTB-CB) in individuals with disabilities. Rehabilitation Psychology, 62(4), 409-412. https://doi.org/10.1037/rep0000201

46.

U.S. Census Bureau. (2018a). American Fact Finder. https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml

47.

U.S. Census Bureau. (2018b). Projected race and Hispanic origin: Main projections series for the United States, 2017-2060. https://www.census.gov/data/tables/2017/demo/popproj/2017-summary-tables.html

48.

Varma

McKean-Cowdin

Vitale

Slotkin

Hays

R. D.

(2013). Vision assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S37-S40. https://doi.org/10.1212/WNL.0b013e3182876e0a

49.

Weintraub

Dikmen

S. S.

Heaton

R. K.

Tulsky

D. S.

Zelazo

P. D.

Bauer

P. J.

Carlozzi

N. E.

lotkin

Blitz

Wallner-Allen

Fox

N. A.

Beaumont

J. L.

Mungas

Nowinski

C. J.

Richler

Deocampo

J. A.

Anderson

J. E.

Manly

J. J.

Borosh

. . . Gershon

R. C.

(2013). Cognition assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S54-S64. https://doi.org/10.1212/WNL.0b013e3182872ded

50.

Wilson

R. H.

(2011). Clinical experience with the words-in-noise test on 3430 veterans: Comparisons with pure-tone thresholds and word recognition in quiet. Journal of the American Academy of Audiology, 22(7), 405-423. https://doi.org/10.3766/jaaa.22.7.3

51.

Wilson

R. H.

Burks

C. A.

(2005). Use of 35 words for evaluation of hearing loss in signal-to-babble ratio: A clinic protocol. Journal of Rehabilitation Research and Development, 42(6), 839-852. https://doi.org/10.1682/JRRD.2005.01.0009

52.

Wolfson

Whipple

Derby

C. A.

Amerman

Nashner

(1994). Gender differences in the balance of healthy elderly as demonstrated by dynamic posturography. Journal of Gerontology, 49(4), M160-M167. https://doi.org/10.1093/geronj/49.4.M160

53.

Woodcock

R. W.

McGrew

K. S.

Mather

(2001). Woodcock-Johnson III. Riverside.

54.

Yorke

A. M.

Curtis

A. B.

Shoemaker

Vangsnes

(2015). Grip strength values stratified by age, gender, and chronic disease status in adults aged 50 years and older. Journal of Geriatric Physical Therapy, 38(3), 115-121. https://doi.org/10.1519/jpt.0000000000000037

55.

Zecker

S. G.

Hoffman

H. J.

Frisina

Dubno

J. R.

Dhar

Wallhagen

Kraus

Griffith

J. W.

Walton

J. P.

Eddins

D. A.

Newman

Victorson

Warrier

C. M.

Wilson

R. H.

(2013). Audition assessment using the NIH Toolbox. Neurology, 80(11 Suppl. 3), S45-S48. https://doi.org/10.1212/WNL.0b013e3182872dd2

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB