Rates and Predictors of Invalid Baseline Test Performance in High School and Collegiate Athletes for 3 Computerized Neurocognitive Tests

Abstract

Background:

Preseason baseline testing using computerized neurocognitive tests (CNTs) is increasingly performed on athletes. Adequate effort is critical to establish valid estimates of ability, but many users do not evaluate performance validity, and the conditions that affect validity are not well understood across the available CNTs.

Purpose:

To examine the rates and predictors of invalid baseline performance for 3 popular CNTs: Automated Neuropsychological Assessment Metrics (ANAM), Axon Sports, and Immediate Post-Concussion and Cognitive Testing (ImPACT).

Study Design:

Controlled laboratory study.

Methods:

High school and collegiate athletes (N = 2063) completed 2 of 3 CNTs each during preseason evaluations. All possible pairings were present across the sample, and the order of administration was randomized. Examiners provided 1-on-1, scripted pretest instructions, emphasizing the importance of good effort. Profile validity was determined by the manufacturers’ standard criteria.

Results:

The overall percentage of tests flagged as of questionable validity was lowest for ImPACT (2.7%) and higher for ANAM and Axon (10.7% and 11.3%, respectively). The majority of invalid baseline profiles were flagged as such because of failure on only 1 validity criterion. Several athlete and testing factors (eg, attention deficit hyperactivity disorder [ADHD], estimated general intellectual ability, administration order) predicted validity status for 1 or more CNTs. Considering only first CNT administrations and participants without ADHD and/or a learning disability (n = 1835) brought the rates of invalid baseline performances to 2.1%, 8.8%, and 7.0% for ImPACT, ANAM, and Axon, respectively. Invalid profiles on the Medical Symptom Validity Test (MSVT) were rare (1.8% of participants) and demonstrated poor correspondence to CNT validity outcomes.

Conclusion:

The validity criteria for these CNTs may not identify the same causes of invalidity or be equally sensitive to effort. The validity indicators may not be equally appropriate for some athletes (eg, those with neurodevelopmental disorders).

Clinical Relevance:

The data suggest that athletes do not put forth widespread low effort or that some validity criteria are more sensitive to invalid performance than others. It is important for examiners to be aware of the conditions that maximize the quality of baseline assessments and to understand what sources of invalid performance are captured by the validity criteria that they obtain.

Keywords

baseline testing computerized testing neurocognitive testing validity concussion mild traumatic brain injury ImPACT ANAM Axon Sports

Neurocognitive testing is widely recognized as an important component of the examination of concussed athletes, as it allows for the objective detection of subtle cognitive changes common in the acute postconcussive period.¹ Because of their numerous purported benefits over traditional paper-and-pencil tests, computerized neurocognitive tests (CNTs) have gained popularity with many athletic programs in recent years. A number of CNTs are commercially available, including Automated Neuropsychological Assessment Metrics (ANAM), Axon Sports (recently renamed the Cogstate Sport Computerized Cognitive Assessment tool and also referred to as CogSport or the Cogstate Brief Battery), and Immediate Post-Concussion and Cognitive Testing (ImPACT).^5,8,18,25,37 Among the potential advantages of these programs are (1) widely accessible Internet-based/electronic platforms, (2) highly standardized test administration and scoring procedures, (3) ready access to numerous alternate test forms (eg, by automatic selection of items from a larger test bank) for use with repeat testing, and (4) storage of performance data in centralized data repositories for the benefit of users and ongoing test development efforts.^11,31

Another major reason why CNTs have become so commonly used is that they make it operationally feasible for athletic programs to perform widespread preseason baseline testing of their athletes. Although there is some debate regarding the value that baseline testing adds to postinjury assessments,^15,38,39 preseason baseline testing is undoubtedly an increasingly common practice, with 94.7% of athletic trainers who use ImPACT reporting that they baseline test their athletes.¹² Clearly, having baseline performance data on injured athletes allows clinicians to take into account premorbid cognitive skills; yet, a number of questions have also been raised regarding how baseline scores are obtained in practice and whether athletes’ motivations preclude valid assessments. For example, the group settings in which athletes commonly complete baseline testing negatively affect performance for some athletes,^22,23 and the availability and ease of use of CNTs for baseline testing have probably led to the tools being used by people who are inadequately trained in psychometrics and test interpretation.²⁹

Yet, even under optimal testing conditions and with the most experienced examiners, many athletes may be unmotivated to perform their best at baseline testing, given that, if concussed, they will have to achieve comparable CNT scores before being cleared to return to play. Using well-established paper-and-pencil neuropsychological measures, for example, 11% of high school football players in one sample failed formal effort tests.²¹ Although invalid baseline rates have been highly variable across CNT studies, some have found over a quarter (27.9%-30.3%) of athletes’ baseline CNT profiles to be flagged as of questionable validity.^16,43 Consequently, it is critical that examiners take steps to maximize the validity of baseline test scores. All of the major CNT publishers readily provide flags of potentially poor effort, but these indices are not readily used by many examiners (eg, in the survey of athletic trainers using ImPACT mentioned above, only 51.9% of the athletic trainers who baseline test their athletes indicated that they examine the validity output of these baseline scores).¹² Further, the relative sensitivity of the various CNTs’ validity criteria and the individual factors that may inadvertently affect validity profiles are not well established.

Overall rates of invalid baseline performances for ImPACT have been reported by several authors, and these rates have varied considerably across studies. At the low end, Moser and colleagues³⁰ reported a failure rate of 0.6% in a sample of high school athletes who were scheduled and accompanied by their parents to an individual testing session in which performance feedback was provided; another sample yielded 27.9%⁴³ invalid baselines. While it is somewhat difficult to make inferences across studies about the role of certain demographic factors (eg, age) in the rates of invalid baselines (because of the evolution of validity criteria over time and differences in testing protocols across studies), the overall invalid percentages in high school, collegiate, and professional (National Football League) samples have been reported to range from 0.4% to 11.9% (high school),^{9,30,36,37,45} 4.1% to 27.9% (collegiate),^33,36,43 and 2.2% to 5.4% (professional).^40,41 Probably more important than age or level of competition is testing environment (with large groups yielding more invalid baselines),²² age by group size interactions (with athletes aged 10-12 years more sensitive than those aged 13-18 years to the effects of group size),²³ and risk factors for distractibility such as attention deficit hyperactivity disorder (ADHD).³⁶ Invalid rates for Axon have varied widely (1.0% in a university boxing sample,²⁸ 4.7% in a high school sample,²⁴ 6.1% in Norwegian soccer players,⁴² and 30.3% in Division I university athletes¹⁶). To our knowledge, rates of invalid baseline performance have not been reported in an athlete sample for ANAM.

Perhaps with more extensive documentation of the performance and properties of the validity criteria of the various CNTs will come increased adherence to guidelines mandating practitioners to estimate the validity of each baseline assessment.^2,29 In this study, we examined the rates and predictors of invalid performance for 3 popular CNTs—ANAM, Axon, and ImPACT—obtained at preseason baseline evaluations in the same sample, with the aim of informing researchers and clinicians about their properties such that they may be more readily utilized by clinicians. Given the literature reviewed, we hypothesized that ADHD (and perhaps learning disability) would be associated with higher rates of invalidity for the 3 CNTs. However, the majority of the measures explored in this study were novel and, consequently, the analyses necessarily exploratory. Similarly, this is the first study to directly compare the rates of invalid baselines for these 3 CNTs; it was unclear to what extent we would observe differences in overall rates of invalidity across the 3 CNTs (1 prior study of a military sample reported invalid rates for ANAM, Axon, and ImPACT of 3.8%, 7.0%, and 12.0%, respectively, although the group sample sizes were small, and no direct statistical comparison of these rates was reported).¹⁰

Methods

Participants

As part of a larger study on the assessment of sport-related concussion and contact and collision sport, student-athletes from 9 high schools and 4 colleges in southeastern Wisconsin completed baseline testing between August 2012 and October 2014. Informed consent was obtained for 2154 participants. Participants (N = 2063) who completed at least 1 CNT were included in the sample for these analyses. On rare occasions, there were problems with the administration of the CNTs, or athletes did not complete the entire baseline session, and as a result, 24 (1.2%) of these participants only have data for 1 CNT. Sample demographics are summarized in Table 1. Adult athletes and parents of minor athletes provided informed consent, and minor participants completed assent forms before baseline testing. Participants were compensated $30 for their time and effort in completing baseline assessments. All testing procedures were approved by the institutional review board at the Medical College of Wisconsin.

Table 1

Sample Characteristics (N = 2063)^a

Characteristics	Mean ± SD or %
Male sex	76.8
Age, y	17.8 ± 1.9
College level of competition (vs high school)	60.0
Race
White	83.4
Black	12.1
Asian	1.2
Native Hawaiian/Pacific Islander	0.6
American Indian/Alaska Native	0.5
Unknown	2.1
ADHD	7.8
Learning disability	3.6
GPA	3.28 ± 0.52
WTAR standard score	101.43 ± 12.62
No. of prior concussions	0.64 ± 0.94
Sport
Football	48.3
Soccer	32.6
Lacrosse	7.4
Wrestling	3.7
Ice hockey	3.7
Rugby	2.5
Field hockey	1.8

ADHD, attention deficit hyperactivity disorder; GPA, grade point average; WTAR, Wechsler Test of Adult Reading.

Baseline Testing Session

The baseline testing protocol consisted of the following in order: contact information, demographics/health history (gathered by 1-on-1 interview), Wechsler Test of Adult Reading (WTAR),⁴⁴ CNT No. 1, Standard Assessment of Concussion (SAC),²⁶ Sport Concussion Assessment Tool–3rd edition (SCAT3) symptom checklist,²⁷ CNT No. 2, Green’s Medical Symptom Validity Test (MSVT),¹⁹ Satisfaction With Life Scale (SWLS),¹⁴ Brief Symptom Inventory 18 (BSI-18),¹³ and Balance Error Scoring System (BESS).²⁰ Tests were individually proctored by a research assistant in quiet settings with computers positioned to minimize distractions. Testing group sizes ranged from 1 to 20 athletes. Each athlete was read a standardized script at the beginning of the baseline testing session and before each of the CNTs about the importance of valid baseline tests (see the Appendix, available in the online version of this article at http://ajsm.sagepub.com/supplemental). Testing sessions lasted approximately 90 minutes.

Each athlete took 2 of 3 CNTs: ANAM (v 4.3; Vista Life Sciences), Axon Sports (Cogstate Ltd), and ImPACT (online version; ImPACT Applications Inc). CNT pairing groups were assigned to each school, with the aim of balancing the demographic distribution across CNTs. The overall distribution of CNT pairings (separately by order) across the sample was 16.2% ANAM-Axon, 14.3% Axon-ANAM, 17.4% ANAM-ImPACT, 16.2% ImPACT-ANAM, 17.9% Axon-ImPACT, and 18.0% ImPACT-Axon. For each participant, the order of administration was selected at random by a computer algorithm. The reliability and validity of ANAM,^5,10 Axon,^10,25 and ImPACT^7,8,10,33 have been reported elsewhere.

Computerized Neurocognitive Tests

ANAM

The ANAM test battery was originally developed by the Department of Defense for the assessment of processing speed and cognitive efficiency. It has been used in predeployment and postdeployment evaluations and has been adapted for the assessment of sport-related concussion.⁶ The version used in this study included 8 subtests: Simple Reaction Time (SRT), Code Substitution–Learning (CDS), Procedural Reaction Time (PRO), Mathematical Processing (MTH), Matching to Sample (M2S), Code Substitution–Delayed (CDD), Simple Reaction Time 2 (SR2), and Go/No-Go (GNG).

Axon

The Axon Sports CNT is composed of 4 tasks—Processing Speed (PS; simple reaction time), Attention (AT; choice reaction time), Learning (LN; visual recognition memory), and Working Memory (WM; one back)—which map onto the Cogstate tasks of Detection, Identification, Visual (One Card) Learning, and One Back, respectively.

ImPACT

ImPACT was developed for the baseline and postinjury assessments of concussed athletes and is composed of 6 tasks—Word Memory Learning (WML), Design Memory Learning (DML), Xs and Os (XO), symbol match, color match, and Three Letters (TL)—which yield the following composite scores: Verbal Memory, Visual Memory, Visual Motor Speed, Reaction Time, and Impulse Control.

Validity Criteria

Test validity was determined by the standard output available from each CNT manufacturer, with the most updated versions of the validity criteria at the time of data analysis used for all participants. The criteria for each CNT are presented in Table 2. ImPACT indicates invalid baseline scores using a mark (“++”) on the report, with a note indicating that the data may be invalid. ANAM provides profile validity indices in a separate effort report document and divides its criteria into 2 categories: “Consider Retest” for accuracy scores so low (below or equal to chance) as to suggest poorly understood instructions, and a “Questionable Effort” checkbox that reflects poor performance on an index developed to discriminate between those putting forth good effort versus those who were instructed to put forth suboptimal effort.³² Any profile with 1 or both of these criteria checked was coded as invalid for our purposes.

Table 2

Validity Criteria for ANAM, Axon, and ImPACT^a

ANAM	Axon (CogSport)	ImPACT
Consider Retest	Integrity Check
SRT accuracy <56%	PS accuracy >80%	ICC >30
CDS accuracy <56%	AT accuracy >80%	WML <69%
PRO accuracy <56%	LN accuracy >53%	DML <50%
MTH accuracy <56%	WM accuracy >53%	XO total incorrect >30
M2S accuracy <56%	PS speed < AT speed	TL total correct <8
CDD accuracy <56%	PS speed < WM speed
SR2 accuracy <56%
Questionable Effort

ANAM and ImPACT criteria are labeled as invalidity criteria, whereas Axon’s criteria are scaled in the direction of validity. ANAM’s Questionable Effort index was derived to differentiate between athletes instructed to put forth suboptimal versus good effort and is described by Roebuck-Spencer et al.³² XO total incorrect refers to errors in the interference trials. AT, Attention; CDD, Code Substitution–Delayed; CDS, Code Substitution–Learning; DML, Design Memory Learning; ICC, Impulse Control Composite; LN, Learning; M2S, Matching to Sample; MTH, Mathematical Processing; PRO, Procedural Reaction Time; PS, Processing Speed; SR2, Simple Reaction Time 2; SRT, Simple Reaction Time; TL, Three Letters; WM, Working Memory; WML, Word Memory Learning; XO, Xs and Os.

For Axon, the primary “Integrity Check” criteria most prominent in the report output are listed in Table 3, although the test also flags profiles of questionable validity because of rare scores (ie, test scores >2 SDs below the mean). Because of the high degree of overlap between this criterion and the Integrity Check criteria, and the fact that it only independently accounted for a small number of invalid profiles (n = 4, or 2.6% of invalid Axon profiles), this was excluded from Table 3 but is discussed in the Results section.

Table 3

Rates of Invalid Baseline Profiles and Failure on Individual Validity Criteria^a

	ANAM (n = 1313)		Axon (n = 1355)		ImPACT (n = 1434)
Baseline profiles flagged as invalid, %	10.7		11.3		2.7
	Validity Criterion	%	Validity Criterion	%	Validity Criterion	%
Relative frequency of failures (invalid profiles only)	Consider Retest		Integrity Check
	SRT accuracy <56%	1.4	PS accuracy >80%	8.5	ICC >30	7.7
	CDS accuracy <56%	0.7	AT accuracy >80%	20.9	WML <69%	7.7
	PRO accuracy <56%	8.6	LN accuracy >53%	83.0	DML <50%	46.2
	MTH accuracy <56%	23.6	WM accuracy >53%	14.4	XO total incorrect >30^b	7.7
	M2S accuracy <56%	35.0	PS speed < AT speed	9.2	TL total correct <8	43.6
	CDD accuracy <56%	38.6	PS speed < WM speed	4.6
	SR2 accuracy <56%	0.0
	Questionable Effort	12.9
	Failed Criteria	%	Failed Criteria	%^c	Failed Criteria	%
Total validity criteria failed	1	85.0	1	79.1	1	89.7
	2	10.7	2	8.5	2	7.7
	3	2.9	3	3.9	3	2.6
	4	1.4	4	0.7	4+	0.0
	5+	0.0	5	1.3
			6	3.9

AT, Attention; CDD, Code Substitution–Delayed; CDS, Code Substitution–Learning; DML, Design Memory Learning; ICC, Impulse Control Composite; LN, Learning; M2S, Matching to Sample; MTH, Mathematical Processing; PRO, Procedural Reaction Time; PS, Processing Speed; SR2, Simple Reaction Time 2; SRT, Simple Reaction Time; TL, Three Letters; WM, Working Memory; WML, Word Memory Learning; XO, Xs and Os.

XO total incorrect refers to errors in the interference trials.

The 2.6% of invalid Axon profiles not accounted for by these criteria were flagged as invalid based on scores that were >2 SDs below the mean but that did not meet these other criteria.

Statistical Analysis

Overall percentages of invalid baseline test performance were computed for each CNT along with 95% CIs. For those profiles that were flagged as invalid, we then computed the frequency of failure on each individual validity criterion. Predictors of baseline validity status (valid, invalid) were explored using simple logistic regression for each CNT, given the high degree of overlap among some predictors explored (Table 4). Predictors considered included sex, age, sport, ADHD, learning disability, number of prior concussions, grade point average (GPA), estimated verbal intellectual ability (WTAR standard score), and order of CNT administration (first or second CNT taken in the testing session). Variables related to participant demographics and history (GPA, ADHD, learning disability, prior concussions) were gathered via self-report. As a large number of participants completing ImPACT had taken the test before (56.2%), prior exposure to ImPACT was also explored as a predictor of baseline validity for this measure (the percentage of participants who had previously taken ANAM and Axon were too low—0.1% and 1.1%, respectively—to include this variable in the analysis of those tests). Because of the exploratory nature of these analyses and the multiple comparisons that were performed for each measure, we applied the false discovery rate control method³ and indicate in Table 4 which findings remained significant after applying this correction. This approach is a sequential Bonferroni-type procedure that, unlike traditional Bonferroni correction (which controls the familywise error rate), is aimed at controlling the expected proportion of incorrectly rejected null hypotheses (“false discoveries”) and, consequently, better preserves statistical power while also providing a reasonable degree of control of type I errors.^3,4

Table 4

Results of Univariate Logistic Regression on Predictors of CNT Validity^a

	ANAM		Axon		ImPACT
	OR (95% CI)	P Value	OR (95% CI)	P Value	OR (95% CI)	P Value
Categorical Variables
Male sex	1.18 (0.74 to 1.89)	.570	1.33 (0.88 to 1.99)	.373	0.84 (0.41 to 1.70)	.709
Level of competition (HS)	1.16 (0.81 to 1.67)	.544	0.85 (0.61 to 1.20)	.489	2.12 (1.11 to 4.05)	.092
ADHD (yes)	2.25 (1.36 to 3.72)	.005	3.18 (2.02 to 5.02)	<.001	1.73 (0.60 to 5.00)	.461
Learning disability (yes)	3.36 (1.80 to 6.25)	<.001	1.84 (0.91 to 3.74)	.226	2.02 (0.47 to 8.70)	.461
CNT order (2nd)	1.35 (0.95 to 1.92)	.232	1.91 (1.35 to 2.72)	.001	1.40 (0.73 to 2.67)	.461
Taken the CNT before (no)	—	—	—	—	1.22 (0.63 to 2.34)	.679
	Estimate (95% CI)	P Value	Estimate (95% CI)	P Value	Estimate (95% CI)	P Value
Continuous Variables
Age	−0.05 (−0.15 to 0.04)	.501	−0.03 (−0.11 to 0.06)	.715	−0.17 (−0.35 to 0.00)	.153
No. of prior concussions	−0.05 (−0.24 to 0.13)	.631	0.04 (−0.14 to 0.22)	.768	0.17 (−0.14 to 0.49)	.461
GPA	−0.55 (−0.88 to −0.22)	.005	−0.83 (−1.14 to −0.52)	<.001	−0.96 (−1.58 to −0.35)	.035
WTAR standard score	−0.05 (−0.06 to −0.03)	<.001	−0.04 (−0.06 to −0.03)	<.001	−0.02 (−0.05 to 0.01)	.268

P values were adjusted for multiple comparisons using the false discovery rate method.³ P values are bolded where significant (< .05). ADHD, attention deficit hyperactivity disorder; CNT, computerized neurocognitive test; GPA, grade point average; HS, high school (vs college); OR, odds ratio; WTAR, Wechsler Test of Adult Reading.

As is described in the Results section, analyses were also conducted to directly statistically compare the 3 CNTs on relevant variables (eg, frequency of invalid baselines) using generalized estimating equations (GEEs) with a logit link function to account for repeated measures. In addition, multiple logistic regression was conducted to compare the 3 CNTs for the subset excluding ADHD, learning disability, and second test order; variable selection was based on univariate GEE analyses using an inclusion criteria of P < .05. First order interactions among the significant variables were examined, and none of them was significant.

Results

Percentage of Invalid Baseline Performances by CNT

Table 3 presents the overall percentage of invalid baseline profiles by CNT. Overall, the percentage of athletes who produced invalid profiles was lowest for ImPACT (2.7% [95% CI, 1.9%-3.7%]) and higher for ANAM (10.7% [95% CI, 9.9%-12.5%]) and Axon (11.3% [95% CI, 9.7%-13.1%]). Statistical comparisons of validity status for each CNT pair (using GEEs with a logit link function to account for repeated measures) revealed equivalent odds of producing invalid baselines for ANAM versus Axon (odds ratio [OR], 0.95; P = .686) and higher odds for Axon and ANAM versus ImPACT (OR, 4.20 [ANAM vs ImPACT] and 4.41 [Axon vs ImPACT]; Ps < .001). Of the 2039 participants who completed 2 CNTs, 1731 (84.9%) produced valid profiles on both CNTs administered, while 284 (13.9%) produced 1 invalid CNT, and only 24 (1.2%) produced invalid profiles on both CNTs completed. Considering CNT and MSVT validity jointly, only 1 participant failed all 3 tests, while 34 (1.7%) failed 2 tests, and 296 (14.5%) failed 1 test.

Table 3 also lists the rates of failure on each validity criterion as well as the number of validity criteria marked as failed for each CNT. Across ANAM, Axon, and ImPACT, the overwhelming majority (79%-90%) of participants who produced invalid baselines did so because of failure on only 1 of the test’s core validity criteria. As mentioned earlier, Axon also flags profiles as of questionable validity because of test scores that are rare normatively (>2 SDs below the mean). Failure on this criterion overlapped significantly with the criteria listed in Table 3 (only 4, or 2.6% of Axon profiles, were flagged as invalid solely because of this additional criterion). Similarly, only 2.1% of ANAM baselines were flagged as invalid solely because of the Questionable Effort criterion.

ANAM profiles were most likely to be flagged as invalid because of failure on the MTH (23.6% of invalid profiles), M2S (35.0%), and CDD (38.6%) tests. For Axon, the most common criterion failed was that for LN accuracy (83.0% of invalid profiles), while for ImPACT, the most common criteria failed were those for DML (46.2% of invalid profiles) and TL total correct (43.6%).

Predictors of Invalid CNT Profiles

Table 4 presents the extent to which various personal and test administration variables predicted the validity status of each CNT profile. Lower GPA predicted invalid baselines for all tests. Similarly, a lower WTAR standard score and a history of ADHDⁱ predicted invalid baseline performance for ANAM and Axon, and a learning disability additionally predicted invalid test profiles for ANAM. In fact, 25.9% of athletes with ADHD (vs 9.9% without ADHD) produced invalid baselines on Axon. Similarly, 20.0% of participants with ADHD (vs 10.0% without), and 27.3% of participants with a learning disability (vs 10.0% without), achieved invalid baselines on ANAM. Further, participants who took Axon as their second CNT had higher odds (OR, 1.91 [95% CI, 1.35-2.72]) of achieving an invalid profile than those who took Axon first. For ImPACT, lower age (and playing at the high school level) was modestly associated with higher odds of invalidity, but these predictors became nonsignificant after adjustment for multiple comparisons. Select sports showed differential patterns of validity for Axon and ImPACT, but these associations were no longer significant after accounting for group differences in other variables (ADHD, GPA, WTAR). Sex and history of concussions (all CNTs) were not predictive of validity status on any CNT, and similarly, familiarity with ImPACT was not predictive of validity status.

Given the influence of test order, ADHD, and learning disability on some of the CNTs, the overall rate of invalid baselines was recomputed for each CNT, excluding second test order and participants with ADHD and/or a learning disability (n = 1835) to yield a fairer comparison of expected baseline rates under more typical testing conditions in the majority of athletes. This yielded an invalid baseline frequency of 8.8% for ANAM, 7.0% for Axon, and 2.1% for ImPACT. Direct comparison of validity status, excluding ADHD, learning disability, and second test order, and additionally adjusting for GPA and WTAR score (as these factors predicted validity status) yielded similar overall findings to unadjusted estimates, with ANAM and Axon showing equivalent odds of producing invalid baselines (OR, 0.92; P = .522) that were both higher than those for ImPACT (OR, 4.36 [ANAM vs ImPACT] and 4.73 [Axon vs ImPACT]; Ps < .001).

Comparison of CNT Validity to the MSVT

The majority (98%) of the sample also completed the MSVT as an additional measure of effort. Failure of ≥1 effort indices on the MSVT was rare (n = 36, or 1.8%, of all participants). Exploratory analyses were conducted to examine the overlap in validity profiles between the CNTs and MSVT. Agreement between the profile validity for each CNT and the MSVT was poor. For example, all 35 participants who produced invalid ImPACT profiles passed the MSVT, while all 26 participants who failed the MSVT but completed ImPACT produced valid ImPACT profiles (Cohen κ = −0.02). Agreement was similarly low when comparing profile validity for the MSVT and ANAM (κ = 0.01) and the MSVT and Axon (κ = 0.08).

Discussion

Establishing valid estimates of athletes’ premorbid cognitive abilities is critical to maximizing the utility of preseason baseline testing for concussion management programs. Although CNTs facilitate the estimation of performance validity using embedded measures that are readily available to examiners, prior work suggests that practitioners underutilize these indices. Here, we reported the rates and predictors of invalid baseline test performance for 3 popular CNTs—ANAM, Axon, and ImPACT—gathered within the same sample of athletes, with the aim of informing users of these tests about the performance and properties of each CNT’s standard validity indices.

The overall rate at which these profiles were flagged as invalid varied by test, with the percentage of invalid ANAM and Axon profiles higher than that of ImPACT. This finding may have several implications, which are not readily teased apart using these data. On the one hand, differences in the rates of invalid baselines may be caused by variably stringent validity criteria across CNTs. Probably more accurate is that the validity criteria for each test are differentially sensitive to differing sources of invalidity. Test scores may be invalid for a variety of reasons, including technological issues during test administration, reading/comprehension problems, fatigue, environmental distractions, and low motivation (which itself reflects a continuum and includes both participants who are not highly motivated to try their best as well as those, probably rarer, athletes who intentionally underperform or “sandbag” to a high degree). Further, legitimate, premorbid cognitive difficulties or neurodevelopmental disorders (eg, ADHD, learning disability) may cause some well-intentioned athletes giving full effort to fail performance validity measures, particularly when those measures are highly stringent or sensitive to the cognitive difficulties associated with these disorders.

As outright sandbagging (ie, putting forth low effort or intentionally working to produce deflated scores) is probably relatively rare, lenient criteria that only require test accuracy to exceed chance levels would probably most commonly capture issues related to technical issues, poor comprehension of test instructions (including left-right confusion on tests that are susceptible to this), and guessing. Each CNT used in this study contains some criteria that appear to be capable of capturing these types of issues. However, it may be that more stringent or sophisticated criteria are needed to capture more subtle sources of invalid performance such as lower levels of suboptimal effort or periodic distraction.

ANAM explicitly provides an empirically derived index (labeled Questionable Effort) aimed at identifying people who are knowingly putting forth poor effort. Axon, while not explicitly labeling its criteria so clearly, also contains requirements more stringent than those requiring chance performance, including criteria that appear to be aimed at capturing inconsistent and unusual patterns of performance (which may indicate issues with engagement in or effort on the test). This is consistent with the language available in the Axon manual, which suggests that the criteria used to establish profile validity are aimed at detecting low effort in addition to factors that would have more obvious/catastrophic influence on performance. The ImPACT manual is less clear with regard to the sources of invalidity targeted by its validity criteria, but its list of common sources of invalid performance (failure to read directions, learning or attentional disorders, excessive fatigue, horseplay, and left-right confusionⁱⁱ)²² might suggest that its criteria are aimed at identifying more obvious sources of poor performance and could explain the lower rate of invalid baselines observed in our sample. However, we also cannot rule out the possibility that our participants’ effort was truly better on this CNT versus the others. Further, prior work suggests that intentional underperforming (“sandbagging”) on ImPACT is relatively difficult to accomplish without detection.^17,34

Most invalid baselines were flagged as such because of failure on only 1 of several validity criteria, supporting the idea that athletes do not broadly sandbag their evaluations. Also consistent with this idea was that failure on the MSVT was quite rare (1.8% of the sample), and across the 3 tests containing validity measures taken by each athlete measures (2 CNTs and the MSVT), only 1 athlete (0.05%) produced invalid profiles on all 3 tests. Instead of sandbagging, some athletes may (1) display continual, mildly suboptimal performance (picked up only by the more stringent criteria) or (2) put forth suboptimal performance (including low effort or unintentional loss of focus) selectively. In the current sample, each CNT had a subset of validity criteria that was systematically more likely to be failed than others. Our review of these data suggests that the more difficult tests tended to be flagged more often and is consistent with both possibilities (1) and (2) above in that the criteria (eg, ANAM MTH, M2S, and CDD; Axon LN, and ImPACT DML and TL) for these tests may be more sensitive to low effort (alternatively, test takers may be more likely to be overwhelmed by and give up on these more challenging or confusing tests). That the failure rate of Axon’s Integrity Check for LN accuracy was substantially higher than that for WM accuracy (despite both criteria only requiring accuracy over 53%), for example, could suggest that some athletes find the LN task too difficult, confusing, or overwhelming to put forth adequate effort and meet this minimal performance threshold. This finding underscores the importance of providing encouragement to examinees to do their best even on challenging tasks and illustrates the notion that the degree to which test scores are valid may vary from task to task (or moment to moment) even within a single CNT administration.

Finally, our evaluation of the predictors of invalid baseline performance revealed some important insights. First, the validity indicators may not be appropriate or equally meaningful for some populations (eg, those with ADHD, especially for ANAM and Axon in which a large minority of participants with ADHD produced invalid profiles). We cannot rule out that these participants failed validity criteria at a higher rate because of lower motivation/effort. One way to tease this out may be to see how frequently participants with ADHD (who failed their first baseline test) produce valid profiles, given another testing opportunity. As most athletes with ADHD who produced invalid baselines did not complete repeat baseline evaluations, we cannot comment on the effect that this may have had. If the ANAM and Axon validity criteria are indeed more stringent (sensitive), then detecting unintended factors (ie, bona fide cognitive impairment) would be an expected natural consequence of more stringent criteria for validity.

Although we did not record data on testing group size to a degree needed for analysis of this variable, given prior findings on the relevance of this variable in CNT validity,^23,30 it would be valuable for future studies to record this and explore interactions among testing conditions (eg, group size) and other individual difference variables (eg, ADHD) to clarify the athletes who may be more or less vulnerable to the impact of such testing conditions. Until then, it would be wise to follow standard recommendations to administer baseline tests individually or in very small, distraction-free groups, especially for those athletes who are most at risk of producing invalid profiles. Of course, given the exploratory nature of our analyses, it will also be important to replicate our findings on new samples and to explore the extent to which procedures used in this study (eg, monetary incentive for participation) influence athlete motivation and baseline test performance. Given the poor agreement between validity status for the CNTs and MSVT and possibility that the validity criteria of these tests tap different aspects of ability and effort, it would also be valuable to perform more systematic manipulation of examinee instructions and other testing factors to better identify what is being tapped by the various validity criteria applied across these tests as a means to further refine these measures and facilitate the interpretation of validity output.

Our results demonstrate that the vast majority of athletes are capable of producing valid baseline profiles, given the proper testing conditions. This is consistent with other published reports, including prior work finding that most athletes (87%) who produce invalid baselines obtain valid profiles given a reassessment.³⁵ Similarly, although we did not require select athletes (in particular those with ADHD, a learning disability, and/or low WTAR scores) to repeat invalid baselines, 73.5% of the athletes in our sample who did repeat testing after invalid baselines achieved valid profiles on the second attempt (with roughly equal percentages across ANAM, Axon, and ImPACT: 75.4%, 71.7%, and 73.3%, respectively). Overall, our rate of invalid test performance was low relative to some published estimates, especially when considering only the first test administrations (which would better match a typical athlete evaluation). Given the published literature on this topic and common assumptions in neuropsychological practice, we might presume that several procedural factors facilitated this, including (1) individually proctored tests,^23,30 (2) scripted test instructions emphasizing the importance of good effort, (3) controlled testing conditions (ie, tests administered in quiet, supervised settings), and (4) payment for study participation. Our data also provide evidence that performance is optimized when (5) athletes are not cognitively fatigued. While we recognize that financial payment is an incentive unique to the experimental setting, because it was a constant across all examinations, we believe that this component of our protocol did not negate our ability to draw comparisons between the CNTs. These findings reinforce, as others have articulated,^2,29 that preseason evaluations should be administered under conditions that will maximize athletes’ interest in and performance on baseline tests, which could feasibly include all factors listed above (with the exception of (4)). Although incorporating some of these factors (eg, individually proctored tests) into routine practice would negate some of the purported advantages of CNTs, it remains important for test users to be aware that decisions around how baseline tests are administered has important implications for their potential value in later postinjury assessments.

Footnotes

Acknowledgements

The authors thank Kwang Woo Ahn, PhD, for his input on the statistical analyses.

The opinions, interpretations, conclusions, and recommendations in this article are those of the authors and are not necessarily endorsed by the US Army. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.

One or more of the authors has declared the following potential conflict of interest or source of funding: This work was supported by the US Army Medical Research and Materiel Command under award number W81XWH-12-1-0004. This publication was also supported by the Clinical and Translational Science Institute grant 1UL1-RR031973 (-01) and by the National Center for Advancing Translational Sciences (National Institutes of Health) grant 8UL1TR000055.

Notes

References

Aubry

Cantu

Dvorak

. Summary and agreement statement of the 1st International Symposium on Concussion in Sport, Vienna 2001. Clin J Sport Med. 2002;12(1):6-11.

Bauer

Iverson

Cernich

Binder

Ruff

Naugle

. Computerized neuropsychological assessment devices: joint position paper of the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology. Arch Clin Neuropsychol. 2012;27(3):362-373.

Benjamini

Hochberg

. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B Stat Methodol. 1995;57:289-300.

Benjamini

Yekutieli

. The control of the false discovery rate in multiple testing under dependency. Ann Stat. 2001;29:1165-1188.

Bleiberg

Cernich

Cameron

. Duration of cognitive impairment after sports concussion. Neurosurgery. 2004;54(5):1073-1078.

Bleiberg

Kane

Reeves

Garmoe

Halpern

. Factor analysis of computerized and traditional tests used in mild brain injury research. Clin Neuropsychol. 2000;14(3):287-294.

Broglio

Ferrara

Macciocchi

Baumgartner

Elliott

. Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007;42(4):509-514.

Broglio

Macciocchi

Ferrara

. Sensitivity of the concussion assessment battery. Neurosurgery. 2007;60(6):1050-1057.

Brooks

Mrazik

Barlow

McKay

Meeuwisse

Emery

. Absence of differences between male and female adolescents with prior sport concussion. J Head Trauma Rehabil. 2014;29(3):257-264.

10.

Cole

Arrieux

Schwab

Ivins

Qashu

Lewis

. Test-retest reliability of four computerized neurocognitive assessment tools in an active duty military population. Arch Clin Neuropsychol. 2013;28(7):732-742.

11.

Collie

Darby

Maruff

. Computerised cognitive assessment of athletes with sports related head injury. Br J Sports Med. 2001;35(5):297-302.

12.

Covassin

Elbin

3rd Stiller-Ostrowski

Kontos

. Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) practices of sports medicine professionals. J Athl Train. 2009;44(6):639-644.

13.

Derogatis

. Brief Symptom Inventory 18 (BSI-18): Administration, Scoring, and Procedures Manual. Bloomington, Minnesota: Pearson; 2001.

14.

Diener

Emmons

Larsen

Griffin

. The Satisfaction With Life Scale. J Pers Assess. 1985;49(1):71-75.

15.

Echemendia

Bruce

Bailey

Sanders

Arnett

Vargas

. The utility of post-concussion neuropsychological data in identifying cognitive change following sports-related MTBI in the absence of baseline data. Clin Neuropsychol. 2012;26(7):1077-1091.

16.

Eckner

Kutcher

Richardson

. Between-seasons test-retest reliability of clinically measured reaction time in National Collegiate Athletic Association Division I athletes. J Athl Train. 2011;46(4):409-414.

17.

Erdal

. Neuropsychological testing for sports-related concussion: how athletes can sandbag their baseline testing without detection. Arch Clin Neuropsychol. 2012;27(5):473-479.

18.

Erlanger

Feldman

Kutner

. Development and validation of a web-based neuropsychological test protocol for sports-related return-to-play decision-making. Arch Clin Neuropsychol. 2003;18(3):293-316.

19.

Green’s Medical Symptom Validity Test for Windows [computer program]. Edmonton, Alberta, Canada: Green’s Publishing Inc; 2003.

20.

Guskiewicz

. Postural stability assessment following concussion: one piece of the puzzle. Clin J Sport Med. 2001;11(3):182-189.

21.

Hunt

Ferrara

Miller

Macciocchi

. The effect of effort on baseline neuropsychological test scores in high school football athletes. Arch Clin Neuropsychol. 2007;22(5):615-621.

22.

ImPACT Applications Inc. Technical manual: online ImPACT 2007-2012. Available at: https://www.impacttest.com/pdf/ImPACTTechnicalManual.pdf. Accessed February 17, 2015.

23.

Lichtenstein

Moser

Schatz

. Age and test setting affect the prevalence of invalid baseline scores on neurocognitive tests. Am J Sports Med. 2014;42(2):479-484.

24.

MacDonald

Duerson

. Reliability of a computerized neurocognitive test in baseline concussion testing of high school athletes [published online July 24, 2014]. Clin J Sport Med. doi:10.1097/JSM.0000000000000139.

25.

Maruff

Thomas

Cysique

. Validity of the CogState brief battery: relationship to standardized tests and sensitivity to cognitive impairment in mild traumatic brain injury, schizophrenia, and AIDS dementia complex. Arch Clin Neuropsychol. 2009;24(2):165-178.

26.

McCrea

Kelly

Kluge

Ackley

Randolph

. Standardized assessment of concussion in football players. Neurology. 1997;48(3):586-588.

27.

McCrory

Meeuwisse

Aubry

. Consensus statement on concussion in sport: the 4th International Conference on Concussion in Sport held in Zurich, November 2012. Clin J Sport Med. 2013;23(2):89-117.

28.

Moriarity

Pietrzak

Kutcher

Clausen

McAward

Darby

. Unrecognised ringside concussive injury in amateur boxers. Br J Sports Med. 2012;46(14):1011-1015.

29.

Moser

Schatz

Lichtenstein

. The importance of proper administration and interpretation of neuropsychological baseline and postconcussion computerized testing. Appl Neuropsychol Child. 2015;4:41-48.

30.

Moser

Schatz

Neidzwski

Ott

. Group versus individual administration affects baseline neurocognitive test performance. Am J Sports Med. 2011;39(11):2325-2330.

31.

Rahman-Filipiak

AAM

Woodward

. Administration and environment considerations in computer-based sports-concussion assessment. Neuropsychol Rev. 2014;23:314-334.

32.

Roebuck-Spencer

Vincent

Gilliland

Johnson

Cooper

. Initial clinical validation of an embedded performance validity measure within the Automated Neuropsychological Metrics (ANAM). Arch Clin Neuropsychol. 2013;28(7):700-710.

33.

Schatz

. Long-term test-retest reliability of baseline cognitive assessments using ImPACT. Am J Sports Med. 2010;38(1):47-53.

34.

Schatz

Glatts

. “Sandbagging” baseline test performance on ImPACT, without detection, is more difficult than it appears. Arch Clin Neuropsychol. 2013;28(3):236-244.

35.

Schatz

Kelley

Ott

. Utility of repeated assessment after invalid baseline neurocognitive test performance. J Athl Train. 2014;49(5):659-664.

36.

Schatz

Moser

Solomon

Ott

Karpf

. Prevalence of invalid computerized baseline neurocognitive test results in high school and collegiate athletes. J Athl Train. 2012;47(3):289-296.

37.

Schatz

Pardini

Lovell

Collins

Podell

. Sensitivity and specificity of the ImPACT test battery for concussion in athletes. Arch Clin Neuropsychol. 2006;21(1):91-99.

38.

Schatz

Robertshaw

. Comparing post-concussive neurocognitive test data to normative data presents risks for under-classifying “above average” athletes. Arch Clin Neuropsychol. 2014;29(7):625-632.

39.

Schmidt

Mihalik

Kerr

Guskiewicz

. Identifying impairments after concussion: normative data versus individualized baselines. Med Sci Sports Exerc. 2012;44(9):1621-1628.

40.

Solomon

Haase

. Biopsychosocial characteristics and neurocognitive test performance in National Football League players: an initial assessment. Arch Clin Neuropsychol. 2008;23:563-577.

41.

Solomon

Haase

Kuhn

. The relationship among neurocognitive performances and biopsychosocial characteristics of elite National Football League draft picks: an exploratory investigation. Arch Clin Neuropsychol. 2013;28:9-20.

42.

Straume-Naesheim

Andersen

Bahr

. Reproducibility of computer based neuropsychological testing among Norwegian elite football players. Br J Sports Med. 2005;39 Suppl 1:i64-i69.

43.

Szabo

Alosco

Fedor

Gunstad

. Invalid performance and the ImPACT in National Collegiate Athletic Association Division I football players. J Athl Train. 2013;48(6):851-855.

44.

Wechsler

. Wechsler Test of Adult Reading: WTAR. San Antonio, Texas: The Psychological Corp; 2001.

45.

Zuckerman

Lee

Odom

Solomon

Sills

. Baseline neurocognitive scores in athletes with attention deficit-spectrum disorders and/or learning disability. J Neurosurg Pediatrics. 2013;12:103-109.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.14 MB