Test-Retest Reliability of Immersive Virtual Reality Measures of Perceptual-Motor Performance

Abstract

The duration, accuracy, and consistency of responses to various types of stimuli are widely accepted as indirect indicators of the efficiency of brain information processing, but current clinical tests appear to lack sufficient sensitivity to detect subtle impairments. Immersive virtual reality (VR) offers a new means to acquire measures of perceptual-motor responses to moving visual stimuli that require rapid conflict resolution, but their test-retest reliability has not yet been demonstrated. Repeated measures. We analyzed data from 19 healthy young adults who performed a 40-trial VR test on three consecutive days. We focused on response time (RT) and perceptual latency (PL) for eye, neck, arm, and whole-body step displacements involved in executing a reaching/lunging movement in a right or left direction toward a peripherally located virtual target. Measures of RT and PL included a 40-trial mean, an intra-individual variability (IIV) value, and a rate correct score (RCS) that incorporated both response duration and accuracy. Most mean and IIV values for PL and RT demonstrated a positive distributional skew that was substantially reduced by natural logarithm transformation. While a learning effect was evident between sessions 1 and 2 for 7 of 8 mean PL and RT measures, 3-session intraclass correlation coefficient (ICC) values were moderate to excellent for 15 of 16 transformed PL and RT measures (range: .618 to .922). The composite RCS metric did not require transformation for either PL or RT, whose respective 3-session ICC values were .877 and .851. This moderate to excellent test-retest reliability for various VR measures of perceptual-motor function, combined with evidence of their validity from both past and future research, suggest that these measures can advance clinical detection of impaired brain processing and longitudinal assessments of potentially modifiable performance deficiencies.

Keywords

choice reaction time cognition visual perception response speed motor skills neural efficiency intra-individual variability performance assessment

Introduction

Virtual reality (VR) has been defined as a computer-generated simulation of an environment that enables user interactions with its elements (Santos et al., 2020). Non-immersive VR methods project a simulated environment onto a screen or computer monitor that is viewed within the surrounding real environment, whereas immersive VR methods project the user into a simulated three-dimensional environment through a head-mounted display that completely removes visual awareness of the real environment (Müller et al., 2023; Santos et al., 2020). A growing awareness of the limitations of existing clinical tests for detecting subtle performance deficiencies among athletes has created a critical need for new clinical measures that will identify those with elevated injury risk (Crasta et al., 2023; Wong et al., 2023). Compared to standard clinical tests of neurocognitive and neuromuscular function, VR can provide greater control of visual stimuli (e.g., location, duration, speed of movement and closer coupling between perceptual and motor responses) (Harris et al., 2020; Müller et al., 2023).

A variety of clinical test protocols have been devised to impose various combinations of visual, cognitive, or motor demands, and these are believed to reveal brain processing deficiencies that are not evident when component tasks are performed in isolation (Slobounov et al., 2006b). Cognitive demand has often been imposed by such activities as reciting months of the year backwards, reciting results of a series of simple mathematical operations, or giving verbal responses to word-color combinations (i.e., Stroop task); and the complexity of motor responses to visual stimuli has ranged from simple button presses to multi-directional whole-body movement patterns (Harper et al., 2022; Lynall et al., 2021). A possible explanation for the limited sensitivity of many clinical tests used for detecting subtle impairment is that they lack an assessment of integrated visual-cognitive-motor challenges that require simultaneous activation of multiple specific brain circuits frequently employed during real-world functioning (Kiefer et al., 2018; McGowan et al., 2018).

Early studies of the expanded potential of VR testing for revealing impaired sensorimotor function utilized a non-immersive projection system to display a virtual “moving room” that induced postural sway, which was measured by a force plate (Slobounov et al., 2006a, 2006b, 2007). The same non-immersive VR projection system was later combined with a head-mounted motion tracking system to compare how visual motion affected college athletes who had sustained a concussion within the preceding 7–10 days versus college students with no history of concussion (Teel et al., 2016a, 2016b; Teel & Slobounov, 2015). In the first study to use an immersive VR measure with a head-mounted display, a simulated classroom environment with auditory and visual distractions elicited deficits in attention and inhibitory control among adolescent athletes who had sustained a concussion within the preceding 2-year period (Nolin et al., 2012). The collective results of these early studies suggested that VR might provide a more sensitive means of identifying a brain processing impairment than standard clinical tests (Santos et al., 2020).

Recent findings from immersive VR studies suggest that the use of multiple visually guided exercises may reduce sport-related injury risk (Reneker et al., 2019) and improve various performance capabilities (Reneker et al., 2020). Similarly, simulating a sport-specific VR environment and permitting user interactions with other players and objects has revealed more high-risk hip, knee, and ankle displacements than clinical tests administered without VR immersion (DiCesare et al., 2020). Although maximally realistic simulated environments (i.e., ecological validity) may best promote naturalistic responses of the musculoskeletal system (Schuermans et al., 2022), a virtual environment that looks and feels real may not necessarily present perceptual, cognitive, and motor challenges in a manner that permits measurement of integrated responses (Harris et al., 2020). For example, an immersive VR application designed to give the user the realistic experience of riding a racehorse quantified the rider’s response times through an unrelated secondary task involving pressing a button in response to superimposed geometric shapes of different colors (Horan et al., 2020).

Whether designated as “reaction time” or “response time” (RT), the interval between a stimulus onset and initiation or completion of some specified response provides a widely accepted representation of the speed of brain information processing (Harper et al., 2022; Marmolejo-Ramos et al., 2015). Overlapping stages of neural processing include stimulus detection (i.e., perceptual awareness), stimulus interpretation and response selection (i.e., cognitive processing), and response execution (i.e., motor activation and control). Despite overlap between the cognitive and motor stages of RT, subdivision of the elapsed time is needed to estimate the durations of component processes that may yield different information (Pfister et al., 2023). The terms “decision time” or “perceptual latency” (PL) can be used to refer to the duration of these various delays between stimulus appearance and initiation of some detectable response, in which case the term “movement time” refers to the additional amount of time required to complete a specified action (Morral-Yepes et al., 2022). Because some amount of cognitive control guides a motor response beyond its initiation, including rapid ballistic movements (Pfister et al., 2023), assessment of only PL may provide an incomplete representation of integrated cognitive and motor processing.

The RT measure derived from a given testing paradigm is often the outcome of primary interest, but the overall accuracy of responses may be an equally important consideration for tests that present a cognitive conflict resolution challenge requiring response inhibition during selection from more than one response option (Goswami et al., 2016). The “rate correct score” (RCS) metric derived from division of the number of correct responses by the sum of response times for a set of trials has been shown to provide an advantageous composite indicator of both speed and accuracy (Vandierendonck, 2017). Another potentially important behavioral indicator of neural processing efficiency is intra-individual variability (IIV) of RTs, which yields information that is distinct from mean RT. Numerous studies have documented an inverse relationship between behavioral performance consistency and moment-to-moment brain signal variability (Fjell et al., 2011; Garrett et al., 2014, 2020; Grady & Garrett, 2018; Johnson et al., 2015; Kelly et al., 2008; McIntosh et al., 2008; Mennes et al., 2011; Roberts et al., 2022; Wang et al., 2017). Furthermore, elevated IIV has been associated with a history of sport-related concussion (McGowan et al., 2018, 2019; Merritt et al., 2019, 2021; Parks et al., 2015; Rabinowitz & Arnett, 2013; Wong et al., 2023). Despite growing evidence that longitudinal assessment of RT-related measures (e.g., RCS and IIV) may serve as important indicators of subtle change in brain processing efficiency, assessment of distribution normality and test-retest reliability have received relatively little attention.

The development of new measures for clinical use requires knowing an estimated magnitude of change over short retest intervals for healthy individuals to properly distinguish this change from change associated with positive gains in performances or declines due to an exacerbated impairment (Horan et al., 2020). In this study, we presented a VR test in which there were moving visual stimuli of two types that required respondents to select stimulus-matched directional responses over multiple successive trials, and to execute of a combination of eye movements, neck rotation, upper extremity reaching, and whole-body lateral lunging toward a virtual target; we then measured the duration of each displacement. Previous research has demonstrated that measures derived from this specific VR test have good predictive validity for the identification of college athletes and military cadets with a remote history of concussion (Wilkerson et al., 2023). Based on the findings reported by the only previous study that has assessed the test-retest reliability of RT values derived from an immersive VR test (Horan et al., 2020), we expected adequate statistical results to justify the clinical use of most, if not all, measures derived from this unique VR test. Thus, our purpose in this study was to assess the test-retest reliability of measures derived from a unique VR system that quantified different aspects of complex perceptual-motor performance in healthy young adults. We hoped this data would permit determinations of the suitability of this type of VR testing for clinical purposes.

Method

Participants

A convenience sample of 24 healthy graduate students voluntarily completed immersive VR assessments conducted on three consecutive days. We obtained Institutional Review Board approval of the study protocol and informed consent from all participants prior to their participation. Potential participants were excluded if they reported any injury-related limitations in physical functioning. Although we did not complete an a priori power analysis, prior studies of test-retest reliability involving dual-task responses to visual stimuli suggested that 20 participants would be a sufficient sample size (Lempke et al., 2022; Lynall et al., 2021).

Instrumentation and Procedures

Prior to initiation of the VR test, virtual response targets (i.e., green spheres) could be seen against the black background of the head-mounted display (Pico Neo 3 Pro Eye, Pico Immersive, Ltd., Mountain View, CA) (Figure 1

Figure 1.

View of Virtual Reality Environment Prior to Relocation of Spherical Response Targets to a Position Beyond the Peripheral Limits of the User’s Field of View, Thereby Requiring Neck Rotation for Visualization. (Figure reproduced with permission from Wilkerson, G. B., et al. (2023). Subtle impairments of perceptual-motor function and wellbeing are detectable among military cadets and college athletes with self-reported history of concussion. Frontiers in Sports and Active Living, 5, 1,046,572.).

). After electronic acquisition of the maximum horizontal distance between hand controllers with both arms elevated to 90° in an upright standing position (i.e., T-pose position), the response targets were relocated to a position 30% beyond the maximum arm reach and outside the limit of peripheral vision. The VR test presented two different visual stimuli – either a “solid” white circle or an “open” white ring, which moved horizontally in a right-to-left or a left-to-right direction for 500 ms across the display. A correct response involved a reaching/lunging movement to make hand controller contact with the virtual response target located in the same direction as the horizontal movement of a solid white circle (i.e., a correct congruent response) or contact with the response target located in the direction opposite to that of an open white ring (i.e., a correct incongruent response). The visual stimuli initially appeared in either a central or peripheral location, with 60° of movement across either the right or left visual field, thereby presenting one of eight different combinations of stimulus type, initial location, and movement direction in random order (Figure 2). A total of 40 successive trials were completed, each requiring eye tracking of the moving visual stimuli, neck rotation to visualize the virtual response targets on either side, and both a lunge step and an arm reach in the same direction to cover a sufficient horizontal distance to make hand controller contact with a virtual target (Figure 3).

Figure 2.

Depictions of 8 Combinations of Moving Visual Stimulus Type (Incongruent Open Ring vs. Congruent Filled Circle), Initial Location (Central vs. Peripheral), and Movement Direction (Left-to-Right vs. Right-to-Left) that were Superimposed on the Virtual Reality Environment. (Figure reproduced with permission from Wilkerson, G. B., et al. (2023). Subtle impairments of perceptual-motor function and wellbeing are detectable among military cadets and college athletes with self-reported history of concussion. Frontiers in Sports and Active Living, 5, 1,046,572.).

Figure 3.

Reaching/Lunging Movement Required to Make Hand Controller Contact with Virtual Target.

Immediately prior to the start of the test, participants assumed a slightly crouched posture with the hand controllers held close to the chest midline, and with their eyes focused on a central fixation cross that disappeared when a moving stimulus appeared. The direction, speed, and displacement distance of eye responses were derived from an eye tracking camera. Corresponding neck, arm, and step responses were measured at a 60-Hz sampling frequency by inertial sensors and an inside-out ultrasonic tracking system within the head-mounted component of the VR system. When a virtual target was contacted by a hand controller, a simultaneous auditory tone and controller vibration served as a cue to return to the initial position and prepare for another trial. The intervals between successive trials were 2, 2.5, or 3 s, which prevented anticipation of the timing of the appearance of the next moving stimulus. A standardized visual and auditory orientation that included congruent and incongruent practice trials was provided prior to the first of 3 test sessions conducted on consecutive days, which each consisted of one 40-trial test.

Perceptual latency was defined as the interval from stimulus appearance to movement initiation, which corresponded to a threshold of 6° for angular displacement of the eyes and neck and a threshold of 10 cm of linear displacement of the hand controller (arm) and the whole-body lunge step (defined by the position of the head). The displacement thresholds were chosen to maximize the number of intentional responses to visual stimuli, while minimizing the number of false responses (i.e., unintentional oscillatory movements preceding intentional responses). Response time (RT) was defined as the interval from stimulus appearance to maximum displacement measured prior to reversal of movement direction in preparation for another trial (Figure 4)

Figure 4.

Operational Definitions of Response Time (RT) and its Perceptual Latency (PL) Subcomponent.

. Mean PL and RT were obtained for 40 trials of the eye, neck, arm, and step movements. The standard deviation of an individual’s 40 trials for PL and RT of each movement type was used to represent IIV. Rate correct per second (RCS) is a composite performance metric representing the response speed-accuracy trade-off, which was calculated as the number of correct responses divided by the sum of either PL durations or RT durations (Vandierendonck, 2017).

Statistical Analysis

Cases missing data for any of the three test sessions were excluded from the analysis, as were those that failed to demonstrate ≥75% response accuracy for all sessions. We assessed the normality of the data distribution by the Shapiro-Wilk test for each measure acquired during each test session. We used natural log(Log_e) transformation to improve the normality of each distribution that demonstrated positive skew and a statistically significant (p < .05) Shapiro-Wilk test result (Dunlap et al., 1994; Olivier & Norberg, 2010). We used repeated measures analysis of variance (ANOVA) to assess statistically significant differences (p < .05) among the 3 sessions, along with follow-up pairwise comparisons. We assessed test-retest reliability through calculation of the intraclass correlation coefficient (ICC) with the two-way random effects, absolute agreement, and average of measures model (2, k) to include variance associated with systematic error among sessions. (Koo & Li, 2016; Weir, 2005). Raw data were used for measures that exhibited distribution normality, whereas Log_e transformed data were used for measures that demonstrated improvement of distribution normality for ≥2 of the 3 test sessions. Qualitative interpretations of the ICC values were <.50: poor, .50 to .74: moderate, .75 to .89: good, and ≥.90: excellent (Koo & Li, 2016). We used an ICC value ≥.60 as a standard for determining that a measure had adequate reliability for clinical use (Broglio et al., 2007). We calculated the standard error of measurement (SEM) and minimal detectable change at a 95% confidence level (MDC₉₅) for measures that did not require transformation to improve distribution normality. All analyses were performed with IBM SPSS version 29.0 (IBM Corporation, Armonk, NY).

Results

Electronic data acquisition failure (3 cases) and invalid test data (2 cases with <75% response accuracy) reduced the initial sample size from 24 to 19 participants (9 males: M_age = 24, SD = 1.4 years; M_height = 177.2, SD = 5.0 cm; M_weight = 80.0, SD = 14.2 kg; and 10 females: M_age = 23, SD = 1.6 years; M_height = 163.6, SD = 9.1 cm; M_weight = 74.2, SD = 20.4 kg). The vast majority, 89% (17/19), had participated in a competitive team sport at the high school or college level. Neither the PL nor RT composite RCS metrics required Log_e transformation. Among the 40-trial mean values for PL and RT of eye, neck, arm, and step movements, transformation improved the normality of the data distribution for 21 of the 24 measures (Supplemental Material: Table S-1). Among the 40-trial IIV values, transformation improved distribution normality for 22 of 24 PL and RT measures (Supplemental Material: Table S-2). Statistical analyses for transformed measures utilized Log_e data, but back-transformation of mean values was considered essential for proper interpretation of results (Lee, 2020). The back-transformed mean value for Log_e is referred to as the Geometric Mean (Olivier & Norberg, 2010), which provides an estimate of the median value for the original data (Tables 1 and 2). A difference in Log_e units multiplied by 100 can be viewed as a symmetric mean percentage difference (%Diff), and a coefficient of variation in percentage units (%CV) can be obtained by multiplication of the Log_e SD by 100 (Cole & Altman, 2017).

Table 1.

Geometric Mean Values (Natural Log Transformation of Original Values) and Intraclass Correlation Coefficients.

	Session 1		Session 2		Session 3		Sessions 1-2–3
Metric	G Mean	(Log_e)	G Mean	(Log_e)	G Mean	(Log_e)	ICC (2,k)	p _Diff
Eye perceptual latency	.558	(−.584)	.563	(−.575)	.593	(−.523)	.903	.150
Neck perceptual latency	.655	(−.423)	.629	(−.463)	.616	(−.484)	.922	.130
Arm perceptual latency	.739	(−.302)	.680	(−.385)	.663	(−.411)	.884	<.001
Step perceptual latency	.760	(−.275)	.708	(−.346)	.699	(−.358)	.907	.004
Eye response time	.826	(−.191)	.964	(−.037)	1.038	(−.038)	.618	.052
Neck response time	1.068	(.066)	1.017	(.017)	.995	(−.005)	.904	.023
Arm response time	1.261	(.232)	1.201	(.183)	1.151	(.141)	.837	.001
Step response time	1.307	(.267)	1.247	(.221)	1.216	(.196)	.882	.054

Note. G Mean, geometric mean (estimated median of original data [ back-transformation of Log_e value]).

ICC (2,k), two-way random effects, absolute agreement, average of measures intraclass correlation coefficient.

p_Diff: p-value for repeated measures analysis of variance difference among sessions.

Table 2.

Geometric Means of Intra-Individual Variability (IIV) Values (Natural Log Transformation of Original Values) and Intraclass Correlation Coefficients.

	Session 1		Session 2		Session 3		Sessions 1-2–3
Metric	G Mean	(Log_e)	G Mean	(Log_e)	G Mean	(Log_e)	ICC (2,k)	p _Diff
Eye perceptual latency IIV	.343	(−1.069)	.302	(−1.198)	.325	(−1.125)	.754	.302
Neck perceptual latency IIV	.196	(−1.632)	.183	(−1.698)	.178	(−1.732)	.836	.560
Arm perceptual latency IIV	.204	(−1.587)	.152	(−1.887)	.153	(−1.877)	.763	.101
Step perceptual latency IIV	.209	(−1.565)	.173	(−1.752)	.175	(−1.746)	.724	.200
Eye response time IIV	.569	(−.565)	.636	(−.453)	.626	(−.469)	.468	.563
Neck response time IIV	.228	(−1.477)	.183	(−1.696)	.185	(−1.686)	.796	.033
Arm response time IIV	.198	(−1.617)	.166	(−1.797)	.155	(−1.862)	.701	.023
Step response time IIV	.202	(−1.601)	.170	(−1.770)	.173	(−1.756)	.693	.233

Note. G Mean, geometric mean (estimated intra-individual variability median of original data [back-transformation of Log_e value]).

ICC (2,k), Intraclass correlation coefficient, two-way random effects, absolute agreement, average of measures.

p_Diff: p-value for repeated measures analysis of variance difference among sessions.

We observed moderate to excellent 3-session ICC values for 15 of 16 Log_e transformed mean and IIV values for RT and PL measures (range: .618 to .922). Assessment of possible learning effects required separate analyses for consecutive pairs of trials. Transformed PL and RT mean values demonstrated substantially better agreement between sessions 2–3 compared to sessions 1–2 for 7 of 8 measures (Table3). We observed good to excellent 2-session IIC values for 14 of 16 mean measures of PL and RT (range: .790 to .919). Transformed IIV values for PL and RT demonstrated better agreement between sessions 1–2 than sessions 2–3 for 6 of 8 measures, which suggests a lack of learning effect (Table 4). Moderate to good 2-session IIV values were observed for 13 of 16 PL and RT measures of IIV (range: .538 to .883). Because the RCS composite metrics did not need Log_e transformation, SEM and MDC₉₅ values could be calculated across the 3 sessions (Table 5) and for pairwise session comparisons (Table 6). A sessions 1-2 RCS learning effect was evident from the results of repeated measures analysis of variance for both PL and RT, but ICC values demonstrated good sessions 1-2 RCS agreement for both PL (.837) and RT (.805). Excellent ICC values were observed for sessions 2–3 agreement for both PL (.925) and RT (.900).

Table 3.

Pairwise Comparisons of Natural Log Transformations of Original Mean Values.

	Session 1 – Session 2					Session 2 – Session 3
Metric	ICC (2,k)	p _Diff	%Diff	(95% CI)	%CV	ICC (2,k)	p _Diff	%Diff	(95% CI)	%CV
Eye perceptual latency	.886	.776	−0.8	(−6.6, 5.3)	13.2	.867	.125	−5.4	(−11.3, 1.6)	15.1
Neck perceptual latency	.919	.142	4.1	(−1.5, 10.0)	12.1	.934	.330	2.1	(−2.3, 6.5)	9.3
Arm perceptual latency	.856	.001	8.7	(3.9, 13.6)	9.7	.937	.093	2.7	(−.5, 5.9)	6.6
Step perceptual latency	.909	<.001	7.4	(3.6, 11.4)	7.9	.925	.525	1.1	(−2.5, 4.9)	7.9
Eye response time	.367	.063	−14.3	(−27.2, .9)	40.3	.591	.988	0.1	(−12.3, 14.2)	31.5
Neck response time	.886	.059	5.0	(−.2, 10.5)	11.1	.908	.247	2.2	(−1.6, 6.2)	8.3
Arm response time	.804	.084	5.0	(−.7, 11.1)	12.4	.887	.016	4.3	(.9, 7.9)	7.2
Step response time	.790	.114	4.8	(−1.2, 11.1)	13.0	.869	.186	2.5	(−1.3, 6.6)	8.3

Note. ICC (2,k), intraclass correlation coefficient, two-way random effects, absolute agreement, average of measures.

p_diff: p-value for paired samples follow-up t-test for difference between sessions.

%Diff, symmetric percentage difference between sessions (Log_e difference X 100).

%CV, percentage coefficient of variation between sessions (Log_e SD X 100).

Table 4.

Pairwise Comparisons of Natural Log Transformations of Original Intra-individual Variability (IIV) Values.

	Session 1 – Session 2					Session 2 – Session 3
Metric	ICC (2,k)	p _Diff	%Diff	(95% CI)	%CV	ICC (2,k)	p _Diff	%Diff	(95% CI)	%CV
Eye perceptual latency IIV	.883	.006	13.8	(4.4, 24.0)	19.6	.538	.462	−7.0	(−24,2, 14.0)	52.7
Neck perceptual latency IIV	.881	.353	6.8	(−7.6, 23.5)	35.3	.744	.749	3.5	(−17.1, 29.1)	58.3
Arm perceptual latency IIV	.736	.017	34.9	(6.3, 13.6)	63.9	.733	.935	−1.0	(−22.5, 26.6)	66.4
Step perceptual latency IIV	.744	.046	20.7	(.4, 45.1)	46.6	.747	.949	−0.7	(−20.4, 23.9)	58.2
Eye response time IIV	.347	.399	−10.6	(−31.9, 17.4)	75.8	.389	.861	1.6	(−16.2, 23.3)	49.3
Neck response time IIV	.792	.009	24.5	(6.4, 45.6)	38.5	.706	.919	−1.0	(−19.5, 21.7)	53.6
Arm response time IIV	.781	.020	19.7	(3.2, 38.8)	35.9	.493	.501	6.7	(−12.5, 30.0)	50.8
Step response time IIV	.819	.015	18.3	(3.7, 35.0)	31.4	.579	.909	−1.3	(−22.6, 25.7)	65.3

Note. ICC (2,k), intraclass correlation coefficient, two-way random effects, absolute agreement, average of measures.

p_Diff: p-value for paired samples follow-up t-test for difference between sessions.

%Diff, symmetric percentage difference between sessions (Log_e difference X 100).

%CV, percentage coefficient of variation between sessions (Log_e SD X 100).

Table 5.

Rate Correct Score Mean Values, Intraclass Correlations, and Minimal Detectable Change Values.

	Session 1	Session 2	Session 3	Sessions 1-2–3
Composite metric	Mean ± SD	Mean ± SD	Mean ± SD	ICC (2,k)	p _Diff	SD_p	SEM	MDC₉₅
Rate correct score – perceptual latency	1.28 ± .26	1.44 ± .24	1.46 ± .23	.887	<.001	.253	.085	.289
Rate correct score – response time	.75 ± .13	.82 ± .11	.84 ± .10	.851	<.001	.118	.046	.155

Note. ICC (2,k), intraclass correlation coefficient, two-way random effects, absolute agreement, average of measures.

p_Diff: p-value for repeated measures analysis of variance difference among sessions.

SD_p, pooled standard deviation.

SEM, standard error of measurement.

MDC₉₅, minimal detectable change at 95% level of confidence.

Table 6.

Pairwise Session Comparisons of Rate Correct Score Mean Values, Intraclass Correlation Coefficients, and Minimal Detectable Change Values.

	Sessions 1–2					Sessions 2–3
Composite metric	ICC (2,k)	P_Diff	s_p	SEM	MDC₉₅	ICC (2,k)	p _Diff	SD_p	SEM	MDC₉₅
Rate correct score – perceptual latency	.837	<.001	.259	.105	.290	.925	.391	.232	.064	.177
Rate correct score – response time	.805	.002	.122	.054	.149	.900	.201	.105	.033	.092

Note. ICC (2,k), intraclass correlation coefficient, two-way random effects, absolute agreement, average of measures.

p_Diff: p-value for repeated measures analysis of variance follow-up test for difference between sessions.

SD_p, pooled standard deviation.

SEM, standard error of measurement.

MDC₉₅, minimal detectable change at 95% level of confidence.

Discussion

A unique aspect of our immersive VR test was simultaneous quantification of four different perceptual-motor responses (i.e., eye, neck, arm, and step) to eight different combinations of moving visual stimuli characteristics. Regarding test validity, our prior VR research demonstrated that a composite metric derived from multiple component measures had very good discriminatory power for distinguishing individuals with a remote history of concussion from those who denied ever having sustained such an injury (Wilkerson et al., 2023). Findings in this study further support clinical use of our VR test by documenting moderate to excellent test-retest reliability for numerous measurements. Our rationale for testing these healthy participants was to quantify the magnitude of performance fluctuations that are attributable to factors other than concussion or other clinical effects (e.g., sleep deprivation, stimulant use, overtraining, or daily stressors). We expect these data to establish a “normal” degree of performance inconsistency among healthy examinees; this is likely to be greater among patients experiencing the adverse effects of a concussion (Broglio et al., 2007).

Pfister et al. (2023) recommended evaluating a diverse set of measures of complex perceptual motor movement as a means of gaining an improved understanding of relationships between cognitive processes and motor behavior). Separating an estimated PL interval from the overall task RT distinguishes the duration of primarily visual detection and decision-making processes from the total amount of time required to complete a movement response after visual stimulus appearance. Although PL may be most relevant to the efficiency of brain information processing, RT can provide an important representation of the overlap between decision-making, motor planning, and response execution. Measures of brain processing speed derived from other perceptual-motor tests are known to exhibit skewed distributions that deviate from normality in proportion to the difficulty of the decision-making process that links perceptual detection of a stimulus and response execution (Marmolejo-Ramos et al., 2015). Distributional skewing of both RT and IIV measures are also known to be sensitive to disruption of white matter integrity, which is most likely to be revealed by complex attention-demanding tasks (Jackson et al., 2012).

In this study, the composite RCS metrics for both PL and RT demonstrated good to excellent reliability, which supports their value as numeric representations of speed-accuracy trade-off. Diminished response inhibition (i.e., impulsivity) has been shown to be a behavioral manifestation of white matter disruption that often results in fast and inaccurate responses to stimuli (Goswami et al., 2016; Shen et al., 2020). Prior research has found RCS to yield an integrated effect size that accounts for a larger proportion of variance than that provided separately by the component measures used for its calculation (Vandierendonck, 2017), and it has been shown to approximate a normal distribution more closely than either of its component measures (Woltz & Was, 2006). Furthermore, composite metrics may be more sensitive than a single raw measure for detection of subtle early changes in cognitive abilities (Jonaitis et al., 2019). Higher ICC values for sessions 2 to 3 compared to sessions 1 to 2 clearly indicate that an RCS learning effect occurred, which might be addressed by providing a more thorough practice session prior to administration of the first test used for data acquisition.

Logarithmic transformation for improved distribution normality of both mean and IIV data is appropriate for test-retest reliability assessment (Dunlap et al., 1994; Olivier & Norberg, 2010), but it complicates interpretation of some aspects of the analysis results. Although Log_e transformation precludes calculation of statistics that require a sample SD, back-transformation of mean values provides estimated median values for the original data that can be used to compare session-to-session changes (Lee, 2020). In this study, most 3-session ICC values for mean PL and RT were good to excellent (7 of 8), whereas the majority for IIV of PL and RT were moderate to good (7 of 8). Despite mean PL and RT having demonstrated better reliability than IIV of PL and RT, the latter measures appear to provide adequate reliability, and they may be more sensitive to subtle change in cognitive function (Jonaitis et al., 2019). An inverse association between brain signal variability and trial-to-trial IIV of RT has been documented in healthy individuals (Grady & Garrett, 2018), as has an inverse association with both IIV of RT and response accuracy among patients with a history of traumatic brain injury (Beharelle et al., 2012).

Very recent research has confirmed that brain signal variability optimizes information exchange among distributed network components (Baracchini et al., 2021), and that disrupted connectivity is likely due to microscopic damage within white matter tracts (Wong et al., 2023). Furthermore, clinical testing that measures responses to congruent and incongruent visual stimuli has been shown to yield mean RT and IIV of RT measures that associate with resting state functional connectivity patterns among female college athletes (Roberts et al., 2022). Intra-individual variability has been conceptualized as either “inconsistency” across successive task trials or “dispersion” of standardized scores for multiple cognitive tasks that differ somewhat from one another (Costa et al., 2019; Holtzer et al., 2008). Because the VR test we used included eight different combinations of stimulus types, initial location, and movement direction that required activation of different neural circuits, the 40-trial IIV (i.e., intra-individual SD) incorporated aspects of both trial-to-trial inconsistency and dispersion among values for trials with different characteristics.

Limitations and Directions for Further Research

We observed the poorest pairwise ICC values between sessions (i.e., 1– 2 and 2–3) for mean RT and IIV of RT for eye movements, which ranged from .347 to .591. Inadequate continuous tracking of eye position may have contributed to the low values, as well as the reversal of eye movement direction required by some stimulus-response combinations. Because neck rotation toward the correct target location could permit its visualization with a lesser amount of eye rotation, differing relative amounts of eye and neck rotation on different trials may have contributed to low test-retest reliability. Interestingly, both pairwise ICC values for mean PL were good (.886 and .867), as well as the sessions 1 to 2 ICC for IIV of PL (.883). Previous research has demonstrated an association between inconsistency of eye movements in response to moving visual stimuli and incidence of head impacts during participation in high school ice hockey, which suggests that efficient eye movements may contribute to collision avoidance (Kiefer et al., 2018). If using VR to assess change in eye responses to visual stimuli, the results of this study suggest that eye PL (i.e., stimulus appearance to 6 degrees of angular displacement) provides better reliability than eye RT (i.e., stimulus appearance to maximum angular displacement).

The primary limitation of this study was our use of a convenience sample that included only 9 male and 10 female graduate students. Although an ICC value for a given test measure can depend on the characteristics of the participants included in the analysis (Weir, 2005), our findings included responses from 17 former competitive athletes that probably have good generalizability to other groups of healthy, competitive young adults. Despite the relatively small sample size, the ICC (2,k) model specification included systematic variance from session to session and imposed a stringent standard for absolute agreement of measures. One-day intervals between test sessions were chosen to obtain a precise estimate of a short-term learning effect. A much longer interval between test sessions would be needed to assess retention or decline of performance capabilities among athletes who have recently sustained a concussion, which may enhance interpretation of change between a pre-participation screening test and a subsequent post-injury evaluation, or assessment of post-injury recovery rate.

Test-retest reliability is an important consideration for selection of a test measure to use for clinical assessment, but a measure that exhibits excellent reliability is not necessarily the most valid indicator of a suboptimal or dysfunctional state (Jonaitis et al., 2019). Thus, these results must be complemented by separate validity research that establishes associations of specific perceptual-motor test measures with other clinically meaningful measures or categorizations of an individual’s health status or functional capabilities. Future research to validate use of these VR metrics for assessment of acute concussion effects should include both injured patients and healthy participants.

Conclusion

Numerous measures of perceptual-motor performance derived from an immersive VR system demonstrated good to excellent test-retest reliability values among healthy graduate students who were tested on three consecutive days. A learning effect was evident for mean PL and RT measures between session 1 and session 2, which was diminished between session 2 and session 3. Reported values for the SEM and MDC₉₅ of the RCS metric for integrated representation of speed-accuracy trade-off provide a basis for estimation of meaningful change between testing sessions. The results of this study affirm the suitability of the immersive VR test for clinical use, but further research is needed to validate it for specific applications, such as assessment of improved or worsened status immediately following diagnosis of a concussion.

Supplemental Material

Supplemental Material - Test-Retest Reliability of Immersive Virtual Reality Measures of Perceptual-Motor Performance

Supplemental Material for Test-Retest Reliability of Immersive Virtual Reality Measures of Perceptual-Motor Performance by Gary B. Wilkerson, Jonathan C. Lansey, Courtney N. Noblett and Caroline E. Sarris in Perceptual and Motor Skills

Footnotes

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: GBW serves as a consultant and JCL is employed as Data Science Lead for REACT Neuro (Cambridge, MA), which supplied the virtual reality equipment used to acquire the study data.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Gary B. Wilkerson

Supplemental Material

Supplemental material for this article is available online.

Author Biographies

Gary B.Wilkerson, EdD, ATC is a professor at the University of Tennessee at Chattanooga, Chattanooga, TN.

Jonathan C. Lansey, MA serves as Data Science Lead for REACT Neuro, Cambridge, MA.

Courtney N. Noblett, MS was a graduate student at the University of Tennessee at Chattanooga when the study was conducted. She currently serves as an athletic trainer for Maury Regional Medical Center, Columbia, TN.

Caroline E. Sarris, MS was a graduate student at the University of Tennessee at Chattanooga when the study was conducted. She currently serves as an athletic trainer for the Miami Marlins professional baseball organization, Miami, FL.

References

Baracchini

Mišić

Setton

Mwilambwe-Tshilobo

Girn

Nomi

J. S.

Uddin

L. Q.

Turner

G. R.

Spreng

R. N.

(2021). Inter-regional BOLD signal variability is an organizational feature of functional brain networks. NeuroImage, 237, 118149. https://doi.org/10.1016/j.neuroimage.2021.118149

Beharelle

A. R.

Kovačević

McIntosh

A. R.

Levine

(2012). Brain signal variability relates to stability of behavior after recovery from diffuse brain injury. NeuroImage, 60(2), 1528–1537. https://doi.org/10.1016/j.neuroimage.2012.01.037

Broglio

S. P.

Ferrara

M. S.

Macciocchi

S. N.

Baumgartner

T. A.

Elliott

(2007). Test-retest reliability of computerized concussion assessment programs. Journal of Athletic Training, 42(4), 509–514.

Cole

T. J.

Altman

D. G.

(2017). Statistics Notes: Percentage differences, symmetry, and natural logarithms. BMJ, 358, j3683. https://doi.org/10.1136/bmj.j3683

Costa

A. S.

Dogan

Schulz

J. B.

Reetz

(2019). Going beyond the mean: Intraindividual variability of cognitive performance in prodromal and early neurodegenerative disorders. The Clinical Neuropsychologist, 33(2), 369–389. https://doi.org/10.1080/13854046.2018.1533587

Crasta

J. E.

Nebel

M. B.

Svingos

Tucker

R. N.

Chen

H. W.

Busch

Caffo

B. S.

Stephens

Suskauer

S. J.

(2023). Rethinking recovery in adolescent concussions: Network‐level functional connectivity alterations associated with motor deficits. Human Brain Mapping, 44(8), 3271–3282. Mar 31. Epub ahead of print. https://doi.org/10.1002/hbm.26280

DiCesare

C. A.

Kiefer

A. W.

Bonnette

Myer

G. D.

(2020). High-risk lower-extremity biomechanics evaluated in simulated soccer-specific virtual environments. Journal of Sport Rehabilitation, 29(3), 294–300. https://doi.org/10.1123/jsr.2018-0237

Dunlap

W. P.

Chen

Greer

(1994). Skew reduces test-retest reliability. Journal of Applied Psychology, 79(2), 310–313. https://doi.org/10.1037/0021-9010.79.2.310

Fjell

A. M.

Westlye

L. T.

Amlien

I. K.

Walhovd

K. B.

(2011). Reduced white matter integrity is related to cognitive instability. Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 31(49), 18060–18072. https://doi.org/10.1523/JNEUROSCI.4735-11.2011

10.

Garrett

D. D.

Epp

S. M.

Kleemeyer

Lindenberger

Polk

T. A.

(2020). Higher performers upregulate brain signal variability in response to more feature-rich visual input. NeuroImage, 217(116836), 116836. https://doi.org/10.1016/j.neuroimage.2020.116836

11.

Garrett

D. D.

McIntosh

A. R.

Grady

C. L.

(2014). Brain signal variability is parametrically modifiable. Cerebral Cortex, 24(11), 2931–2940. https://doi.org/10.1093/cercor/bht150

12.

Goswami

Dufort

Tartaglia

Green

Crawley

Tator

Wennberg

Mikulis

Keightley

Davis

K. D.

(2016). Frontotemporal correlates of impulsivity and machine learning in retired professional athletes with a history of multiple concussions. Brain Structure and Function, 221(4), 1911–1925. https://doi.org/10.1007/s00429-015-1012-0

13.

Grady

C. L.

Garrett

D. D.

(2018). Brain signal variability is modulated as a function of internal and external demand in younger and older adults. NeuroImage, 169, 510–523. https://doi.org/10.1016/j.neuroimage.2017.12.031

14.

Harper

Shiraishi

Soangra

(2022). Reliability and validity of inertial sensor assisted reaction time measurement tools among healthy young adults. Sensors, 22(21), 8555. https://doi.org/10.3390/s22218555

15.

Harris

D. J.

Bird

J. M.

Smart

P. A.

Wilson

M. R.

Vine

S. J.

(2020). A framework for the testing and validation of simulated environments in experimentation and training. Frontiers in Psychology, 11, 605. https://doi.org/10.3389/fpsyg.2020.00605

16.

Holtzer

Verghese

Wang

Hall

C. B.

Lipton

R. B.

(2008). Within-person across-neuropsychological test variability and incident dementia. JAMA, 300(7), 823–830. https://doi.org/10.1001/jama.300.7.823

17.

Horan

Heckenberg

Maruff

Wright

(2020). Development of a new virtual reality test of cognition: Assessing the test-retest reliability, convergent and ecological validity of CONVIRT. BMC Psychology, 8(1), 61. https://doi.org/10.1186/s40359-020-00429-x

18.

Jackson

J. D.

Balota

D. A.

Duchek

J. M.

Head

(2012). White matter integrity and reaction time intraindividual variability in healthy aging and early-stage alzheimer disease. Neuropsychologia, 50(3), 357–366. https://doi.org/10.1016/j.neuropsychologia.2011.11.024

19.

Johnson

B. P.

Pinar

Fornito

Nandam

L. S.

Hester

Bellgrove

M. A.

(2015). Left anterior cingulate activity predicts intra-individual reaction time variability in healthy adults. Neuropsychologia, 72, 22–26. https://doi.org/10.1016/j.neuropsychologia.2015.03.015

20.

Jonaitis

E. M.

Koscik

R. L.

Clark

L. R.

Betthauser

T. J.

Berman

S. E.

Allison

S. L.

Mueller

K. D.

Hermann

B. P.

Van Hulle

C. A.

Christian

B. T.

Bendlin

B. B.

Blennow

Zetterberg

Carlsson

C. M.

Asthana

Johnson

S. C.

(2019). Measuring longitudinal cognition: Individual tests versus composites. Alzheimer’s and Dementia, 11, 74–84. https://doi.org/10.1016/j.dadm.2018.11.006

21.

Kelly

A. M. C.

Uddin

L. Q.

Biswal

B. B.

Castellanos

F. X.

Milham

M. P.

(2008). Competition between functional brain networks mediates behavioral variability. NeuroImage, 39(1), 527–537. https://doi.org/10.1016/j.neuroimage.2007.08.008

22.

Kiefer

A. W.

DiCesare

Nalepka

Foss

K. B.

Thomas

Myer

G. D.

(2018). Less efficient oculomotor performance is associated with increased incidence of head impacts in high school ice hockey. Journal of Science and Medicine in Sport, 21(1), 4–9. https://doi.org/10.1016/j.jsams.2017.06.016

23.

Koo

T. K.

M. Y.

(2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012

24.

Lee

D. K.

(2020). Data transformation: A focus on the interpretation. Korean Journal of Anesthesiology, 73(6), 503–508. https://doi.org/10.4097/kja.20137

25.

Lempke

L. B.

Shumski

E. J.

Prato

T. A.

Lynall

R. C.

(2022). Reliability and minimal detectable change of the standardized assessment of reaction time (StART). Journal of Athletic Training, 58(6), 579–587. Oct 17. Epub ahead of print. https://doi.org/10.4085/1062-6050-0391.22

26.

Lynall

R. C.

Johnson

R. S.

Lempke

L. B.

Schmidt

J. D.

(2021). Test–retest reliability of a functional reaction time assessment battery. Journal of Sport Rehabilitation, 30(8), 1237–1241. https://doi.org/10.1123/jsr.2021-0021

27.

Marmolejo-Ramos

Cousineau

Benites

Maehara

(2015). On the efficacy of procedures to normalize Ex-Gaussian distributions. Frontiers in Psychology, 5(1548), 1548. https://doi.org/10.3389/fpsyg.2014.01548

28.

McGowan

A. L.

Bretzin

A. C.

Savage

J. L.

Petit

K. M.

Covassin

Pontifex

M. B.

(2019). Acute and protracted disruptions to inhibitory control following sports-related concussion. Neuropsychologia, 131, 223–232. https://doi.org/10.1016/j.neuropsychologia.2019.05.026

29.

McGowan

A. L.

Bretzin

A. C.

Savage

J. L.

Petit

K. M.

Parks

A. C.

Covassin

Pontifex

M. B.

(2018). Preliminary evidence for differential trajectories of recovery for cognitive flexibility following sports-related concussion. Neuropsychology, 32(5), 564–574. https://doi.org/10.1037/neu0000475

30.

McIntosh

A. R.

Kovacevic

Itier

R. J.

(2008). Increased brain signal variability accompanies lower behavioral variability in development. PLoS Computational Biology, 4(7), Article e1000106. https://doi.org/10.1371/journal.pcbi.1000106

31.

Mennes

Zuo

X.-N.

Kelly

Di Martino

Zang

Y.-F.

Biswal

Castellanos

F. X.

Milham

M. P.

(2011). Linking inter-individual differences in neural activation and behavior to intrinsic brain dynamics. NeuroImage, 54(4), 2950–2959. https://doi.org/10.1016/j.neuroimage.2010.10.046

32.

Merritt

V. C.

Greenberg

L. S.

Guty

Bradson

M. L.

Rabinowitz

A. R.

Arnett

P. A.

(2019). Beyond measures of central tendency: Novel methods to examine sex differences in neuropsychological performance following sports-related concussion in collegiate athletes. Journal of the International Neuropsychological Society: JINS, 25(10), 1094–1100. https://doi.org/10.1017/S1355617719000882

33.

Merritt

V. C.

Greenberg

L. S.

Meyer

J. E.

Arnett

P. A.

(2021). Loss of consciousness is associated with elevated cognitive intra-individual variability following sports-related concussion. Journal of the International Neuropsychological Society: JINS, 27(2), 197–203. https://doi.org/10.1017/S1355617720000727

34.

Morral-Yepes

Moras

Bishop

Gonzalo-Skok

(2022). Assessing the reliability and validity of agility testing in team sports: A systematic review. The Journal of Strength & Conditioning Research, 36(7), 2035–2049. https://doi.org/10.1519/JSC.0000000000003753

35.

Müller

Dekker

Morris-Binelli

Piggott

Hoyne

Christensen

Fadde

Zaichkowsky

Brenton

Hambrick

D. Z.

(2023). Attributes of expert anticipation should inform the design of virtual reality simulators to accelerate learning and transfer of skill. Sports Medicine, 53(2), 301–309. https://doi.org/10.1007/s40279-022-01735-7

36.

Nolin

Stipanicic

Henry

Joyal

C. C.

Allain

(2012). Virtual reality as a screening tool for sports concussion in adolescents. Brain Injury, 26(13–14), 1564–1573. https://doi.org/10.3109/02699052.2012.698359

37.

Olivier

Norberg

M. M.

(2010). Positively skewed data: Revisiting the box-cox power transformation. International Journal of Psychological Research, 3(1), 68–77. https://doi.org/10.21500/20112084.846

38.

Parks

A. C.

Moore

R. D.

C.-T.

Broglio

S. P.

Covassin

Hillman

C. H.

Pontifex

M. B.

(2015). The association between a history of concussion and variability in behavioral and neuroelectric indices of cognition. International Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology, 98(3 Pt 1), 426–434. https://doi.org/10.1016/j.ijpsycho.2015.08.006

39.

Pfister

Neszmélyi

Kunde

(2023). Response durations: A flexible, no-cost tool for psychological science. Current Directions in Psychological Science, 32(2), 160–166. https://doi.org/10.1177/09637214221141692

40.

Rabinowitz

A. R.

Arnett

P. A.

(2013). Intraindividual cognitive variability before and after sports-related concussion. Neuropsychology, 27(4), 481–490. https://doi.org/10.1037/a0033023

41.

Reneker

J. C.

Babl

Pannell

W. C.

Adah

Flowers

M. M.

Curbow-Wilcox

Lirette

(2019). Sensorimotor training for injury prevention in collegiate soccer players: An experimental study. Physical Therapy in Sport: Official Journal of the Association of Chartered Physiotherapists in Sports Medicine, 40, 184–192. https://doi.org/10.1016/j.ptsp.2019.09.012

42.

Reneker

J. C.

Pannell

W. C.

Babl

R. M.

Zhang

Lirette

S. T.

Adah

Reneker

M. R.

(2020). Virtual immersive sensorimotor training (VIST) in collegiate soccer athletes: A quasi-experimental study. Heliyon, 6(7), Article e04527. https://doi.org/10.1016/j.heliyon.2020.e04527

43.

Roberts

S. D.

Wilson

Rahimi

Gorbet

Sergio

Stevens

W. D.

Wojtowicz

(2022). Investigation of baseline attention, executive control, and performance variability in female varsity athletes. Brain Imaging and Behavior, 16(4), 1636–1645. https://doi.org/10.1007/s11682-022-00635-8

44.

Santos

F. V.

Yamaguchi

Buckley

T. A.

Caccese

J. B.

(2020). Virtual reality in concussion management: From lab to clinic. Journal of Clinical and Translational Research, 5(4), 148–154. http://dx.doi.org/10.18053/jctres.05.2020S4.008

45.

Schuermans

Van Hootegem

Van den Bossche

Van Gendt

Witvrouw

Wezenbeek

(2022). Extended reality in musculoskeletal rehabilitation and injury prevention-A systematic review. Physical Therapy in Sport: Official Journal of the Association of Chartered Physiotherapists in Sports Medicine, 55, 229–240. https://doi.org/10.1016/j.ptsp.2022.04.011

46.

Shen

I.-H.

Lin

Y.-J.

Chen

C.-L.

Liao

C.-C.

(2020). Neural correlates of response inhibition and error processing in individuals with mild traumatic brain injury: An event-related potential study. Journal of Neurotrauma, 37(1), 115–124. https://doi.org/10.1089/neu.2018.6122

47.

Slobounov

Newell

(2006a). Application of virtual reality graphics in assessment of concussion. CyberPsychology and Behavior: The Impact of the Internet, Multimedia and Virtual Reality on Behavior and Society, 9(2), 188–191. https://doi.org/10.1089/cpb.2006.9.188

48.

Slobounov

Sebastianelli

Cao

Newell

(2007). Differential rate of recovery in athletes after first and second concussion episodes. Neurosurgery, 61(2), 338–344. https://doi.org/10.1227/01.neu.0000280001.03578.FF

49.

Slobounov

Tutwiler

Sebastianelli

Slobounov

(2006b). Alteration of postural responses to visual field motion in mild traumatic brain injury. Neurosurgery, 59(1), 134–193. https://doi.org/10.1227/01.NEU.0000219197.33182.3F

50.

Teel

Gay

Johnson

Slobounov

(2016a). Determining sensitivity/specificity of virtual reality-based neuropsychological tool for detecting residual abnormalities following sport-related concussion. Neuropsychology, 30(4), 474–483. https://doi.org/10.1037/neu0000261

51.

Teel

E. F.

Gay

M. R.

Arnett

P. A.

Slobounov

S. M.

(2016b). Differential sensitivity between a virtual reality balance module and clinically used concussion balance modalities. Clinical Journal of Sport Medicine: Official Journal of the Canadian Academy of Sport Medicine, 26(2), 162–166. https://doi.org/10.1097/jsm.0000000000000210

52.

Teel

E. F.

Slobounov

S. M.

(2015). Validation of a virtual reality balance module for use in clinical concussion assessment and management. Clinical Journal of Sport Medicine: Official Journal of the Canadian Academy of Sport Medicine, 25(2), 144–148. https://doi.org/10.1097/jsm.0000000000000109

53.

Vandierendonck

(2017). A comparison of methods to combine speed and accuracy measures of performance: A rejoinder on the binning procedure. Behavior Research Methods, 49(2), 653–673. https://doi.org/10.3758/s13428-016-0721-5

54.

Wang

C.-H.

Yang

C.-T.

Moreau

Muggleton

N. G.

(2017). Motor expertise modulates neural oscillations and temporal dynamics of cognitive control. NeuroImage, 158, 260-270. https://doi.org/10.1016/j.neuroimage.2017.07.009

55.

Weir

J. P.

(2005). Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM. The Journal of Strength & Conditioning Research, 19(1), 231–240. https://doi.org/10.1519/15184.1

56.

Wilkerson

G. B.

Colston

M. A.

Acocello

S. N.

Hogg

J. A.

Carlson

L. M.

(2023). Subtle impairments of perceptual-motor function and well-being are detectable among military cadets and college athletes with self-reported history of concussion. Frontiers in Sports and Active Living, 5, 1046572. https://doi.org/10.3389/fspor.2023.1046572

57.

Woltz

D. J.

Was

C. A.

(2006). Availability of related long-term memory during and after attention focus in working memory. Memory & Cognition, 34(3), 668–684. https://doi.org/10.3758/bf03193587

58.

Wong

J. K.

Churchill

N. W.

Graham

S. J.

Baker

A. J.

Schweizer

T. A.

(2023). Altered connectivity of default mode and executive control networks among female patients with persistent post-concussion symptoms. Brain Injury, 37(2), 147–158. https://doi.org/10.1080/02699052.2022.2163290

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.53 MB