Multicenter Validation of a Telephone-Based Caregiver Interview for Longitudinal Assessment of Outcome after Severe Brain Injury: A Traumatic Brain Injury Model Systems Study

Abstract

There is limited information about long-term outcomes following severe acquired brain injury (ABI). This is due, in part, to the lack of validated longitudinal assessment measures that can be administered remotely. To address this gap, we developed a caregiver-administered telephone interview designed for the remote evaluation of the functional status of patients who are too impaired to provide reliable self-report. The interview comprises items drawn from three existing standardized instruments: the Coma Recovery Scale–Revised (CRS-R), Cognitive Impairment subscale of the Confusion Assessment Protocol (CAP-Cog), and Galveston Orientation and Amnesia Test (GOAT) (subsequently referred to as the CRS-RT, CAP-CogT, and GOAT-T to reflect telephone administration). The CRS-RT items evaluate the level of consciousness, while the CAP-CogT and GOAT-T items assess basic aspects of cognition. We administered the caregiver interview to 48 caregivers of persons with severe acquired disability (i.e., vegetative state to confusional state) and validated caregiver responses by conducting in-person patient examinations using the original versions of the assessment instruments. To establish the concurrent validity of the caregiver interview, we assessed the correlation between the findings from the caregiver interview and the patient examination, using Lin’s concordance correlation coefficient (CCC). The mean (standard error) sensitivity, specificity, and accuracy across both sets of interview items were 0.82 (0.03), 0.68 (0.04), and 0.75 (0.03), respectively. Lin’s CCC between caregiver responses to the nine interview items addressing the level of consciousness and the corresponding patient examination findings was 0.78 (95% confidence interval [CI]: 0.65, 0.90), with six items exceeding our a priori cut-off of ≥0.70. However, the correlation between caregiver responses to the eight basic cognition items and the patient examination findings was poor (Lin’s CCC = 0.37, 95% CI: −0.09, 0.82), with only three items at or above the cut-off. These results indicate that the CRS-RT can be administered remotely to caregivers of persons with severe ABI-related disability to monitor neurobehavioral status longitudinally. The CAP-CogT and GOAT-T items require further study before they can be used for clinical outcome assessment.

Keywords

brain injury Coma Recovery Scale–Revised Confusion Assessment Protocol Galveston Orientation and Amnesia Test telephone neurobehavioral assessments

Introduction

Traumatic brain injury (TBI), a subtype of acquired brain injury (ABI), is among the top three contributors to disability and injury-related death in the United States and is projected to remain so until at least 2030.¹ The Centers for Disease Control and Prevention (CDC) estimate that nearly 250,000 persons are hospitalized annually in the United States for moderate-to-severe TBI (msTBI).² TBI impacts physical, cognitive, psychological, behavioral, and social aspects of everyday function. While the effects can be chronic, the natural history of recovery is highly variable.³ A 2020 report issued by the CDC and the National Institute on Disability, Independent Living and Rehabilitation Research on 4838 persons with msTBI enrolled in the TBI Model Systems National Database found that for most participants, functional outcomes were dynamic between 1 and 5 years post-injury. Among those followed up, 26% improved, 39% remained stable, and 35% declined or died. Although the report also noted that nearly 50% required some cognitive assistance at both time points,⁴ additional evidence suggests that approximately 20% of those with msTBI regain functional independence between 1 and 10 years post-injury.^5–11 The predictors of recovery and decline remain elusive, complicating prognostication and treatment planning. The course of recovery from nontraumatic causes (e.g., hypoxic-ischemic insults, stroke) of severe ABI has been understudied but is generally less favorable compared with TBI.¹²

The limitations in our understanding of the long-term trajectory of recovery and disability across the lifespan are due, in part, to the lack of a national surveillance system for monitoring the evolution of ABI. The majority of those who sustain severe ABI become disconnected from the mainstream health care system early after injury, increasing the caregiver burden.¹³ The most severely injured experience prolonged disorders of consciousness (DoC), including the vegetative state (VS)¹² and minimally conscious state (MCS).¹² VS is a state of wakeful unconsciousness that typically follows emergence from coma.¹² MCS is characterized by at least one clearly discernible behavioral sign of consciousness.¹⁴ When there is evidence of preserved receptive or expressive language function, the condition is referred to as MCS-plus (MCS+), and when the evidence for consciousness does not include signs of language, the term MCS-minus (MCS−) is applied.^12,15 Emergence from MCS (eMCS) occurs when the patient regains the ability to reliably communicate or use objects in a functional manner.¹⁴ Persons with prolonged DoC frequently have significant physical limitations, recurrent medical instability, insufficient insurance coverage, and lack access to transportation and other public accommodations, placing them at high risk of loss to follow-up.¹⁶ These barriers impede access to health services and reduce the opportunity to better understand the incidence and causes of improvement and decline.

The dearth of information about long-term outcomes after severe ABI is further exacerbated by the paucity of validated assessment measures designed to track the changes in functional status over time. The Glasgow Outcome Scale-Extended (GOSE)¹⁷ and Disability Rating Scale (DRS)¹⁸ are often used to assess global outcomes after ABI; however, both were developed for use with persons with TBI, and neither is sensitive to the subtle but meaningful changes in function.^19,20 The Coma Recovery Scale–Revised (CRS-R)²¹ is more sensitive to subtle changes but requires in-person administration, limiting its utility in severely disabled patients. Accessible efficient approaches to outcome assessment are needed for the longitudinal monitoring of persons with severe ABI-related disability. Family caregivers may be an untapped resource for acquiring long-term outcome data given their intimate role in the provision of direct care.

To address the need for an outcome assessment measure that can be administered remotely to follow up persons with severe disability who are unable to provide self-report or undergo in-person follow-up, we developed a telephone-administered caregiver interview comprising modified items drawn from three existing performance-based measures of neurobehavioral function: the CRS-R, the Cognitive Impairment subscale from the Confusion Assessment Protocol (CAP-Cog),^22,23 and the Galveston Orientation and Amnesia Test (GOAT).²⁴ We assessed the concurrent validity of the caregiver interview by comparing the caregiver responses acquired over the telephone with scores obtained during in-person assessments performed by trained examiners using the original CRS-R, CAP-Cog, and GOAT.

Methods

Participants

We enrolled a convenience sample of 48 persons with severe ABI and their caregivers (i.e., dyads) at five rehabilitation hospitals between August 2020 and March 2023. For persons with severe ABI, informed consent for participation was obtained from the legally authorized surrogate by IRB-approved members of the study team. Caregivers provided consent for their own participation. The inclusion criteria for persons with ABI were as follows: (1) ≥18 years of age, (2) history of ABI occurring at least 10 months prior to study enrollment, and (3) inability to provide reliable responses to self-report outcome measures due to cognitive impairment (see Supplementary Data for further details on the inclusion criteria for persons with ABI). The inclusion criteria for the caregivers were as follows: (1) ≥18 years of age and (2) responsibility for providing direct care (e.g., feeding, grooming, or toileting) to the person with ABI for at least 3 months over the 12 months preceding study enrollment.

Procedure

The CRS-R was designed to monitor recovery in persons with DoC, including those in coma, VS, MCS−, MCS+, and eMCS.^14,15,25 The scale assesses arousal level, receptive and expressive language, visuoperception, motor function, and communication ability.²¹ Prior studies have shown that the CRS-R detects behavioral changes missed by other assessment measures.^26,27 On the basis of its strong psychometric characteristics, the CRS-R was selected as a TBI common data element by the National Institute of Neurological Diseases and Stroke.²⁸ The CAP-Cog is one of seven subscales that comprise the CAP,²² which is used to track the seven cardinal features of the acute confusional state (i.e., disorientation, cognitive impairment, restlessness, nighttime sleep disturbance, daytime somnolence, psychotic-type symptoms, and symptom fluctuation). The CAP-Cog²³ includes seven items that assess attention, visual memory, and language comprehension. The GOAT²⁴ was developed to detect and monitor signs of post-traumatic amnesia during early recovery from TBI. These three measures are responsive to cognitive and behavioral changes over time after brain injury^21,23,24 and, in combination, provide a comprehensive profile of functional status in persons with severe disability, ranging from coma through recovery from the confusional state. We followed a three-step process to develop and validate the telephone versions of the CRS-R, CAP-Cog and GOAT (subsequently referred to as the CRS-RT, CAP-CogT, and GOAT-T to reflect telephone administration).

Step 1: Item selection

To develop the CRS-RT, the investigator team identified the 13 items from the CRS-R²¹ that are consistent with a diagnosis of MCS−, MCS+, or eMCS (i.e., to differentiate persons who are conscious from those who are not conscious). We eliminated “object localization” and “object manipulation,” as these two behaviors are almost always preceded by other signs of consciousness.²⁹ We also collapsed consistent and reproducible command-following into a single question, as the latter is subsumed by the former. Finally, we omitted visual fixation but retained visual pursuit due to the previously established overlap in the item content of these two behaviors.^21,30 The nine remaining CRS-RT items, representing three functional domains (i.e., language, visuoperception, motor function) were converted into interview questions. The CAP-CogT consists of seven items derived from the CAP-Cog, and the GOAT-T includes the temporal and spatial orientation items from the GOAT, which were collapsed into a single item. We further combined the CAP-CogT and the orientation item of the GOAT-T into a single measure.

We established a common scoring system for each item of the interview anchored to the frequency with which each behavior was observed by the caregiver over the 2 weeks preceding the study interview (i.e., never = 0, some of the time = 1, most of the time = 2, and always = 3; see Supplementary Data for the CRS-RT, CAP-CogT, and GOAT-T record forms). Of note, the total possible scoring ranges differ between the in-person and telephone measures: CRS-R: 0–23 versus CRS-RT: 0–27; CAP-Cog: 0–28 versus CAP-CogT: 0–21; GOAT: −8–100 versus GOAT-T: 0–3. Test Completion Codes (TCCs, Supplementary Table S1) were also developed to identify confounding factors that interfere with or invalidate administration of the measures.

Step 2: Stakeholder evaluation

We conducted focus groups with caregivers at four participating sites to obtain feedback on the readability and content of the items selected for inclusion in the CRS-RT, CAP-CogT, and GOAT-T (see Supplementary Appendix for the focus group interview guide and Supplementary Data for modifications to the instruments made based on the stakeholder feedback). We also consulted experts at the Model Systems Knowledge Translation Center (MSKTC.org), a national center that collaborates with researchers to translate health information into easily understandable language for caregivers and persons with TBI, spinal cord, and burn injury. As per MSKTC recommendations, we revised the interview to achieve a Flesch–Kincaid³¹ 7th-grade reading level.

Step 3: Validation

To validate the CRS-RT, CAP-CogT, and GOAT-T caregiver ratings, Rater 1 (R1) administered the standard version of each scale in person to each participant with ABI, while Rater 2 (R2) administered the telephone interview version to the caregiver. These assessments were conducted such that the raters (R1 and R2) were blinded to each other’s assessments and the caregivers were blinded to the corresponding in-person assessment. Administration of the CAP-Cog/GOAT and CAP-CogT/GOAT-T was contingent upon demonstrating the presence of both command-following and intelligible verbalization on the CRS-R and CRS-RT, respectively, as these abilities are prerequisites for the valid administration of these measures. Further details on the approach used to develop the item content and scoring criteria for all three measures, as well as the validation procedure, are available in the Supplementary Data (see Methods and Supplementary Tables S2 and S3).

Data analysis

Behaviors assessed by the CRS-RT, CAP-CogT, and GOAT-T items were dichotomized as “present” (i.e., reported by the caregiver to be observed “most of the time” or “always”) versus “absent” (i.e., reported by the caregiver to be observed “some of the time” or “never”) to allow comparison with the scores of the in-person versions of the CRS-R, CAP-Cog, and GOAT. We assessed the concurrent validity of the CRS-RT, CAP-CogT, and GOAT-T by calculating Lin’s concordance correlation coefficient (CCC)^32,33 and 95% confidence intervals (CI) for the number of behaviors “present” on the in-person versus telephone version of each measure. Our a priori threshold for validity was Lin’s CCC ≥0.70.³⁴ We also evaluated the accuracy, sensitivity, and specificity of the caregiver’s rating relative to the examiner’s in-person rating on each of the corresponding CRS-R, CAP-Cog, and GOAT items and for the average of all the items. We used Cronbach’s alpha to assess the internal consistency reliability of the instruments, a function of the number of test items and the average intercorrelation among the items of the concept.

To address missing data, we generated multiple imputation samples (100) for items that were rated “not applicable” or “not testable” using the SAS proc mi procedure with fully conditional specification methods (see Supplementary Data for additional details, including sample size calculations). Any items designated “not testable” indicate that permanent or temporary medical problems unrelated to the level of consciousness may interfere with the ability to perform a behavior (e.g., jaw wired shut). The ninth and final item (i.e., functional communication) of the CRS-RT was designated as “not applicable” if the response to the prior item (i.e., intentional communication) was “never.”

We coupled multiple imputation with an approach that imputes missing values based on iterations with model convergence as described by Rubin.³⁵ The assumptions of multiple imputation as described by Rubin³⁵ were met (see Supplementary Table S4 and corresponding Supplementary Data text). To impute the missing data, we used Bayesian regression imputation, or predictive mean matching, including demographic variables. This approach draws upon multiple imputed samples (fitted posteriori distributions). We used this approach because we had sufficient data to maintain realistic associations between the missing variables and the predictor variables without missing values.

Results

Participants

We enrolled 48 patient–caregiver dyads. Among the persons with ABI, 69% were male, the mean (standard deviation (SD)) age was 37.50 (14.40) years, 21% were non-white, the mean time post-injury was 5.13 (5.41) years, and 85% had TBI (see Table 1 for the sample characteristics). Based on the in-person CRS-R assessments, there were 4 (8%) participants who were in VS, 11 (23%) in MCS−, 8 (17%) in MCS+, 22 (46%) in eMCS, and 3 (6%) whose diagnosis could not be determined. There were 18 participants who met the criteria for the administration of the CAP-Cog and GOAT (including 16 who were in eMCS, 1 who was in MCS+, and 1 whose diagnosis could not be determined according to the CRS-R); however, one of these participants did not have valid CAP-Cog and GOAT scores and was excluded. Among the 17 participants who underwent in-person CAP-Cog and GOAT assessments, 9 (53%) were in a confusional state. Distributions of total scores for each measure are provided in Supplementary Figures S1–S3.

Table 1.

Participants’ Demographic and Injury Characteristics

Persons with ABI
Sex (n [%])
Female	15 [31.25%]
Male	33 [68.75%]
Age at Interview
Mean (SD) years	37.50 (14.40)
Time Since Injury (n [%])
1–2 years	19 [39.58%]
>2–5 years	11 [22.92%]
>5 years	18 [37.5%]
Mean (SD) years	5.13 (5.41)
Race (n [%])
White	38 [79.17%]
Other	10 [20.83%]
(Black/African American, Asian, Indian, Pacific Islander, Other, Refused)
Etiology/mechanism of injury (n [%])
TBI	41 [85.42%]
Motor vehicle	26 [63.41%]
Pedestrian	6 [14.63%]
Fall	5 [12.20%]
Gunshot wound	2 [4.87%]
Sports	2 [4.87%]
Non-TBI	7 [14.58%]
Intracerebral hemorrhage (ICH)	3 [42.86%]
Hypoxic-ischemic brain injury (HI)	4 [57.14%]
Pre-injury employment status (n [%])
Student	10 [20.83%]
Competitively employed	29 [60.42%]
Taking care of house or family	1 [2.08%]
Retired	3 [6.25%]
Unemployed: Looking	3 [6.25%]
Unknown	2 [4.16%]
Length of acute (n [%])
<7 days	2 [4.17%]
8–14 days	3 [6.25%]
15–28 days	8 [16.67%]
>28 days	33 [68.75%]
Unknown	2 [4.17%]
Length of rehab (n [%])
<7 days	0 [0.00%]
8–14 days	1 [2.08%]
15–28 days	5 [10.42%]
>28 days	42 [87.5%]
Living with before injury (n [%])
Alone	6 [12.50%]
Spouse or significant other	17 [35.42%]
Other family	19 [39.58%]
Someone else	6 [12.50%]
Residential status at interview (n [%])
Private residence	31 [64.58%]
Nursing home/subacute care	9 [18.75%]
Adult home	4 [8.33%]
Hospital: Rehabilitation	4 [8.33%]

Caregivers of persons with ABI
Sex (n [%])
Female	36 [75.00%]
Male	12 [25.00%]
Age
Mean (SD) years	53.44 (12.12)
Relationship to person with ABI (n [%])
Spouse	10 [20.83%]
Parent(s)	27 [56.25%]
Sibling	4 [8.33%]
Other relative	2 [4.17%]
Professional caregiver	5 [10.42%]
Race/ethnicity (n [%])
White	33 [68.75%]
Other	15 [31.25%]
(Black, African American, Asian, Hispanic Origin, other, refused to answer)
Marital status (n [%])
Married	34 [70.83%]
Single (Never married)	3 [6.25%]
Partner, fiancé	2 [4.17%]
Divorced, separated	7 [14.58%]
Widowed	1 [2.08%]
Refused to answer	1 [2.08%]
Employment status (n [%])
Competitively employed	28 [58.33%]
Taking care of house or family	13 [27.08%]
Retired	6 [12.5%]
Unknown	1 [2.08%]
Education (n [%])
Less than high school diploma	1 [2.08%]
High school diploma	19 [39.58%]
College degree or higher	28 [58.33%]
Number of months of caregiving in past year (n [%])
3–6 months	3 [6.25%]
>6–9 months	6 [12.50%]
>9–12 months	6 [12.50%]
>12 months	33 [68.75%]
Hours of care provided per week (n [%])
<10 h	6 [12.50%]
10–20 h	8 [16.67%]
>20–30 h	3 [6.25%]
>30–40 h	3 [6.25%]
>40 h	28 [58.33%]

ABI, acquired brain injury; SD, standard deviation; TBI, traumatic brain injury.

Among the 48 caregivers, 75% were female, the mean age was 53.44 (12.12) years, 31% were non-white, and 58% had a college degree or higher (Table 1). Most caregivers (58%) reported providing >40 h per week of care (Table 1). The mean (SD) time interval between in-person and telephone assessments was 2.52 (4.03) days; 32 (67%) of the in-person and telephone assessments were conducted within a 1-day window. Individual participant data are available in Supplementary Tables S5 and S6. The response frequency of each rating scale score for each item of the telephone measures is available in Figure 1.

FIG. 1.

CRS-RT and CAP-CogT/GOAT-T rating scale score response frequency. (A) Rating scale score response frequency for each CRS-RT item. (B) Rating scale score response frequency for each CAP-CogT/GOAT-T item. CRS-RT item score rating abbreviations: CF, command-following; VD, visual discrimination; Pur, visual pursuit; FOU, functional object use; Auto, automatic motor responses; Loc, localization to noxious stimulation; Exp, intelligible expression; IC, intentional communication; FC, functional communication; NT, not tested; NA, not applicable (i.e., did not meet criteria for item administration). CAP-Cog and CAP-CogT item rating abbreviations: CF, counting forward; CB, counting backward; MF, months forward; MB, months backward; V, vigilance; C, comprehension; R, recognition; CRS-RT, telephone version of the Coma Recovery Scale–Revised; CAP-CogT/GOAT-T, telephone versions of Cognitive Impairment subscale of the Confusion Assessment Protocol and Galveston Orientation and Amnesia Test.

CRS-RT

Among the 48 persons with ABI who underwent in-person CRS-R assessments, 44 (92%) had valid ratings on all the items administered. Among the 48 caregivers, 38 (79%) had valid ratings on all the items of the CRS-RT. Among the 48 dyads, 36 (75%) had valid ratings on all the CRS-R and CRS-RT items. Concurrent validity exceeded our a priori Lin’s CCC threshold of 0.70 for agreement between the CRS-R and CRS-RT ratings (Lin’s CCC = 0.78, 95% CI: 0.65, 0.90; Table 2). The correlations between the CRS-R and CRS-RT total scores for participants who had valid scores on both measures were strong³⁶ (r(34) = 0.88, p < 0.001; Fig. 2).

Table 2.

Telephone Measure Accuracy, Sensitivity, and Specificity

A. CRS-RT accuracy, sensitivity, and specificity (N = 48)
	Item	Accuracy (SE)	Sensitivity (SE)	Specificity (SE)
1	Command-following	0.81 (0.06)*	0.81 (0.07)*	0.82 (0.09)*
2	Object recognition	0.80 (0.06)*	0.88 (0.06)*	0.70 (0.10)*
3	Visual pursuit	0.70 (0.07)*	0.70 (0.07)*	0.60 (0.27)
4	Functional object use	0.76 (0.07)*	0.90 (0.07)*	0.66 (0.10)
5	Automatic motor responses	0.69 (0.08)	0.78 (0.08)*	0.58 (0.13)
6	Localization to noxious stimulation	0.67 (0.07)	0.71 (0.08)*	0.60 (0.14)
7	Intelligible expression	0.85 (0.06)*	0.93 (0.06)*	0.80 (0.08)*
8	Intentional communication	0.91 (0.04)*	0.95 (0.05)*	0.87 (0.08)*
9	Functional communication	0.57 (0.14)	0.73 (0.10)*	0.47 (0.23)
Mean (SE)		0.75 (0.03)*	0.82 (0.03)*	0.68 (0.04)

B. CRS-RT Lin’s CCC (N = 48)
Number of items	Lin’s CCC (95% CI)
8 (not including “functional communication” item)	0.78 (0.67, 0.91)*
9 (including “functional communication” item)	0.78 (0.65, 0.90)*

C. CAP-CogT/GOAT-T accuracy, sensitivity, and specificity (N = 18)
	Item	Accuracy (SE)	Sensitivity (SE)	Specificity (SE)
1	Counting forward	0.78 (0.10)*	0.87 (0.09)*	0.33 (0.27)
2	Counting backward	0.56 (0.12)	0.67 (0.14)	0.33 (0.19)
3	Months forward	0.56 (0.12)	0.89 (0.10)*	0.22 (0.14)
4	Months backward	0.67 (0.11)	0.40 (0.22)	0.77 (0.12)*
5	Vigilance	0.56 (0.12)	0.56 (0.12)	0.51 (0.35)
6	Comprehension	0.50 (0.12)	0.75 (0.15)*	0.30 (0.14)
7	Recognition	0.78 (0.10)*	0.67 (0.27)	0.80 (0.10)*
8	Orientation (GOAT-T)	0.71 (0.11)*	0.69 (0.18)	0.72 (0.14)*
Mean (SE)		0.64 (0.04)	0.69 (0.05)	0.50 (0.08)

D. CAP-CogT/GOAT-T Lin’s CCC (N = 18)
Number of Items	Lin’s CCC (95% CI)
8	0.37 (−0.09, 0.82)

*Item value ≥0.70.

Accuracy, sensitivity, specificity, and Lin’s CCC were calculated with multiple imputations (100) for “not testable” and “not applicable” responses.

CRS-RT, Coma Recovery Scale–Revised Telephone Version; CAP-CogT/GOAT-T, Confusion Assessment Protocol Cognitive Impairment Subscale/Galveston Orientation and Amnesia Test Telephone Version; SE, standard error; Lin’s CCC, Lin’s Concordance Correlation Coefficient; CI, confidence interval.

FIG. 2.

CRS-RT, CAP-CogT, and GOAT-T total score correlations. (A) Among the 48 dyads assessed with the CRS-R and CRS-RT, 36 (75%) had valid ratings on all the CRS-R and CRS-RT items. The correlations between the CRS-R and CRS-RT total scores were r(34) = 0.88, p < 0.001. (B) Among the 18 dyads assessed with the CAP-Cog and CAP-CogT, 17 (94%) had valid ratings on all the CAP-Cog and CAP-CogT items. The correlations between the CAP-Cog and CAP-CogT total scores were r(15) = 0.59, p = 0.01. (C) Among the 18 dyads assessed with the GOAT and GOAT-T, 17 (94%) had valid ratings on all GOAT and GOAT-T items. The correlations between the GOAT and GOAT-T total scores were r(15) = 0.36, p = 0.16. CRS-RT, telephone version of the Coma Recovery Scale–Revised; CAP-CogT/GOAT-T, telephone versions of Cognitive Impairment subscale of the Confusion Assessment Protocol and Galveston Orientation and Amnesia Test.

Accuracy for six of the nine CRS-RT items met or exceeded the previously established 0.70 cut-off³⁷ (Table 2, Fig. 3). For three items (i.e., automatic motor response, localization to noxious stimulation, functional communication), although the standard error (SE) exceeded 0.70, accuracy (SE) was 0.69 (0.08), 0.67 (0.07), and 0.57 (0.14), respectively (Table 2). The functional communication item had the lowest accuracy and the widest SE due to the small number of observations (n = 29) available for this item as a function of the CRS-R administration guidelines, which require that intentional communication must be “present” before functional communication can be assessed. The mean (SE) sensitivity, specificity, and accuracy for all the CRS-RT items were 0.82 (0.03), 0.68 (0.04), and 0.75 (0.03), respectively (Table 2). When the results were analyzed without imputation to account for the items that were “not testable” and “not applicable,” four of the nine CRS-RT items met or exceeded the previously established 0.70 cut-off (Supplementary Table S7). Supplementary Table S8 shows the diagnostic category (i.e., VS, MCS−, MCS+, eMCS) assigned based on the CRS-R and CRS-RT ratings.

FIG. 3.

CRS-RT and CAP-CogT/GOAT-T item level accuracy. (A) Accuracy and standard error for each CRS-RT item; N = 48, except for the Functional Communication item where N = 29 because administration is contingent upon response to the preceding Intentional Communication item. (B) Accuracy and standard error for each CAP-CogT/GOAT-T Item; N = 18. (A–B) The dashed vertical line indicates a previously established 0.70 cut-off for accuracy.³⁷ Accuracy was calculated with multiple imputations (100) for “not testable” and “not applicable” responses. CRS-RT, Coma Recovery Scale–Revised Telephone version; CAP-CogT/GOAT-T, Confusion Assessment Protocol Cognitive Impairment subscale/Galveston Orientation and Amnesia Test Telephone version.

CAP-CogT and GOAT-T

Valid ratings were obtained on all the CAP-Cog and GOAT items for 17 (94%) of the 18 persons with ABI assessed on these measures. All 17 caregivers who completed the telephone interview for the 17 persons with ABI provided valid ratings on all the CAP-CogT and GOAT-T items. Concurrent validity between the eight items common to the CAP-Cog and CAP-CogT and to the GOAT and GOAT-T was poor³⁶ (Lin’s CCC = 0.37, 95% CI: −0.09, 0.82; Table 2). The correlations between the CAP-Cog and CAP-CogT total scores were moderate³⁶ (r(15) = 0.59, p = 0.01; Fig. 2). The correlations between the GOAT and GOAT-T total scores were weak³⁶ (r(15) = 0.36, p = 0.16; Fig. 2). Sensitivity, specificity, and accuracy at the item level are presented in Table 2. Mean (SE) sensitivity, specificity, and accuracy for all the items on the CAP-CogT and GOAT-T were 0.69 (0.05), 0.50 (0.08), and 0.64 (0.04), respectively (Table 2). Only three of the eight items met or exceeded the previously established 0.70 cut-off (Table 2, Fig. 3) for accuracy.

Cronbach’s alpha for the CRS-RT was 0.84 and for the CAP-CogT/GOAT-T was 0.80, indicating highly credible reliabilities. The results were comparable when the data were analyzed without imputation to account for the items that were “not testable” (Supplementary Table S7). The definitions and formulas for establishing sensitivity, specificity, and accuracy are available in Supplementary Table S9.

Discussion

The long-term follow-up of persons discharged from the acute hospital and inpatient rehabilitation settings with severe residual disability is fraught with challenges that limit understanding of the natural history of recovery and long-term response to interventions. Persons with severe disability often experience significant complications that prevent hospital and clinic-based follow-up visits (e.g., insurance coverage, transportation barriers, medical complications). While a few telephone-administered outcome measures are available for persons with severe disability caused by ABI,^17,38 these instruments lack the precision to detect subtle, clinically meaningful changes in function and/or were not validated against a performance-based reference standard. We developed and evaluated the validity of two structured telephone-administered caregiver interviews—one based on the CRS-R and the other on the CAP-Cog and GOAT—to obtain more granular information about long-term behavioral and cognitive recovery in persons with severe ABI-related disability. We found credible agreement between the caregiver and in-person examiner ratings for the CRS-RT, but not for the CAP-CogT/GOAT-T.

A recently published observational study from the 18-site TRACK-TBI Network focusing on a large cohort of patients with severe disability (n = 316) through the first 12 months post-injury reported valid completion rates as low as 34% on the in-person multidimensional outcome battery that was designed for persons with DoC.¹⁶ Logistical factors (e.g., inability to return to the study site due to scheduling or transportation problems, need for disability accommodations) accounted for the most common cause of failure to obtain valid follow-up data on outcomes at all time points. A trend toward more frequent logistical problems interfering with follow-up was also noted between 2 weeks and 12 months post-injury. The availability of the CRS-RT as a psychometrically sound tool for detecting changes in functional status will facilitate systematic long-term follow-up in persons with severe ABI and should help advance our understanding of the natural history of recovery from severe ABI.

Family caregivers play an integral role in providing support and health care to persons who are unable to do so independently. They make it possible for persons with severe ABI to be cared for at home, reducing the cost burden, and are often the first to recognize new symptoms and changes in health status. The 2022 National Strategy to Support Family Caregivers report,³⁹ which was developed by a large multistakeholder group based on extensive evidence gathered through white papers, briefings, presentations, and listening sessions, recommended that family caregivers be included as essential partners in the care teams of the person to whom they provide support. The development of telephone-based outcome assessment measures is responsive to this call. The CRS-RT provides clinicians and investigators with an efficient tool that can be administered remotely to caregivers of persons who cannot provide self-report data due to severe cognitive impairment, enabling the acquisition of accurate information on behavioral recovery or decline across the lifespan. This measure fills an important gap, as it is particularly well-suited to following up persons with DoC, who typically score at the floor of existing functional outcome measures and are major contributors to missing data rates in longitudinal outcome studies. An important takeaway from our findings is that caregivers can accurately report on the clinical status of persons with severe disability caused by ABI. Caregivers can also play an important role in reducing the lost-to-follow-up rates, which tend to be high among those who sustain severe ABI.¹⁶ Brain injury professionals are often reticent to rely on caregiver self-report for fear that caregivers lack a sufficient knowledge base or may be biased in their reporting. While these are legitimate concerns, our results suggest that they can be effectively mitigated through use of a properly vetted structured interview. Future work is needed to determine the sensitivity of caregiver-facing outcome measures in detecting clinically meaningful changes over time. Our team is currently testing the responsiveness of an abbreviated version of the CRS-RT.

Our study has some limitations that should be considered when interpreting the results. Although this study had five participating sites, the overall sample size (n = 48 patient–caregiver dyads) was relatively small and differentially impacted subanalyses involving some diagnostic subgroups. The small sample size reflects two factors. The first is that the number of patients with severe brain injury admitted for inpatient rehabilitation has been progressively declining due to changes in medical authorization policies governing admission to inpatient rehabilitation facilities.⁴⁰ Thus, our sample was skewed toward participants functioning at the upper limit of the DoC range, potentially limiting generalizability for those near the lower limit (i.e., VS). However, our results are likely generalizable to patients functioning at the level of MCS− or higher, which represents the majority of those with DoC. In a similar vein, because the sample was skewed, we are unable to determine how well the diagnostic ratings between the phone-based measure and in-person measure align. Most items on the CRS-RT had high accuracy relative to the CRS-R, suggesting that diagnostic ratings are likely to be consistent between measures. We also note that all of our analyses were conducted at the group level, indicating the need for caution when applying these results to individual cases. Future research will need to consider the consistency of responses at the individual case level.

The second factor limiting sample size concerns the rules for the administration of the CAP-CogT and GOAT-T. These measures are only administered to patients who can follow commands reproducibly and communicate intelligibly. Because the study aims required that we enroll the full continuum of patients with DoC, those at the lower end of the spectrum did not meet the criteria for CAP-CogT and GOAT-T administration and therefore could not contribute data to analyses involving these two measures. Therefore, the number of participants able to undergo evaluation on the CAP-CogT and GOAT-T was substantially smaller (N = 18) than the number able to complete the CRS-RT (N = 48), resulting in CAP-CogT and GOAT-T analyses that were likely statistically underpowered. Thus, it is important to recognize that our inability to validate these two measures does not necessarily reflect an inherent weakness in either instrument or suggest that they are unusable. This issue is best reconciled through the inclusion of a larger sample size in future studies.

Finally, because the telephone measures are based on ordinal scores, the distance (i.e., magnitude of the difference) between consecutive scores is unknown. Item Response Theory using Rasch-based methods can be used in future work to convert ordinal scores to interval-based scores, so that the magnitude of change and the determination of what is clinically relevant can be assessed, as has been done previously with the full-length CRS-R.⁴¹

An interim approach that could be considered to systematically capture longitudinal changes in function across the range of injury severity from coma through the confusional state would be to complement the CRS-RT with the Brain Injury Functional Outcome Measure (BI-FOM).⁴² The BI-FOM is a unidimensional measure of global function that was developed using Rasch analysis. This measure can be administered by telephone to persons who are able to respond to questions. In contrast to the existing global outcome measures, such as the DRS,¹⁸ GOSE,¹⁹ and Functional Independence Measure (FIM),⁴³ the BI-FOM includes items that target a wider range of function, increasing its sensitivity to subtle but meaningful changes in function and minimizing the well-recognized floor and ceiling effects of these scales. Additionally, the BI-FOM satisfies the requirements for interval scales, which can be used for both within-subject and between-subject comparisons. A limitation of the BI-FOM is that it does not include telephone-administered items that are appropriate for persons who are unable to communicate reliably. Pairing the CRS-RT with the BI-FOM eliminates this limitation. While our original motivation to develop the CRS-RT was to improve follow-up rates in longitudinal research involving persons with severe TBI-related disability, additional research will help elucidate the clinical performance of the CRS-RT. At present, the standard CRS-R and the CRS-R For Accelerated Standardized Testing (CRSR-FAST),⁴⁴ which require in-person administration, remain the preferred measures for clinical decision making.

Conclusion

The current understanding of the trajectory of recovery from ABI across the lifespan is hampered by challenges associated with conducting long-term follow-up. As disability severity increases, the probability of follow-up decreases, limiting the opportunity for advances in prognostication, identification of factors that contribute to late decline, and monitoring treatment effects. Validated standardized assessment methods that are capable of detecting clinically meaningful changes in function and can be administered remotely are key to filling these knowledge gaps.

The CRS-RT, a telephone-administered caregiver interview, was developed and validated to facilitate longitudinal outcome assessment in persons with severe disability and DoC caused by ABI. The CRS-RT captures bidirectional changes in behavior across three domains of function—language (i.e., command-following, intelligible speech, yes–no communication), visuoperception (i.e., object detection and recognition), and functional movement (i.e., upper extremity motor control, functional object use). Harnessing the unique knowledge and experience of caregivers, the CRS-RT offers investigators and clinicians an efficient, precise, and low-cost method of tracking recovery after severe ABI. Future multisite research should seek to extend the ceiling of the CRS-RT by validating additional items that can be incorporated to track resolution of the confusional state.

Transparency, Rigor, and Reproducibility Statement

This study was pre-registered at the National Rehabilitation Information Center (https://naric.com/), and the analysis plan was submitted as part of a grant application. Data were acquired between 2020 and 2023. The CRS-R, CAP, and GOAT are validated tools for assessment of the level of consciousness, the confusional state, and post-traumatic amnesia, respectively. The sample size was determined with a power calculation indicating that with 40–50 participants and caregivers, we will have 80% power to detect Lin’s CCC ≥0.85, ensuring that our threshold of ≥0.70 agreement between in-person and caregiver-reported neurobehavioral findings is achieved. We enrolled 48 persons with ABI and 48 caregivers. Among the 48 dyads assessed with the CRS-R and CRS-RT, 36 (75%) had valid data on all the items. Among the 18 dyads assessed with the CAP-Cog and CAP-CogT and GOAT and GOAT-T, 17 (94%) had valid data on all the items. The study team members were blinded to each other’s assessment and the clinical diagnosis of the participants. Caregiver participants were not told the results of the in-person assessment. The CRS-RT, CAP-CogT, and GOAT-T measures are available as a part of this submission. Data requests will be considered on a case-by-case basis.

Authors’ Contributions

A.S.: Formal analysis; investigation; data curation; writing original draft (lead); writing/reviewing and editing (lead); visualization. Y.G.B.: Conceptualization; formal analysis; investigation; writing original draft (lead); writing/reviewing and editing (lead); supervision; project administration. K.G.: Investigation; writing/reviewing and editing (supporting). F.M.H.: Investigation; writing/reviewing and editing (supporting); supervision. L.E.K.: Formal analysis; investigation; writing/reviewing and editing (supporting); supervision. R.N.-R.: Investigation; writing/reviewing and editing (supporting); supervision. P.N.: Formal analysis; investigation; writing/reviewing and editing (supporting); supervision. T.M.O.-P.: Investigation; writing/reviewing and editing (supporting). J.O.: Investigation; writing/reviewing and editing (supporting); supervision. W.R.S.: Investigation; writing/reviewing and editing (supporting). M.S.: Investigation; writing/reviewing and editing (supporting); supervision. A.B.W.: Investigation; writing/reviewing and editing (supporting). R.D.Z.: Investigation; writing/reviewing and editing (supporting); supervision. J.T.G.: Conceptualization; investigation; writing original draft (lead); writing/reviewing and editing (lead); supervision; project administration; funding acquisition.

Footnotes

Acknowledgements

The study authors would like to acknowledge the staff at the following study sites for their efforts in recruitment, data collection, project management, and study design: James A. Haley Veterans’ Administration: Jennifer Boyce, LPN, CRC; Marc A. Silva, PhD.

Author Disclosure Statement

No competing financial interests exist.

Funding Information

The research reported in this article was funded through the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR); Spaulding Rehabilitation Hospital, Harvard Medical School, Spaulding-Harvard TBI Model System, 90DPTB0011 (2017–2022); Indiana University School of Medicine, Indiana TBI Model System, 90DPTB0002 (2017–2022); TIRR Memorial Hermann, Texas TBI Model System at TIRR, 90DPTB0016 (2017–2022).

Supplemental Material

References

Maas

AIR

, Menon

, Adelson

, et al.; InTBIR Participants and Investigators. Traumatic brain injury: Integrated approaches to improve prevention, clinical care, and research. Lancet Neurol, 2017; 16(12):987–1048; doi: 10.1016/S1474-4422(17)30371-X

Peterson

, Zhou

, Thomas

, et al. Traumatic Brain Injury-Related Hospitalizations and Deaths by Age Group, Sex, and Mechanism of Injury: United States 2016/2017. TBI-Surveillance-Report-2016-2017-508.Pdf. 2021.

Dams-O’Connor

, Juengst

, Bogner

, et al. Traumatic brain injury as a chronic disease: Insights from the United States traumatic brain injury model systems research program. Lancet Neurol, 2023; 22(6):517–528; doi: 10.1016/S1474-4422(23)00065-0

Whiteneck

, Eagye

, Cuthbert

, et al. One and Five Year Outcomes after Moderate-to-Severe Traumatic Brain Injury Requiring Inpatient Rehabilitation: Traumatic Brain Injury Report. 2018.

Hammond

, Giacino

, Nakase Richardson

, et al. Disorders of consciousness due to traumatic brain injury: Functional status ten years post-injury. J Neurotrauma, 2019; 36(7):1136–1146; doi: 10.1089/neu.2018.5954

McCrea

, Giacino

, Barber

, et al.; TRACK-TBI Investigators. Functional outcomes over the first year after moderate to severe traumatic brain injury in the prospective, longitudinal TRACK-TBI study. JAMA Neurol, 2021; 78(8):982–992; doi: 10.1001/jamaneurol.2021.2043

Nakase-Richardson

, Whyte

, Giacino

, et al. Longitudinal outcome of patients with disordered consciousness in the NIDRR TBI model systems programs. J Neurotrauma, 2012; 29(1):59–65; doi: 10.1089/neu.2011.1829

Kowalski

, Hammond

, Weintraub

, et al. Recovery of consciousness and functional outcome in moderate and severe traumatic brain injury. JAMA Neurol, 2021; 78(5):548–557; doi: 10.1001/jamaneurol.2021.0084

Hammond

, Perkins

, Corrigan

, et al. Functional change from five to fifteen years after traumatic brain injury. J Neurotrauma, 2021; 38(7):858–869; doi: 10.1089/neu.2020.7287

10.

Whyte

, Nakase-Richardson

, Hammond

, et al. Functional outcomes in traumatic disorders of consciousness: 5-year outcomes from the national institute on disability and rehabilitation research traumatic brain injury model systems. Arch Phys Med Rehabil, 2013; 94(10):1855–1860; doi: 10.1016/j.apmr.2012.10.041

11.

Luauté

, Maucort-Boulch

, Tell

, et al. Long-term outcomes of chronic minimally conscious and vegetative states. Neurology, 2010; 75(3):246–252; doi: 10.1212/WNL.0b013e3181e8e8df

12.

Giacino

, Katz

, Schiff

, et al. Comprehensive systematic review update summary: Disorders of consciousness: Report of the guideline development, dissemination, and implementation subcommittee of the American Academy of Neurology; the American congress of rehabilitation medicine; and the national institute on disability, independent living, and rehabilitation research. Neurology, 2018; 91(10):461–470; doi: 10.1212/WNL.0000000000005928

13.

Fins

. Disorders of consciousness and disordered care: Families, caregivers, and narratives of necessity. Arch Phys Med Rehabil, 2013; 94(10):1934–1939; doi: 10.1016/j.apmr.2012.12.028

14.

Giacino

, Ashwal

, Childs

, et al. The minimally conscious state. Neurology, 2002; 58(3):349–353; doi: 10.1212/WNL.58.3.349

15.

Thibaut

, Bodien

, Laureys

, et al. Minimally conscious state “plus”: diagnostic criteria and relation to functional recovery. J Neurol, 2020; 267(5):1245–1254; doi: 10.1007/s00415-019-09628-y

16.

Bodien

, Barber

, Taylor

, et al. Feasibility and utility of a flexible outcome assessment battery for longitudinal traumatic brain injury research: A TRACK-TBI study. J Neurotrauma, 2023; 40(3–4):337–348; doi: 10.1089/neu.2022.0141

17.

Wilson

JTL

, Pettigrew

LEL

, Teasdale

. Structured interviews for the glasgow outcome scale and the extended Glasgow outcome scale: Guidelines for their use. J Neurotrauma, 1998; 15(8):573–585; doi: 10.1089/neu.1998.15.573

18.

Rappaport

, Hall

, Hopkins

, et al. Disability rating scale for severe head trauma: Coma to community. Arch Phys Med Rehabil, 1982; 63(3):118–123.

19.

McMillan

, Wilson

, Ponsford

, et al. The Glasgow outcome scale - 40 years of application and refinement. Nat Rev Neurol, 2016; 12(8):477–485; doi: 10.1038/nrneurol.2016.89

20.

Snider

, Kowalski

, Hammond

, et al. Comparison of common outcome measures for assessing independence in patients diagnosed with disorders of consciousness: A traumatic brain injury model systems study. J Neurotrauma, 2022; 39(17–18):1222–1230; doi: 10.1089/neu.2022.0076

21.

Giacino

, Kalmar

, Whyte

. The JFK coma recovery scale-revised: Measurement characteristics and diagnostic utility. Arch Phys Med Rehabil, 2004; 85(12):2020–2029; doi: 10.1016/j.apmr.2004.02.033

22.

Sherer

, Nakase-Thompson

, Yablon

, et al. Multidimensional assessment of acute confusion after traumatic brain injury. Arch Phys Med Rehabil, 2005; 86(5):896–904; doi: 10.1016/j.apmr.2004.09.029

23.

Sherer

, Katz

, Bodien

, et al. Post-traumatic confusional state: A case definition and diagnostic criteria. Arch Phys Med Rehabil, 2020; 101(11):2041–2050; doi: 10.1016/j.apmr.2020.06.021

24.

Levin

, O’Donnell

, Grossman

. The Galveston orientation and Amnesia test: A practical scale to assess cognition after head injury. J Nerv Ment Dis, 1979; 167(11):675–684; doi: 10.1097/00005053-197911000-00004

25.

Giacino

, Kezmarsky

, DeLuca

, et al. Monitoring rate of recovery to predict outcome in minimally responsive patients. Arch Phys Med Rehabil, 1991; 72(11):897–901; doi: 10.1016/0003-9993(91)90008-7

26.

Schnakers

, Giacino

, Kalmar

, et al. Does the FOUR score correctly diagnose the vegetative and minimally conscious states? Ann Neurol, 2006; 60(6):744–745; author reply 745; doi: 10.1002/ana.20919

27.

Seel

, Sherer

, Whyte

, et al.; American Congress of Rehabilitation Medicine, Brain Injury-Interdisciplinary Special Interest Group, Disorders of Consciousness Task Force. Assessment scales for disorders of consciousness: Evidence-based recommendations for clinical practice and research. Arch Phys Med Rehabil, 2010; 91(12):1795–1813; doi: 10.1016/j.apmr.2010.07.218

28.

Hicks

, Giacino

, Harrison-Felix

, et al. Progress in developing common data elements for traumatic brain injury research: Version two – the end of the beginning. J Neurotrauma, 2013; 30(22):1852–1861; doi: 10.1089/neu.2013.2938

29.

Martens

, Bodien

, Sheau

, et al. Which behaviours are first to emerge during recovery of consciousness after severe brain injury? Ann Phys Rehabil Med, 2020; 63(4):263–269; doi: 10.1016/j.rehab.2019.10.004

30.

Weaver

, Cogan

, O’Brien

, et al. Determining the hierarchy of coma recovery scale-revised rating scale categories and alignment with aspen consensus criteria for patients with brain injury: A Rasch analysis. J Neurotrauma, 2022; 39(19–20):1417–1428; doi: 10.1089/neu.2022.0095

31.

Kincaid

. Derivation of new readability formulas: (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Research Branch Report. Chief of Naval Technical Training, Naval Air Station Memphis; 1975.

32.

Feinstein

, Cicchetti

. High agreement but low kappa: I. the problems of two paradoxes. J Clin Epidemiol, 1990; 43(6):543–549; doi: 10.1016/0895-4356(90)90158-L

33.

Lin

, Hedayat

, Wu

. Statistical Tools for Measuring Agreement. Springer New York: New York, NY; 2012.; doi: 10.1007/978-1-4614-0562-7

34.

Koo

, Li

. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med, 2016; 15(2):155–163; doi: 10.1016/j.jcm.2016.02.012

35.

Rubin

. Multiple Imputation for Nonresponse in Surveys. 1st ed. Wiley Series in Probability and Statistics. Wiley; 1987; doi: 10.1002/9780470316696

36.

Altman

. Practical Statistics for Medical Research. 2nd ed, Rev. Chapman & Hall/CRC Texts in Statistical Science 12. Chapman & Hall [u.a.]: London; 2020.

37.

Hosmer

, Lemeshow

. Applied Logistic Regression. 1st ed. Wiley; 2000; doi: 10.1002/0471722146

38.

Malec

, Hammond

, Giacino

, et al. Structured interview to improve the reliability and psychometric integrity of the Disability Rating Scale. Arch Phys Med Rehabil, 2012; 93(9):1603–1608; doi: 10.1016/j.apmr.2012.04.003

39.

Anonymous. 2022 National Strategy to Support Family Caregivers. Administration for Community Living; n.d.

40.

Totman

, Lamm

, Goldstein

, et al. Longitudinal trends in severe traumatic brain injury inpatient rehabilitation. J Head Trauma Rehabil, 2023; 38(3):E186–E194; doi: 10.1097/HTR.0000000000000814

41.

Merbitz

, Morris

, Grip

. Ordinal scales and foundations of misinference. Arch Phys Med Rehabil, 1989; 70(4):308–312.

42.

Whyte

, Giacino

, Heinemann

, et al. Brain Injury Functional Outcome Measure (BI-FOM): A single instrument capturing the range of recovery in moderate-severe traumatic brain injury. Arch Phys Med Rehabil, 2021; 102(1):87–96; doi: 10.1016/j.apmr.2020.09.377

43.

Granger

, Divan

, Fiedler

. Functional assessment scales. A study of persons after traumatic brain injury. Am J Phys Med Rehabil, 1995; 74(2):107–113.

44.

Bodien

, Vora

, Barra

, et al. Feasibility and Validity of the Coma Recovery Scale-Revised For Accelerated Standardized Testing (CRSR-FAST): A practical assessment scale for detecting consciousness in the intensive care unit. Arch Phys Med Rehabil, 2024; 105(4):e117; doi: 10.1016/j.apmr.2024.02.330

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.03 MB

0.47 MB

0.00 MB

0.03 MB