Abstract
The Functional Loewenstein Occupational Therapy Cognitive Assessment (FLOTCA) was developed to assess integrative higher cognitive abilities in people with traumatic brain injury (TBI). The FLOTCA measures performance on three tasks: navigating on a map, organizing a toolbox, and planning a daily schedule. This study assessed the psychometric properties of the FLOTCA with a sample of 25 participants with TBI ages 18â49 and 25 matched healthy participants. The FLOTCA showed high interrater reliability (intraclass correlation = .996) and internal consistency reliability for the total score (α = .82). Construct validity was supported for the total score, tâ(48) = â5.48, d = 1.52, and the separate tasks. Moderate ecological validity was obtained with the combined FIMâą and Functional Assessment Measure, râ(19) = .44, p < .05. The results indicate that the FLOTCA can be used to assess higher cognitive abilities in functioning and can serve as the basis for intervention planning.
During the past decade, the focus of rehabilitation medicine has shifted to occupational performance and participation in life roles. The International Classification of Functioning, Disability and Health (ICF;âWorld Health Organization, 2001) emphasizes the concepts of participation and quality of life as the most important outcomes of health conditions. According to the ICF, health is influenced by the interaction between body functions and structures and the domains of activity and participation, which in turn are influenced by personal and environmental factors.
Traumatic brain injury (TBI) is a major public health problem that causes death and disability in young and productive people (Shukla, Devi, & Agrawal, 2011). In those who survive, the resulting cognitive and neuropsychiatric impairments are often long lasting (Ham & Sharp, 2012). Cognitive impairments have the greatest influence on occupation and participation after TBI (Devitt et al., 2006). Such impairments are usually persistent, and 95% of post-TBI rehabilitation resources are invested in cognitive rehabilitation (Mazmanian, Kreutzer, Devany, & Martin, 1993).
Researchers have found that a variety of cognitive and metacognitive disabilities associated with TBI cause functional limitations (Cicerone et al., 2000). The most common cognitive deficits found in the acute stage after TBI are in the domains of attention, memory, processing speed, cognitive flexibility, and mental endurance (de Guise, Feyz, LeBlanc, Richard, & Lamoureux, 2005). Strong associations have been found between executive functions and independent living, employment, and leisure activities (Finnanger et al., 2013). Several studies, at various time intervals postinjury, have concluded that cognitive dysfunction mediates functional problems (Skandsen et al., 2010). Cognitive deficits have been shown to decrease the efficiency of daily functioning and to cause difficulties in adaptation to new situations and in problem solving (Cicerone et al., 2000). In addition, cognitive impairments are often hidden and are not always understood by family, friends, teachers, or employers. Such dysfunction may affect participation and cause problems in independent living and social relationships (Finnanger et al., 2013).
Cognitive skills have a major influence on the rehabilitation outcomes of clients with brain injury (Averbuch & Katz, 2011; Toglia, 2011). According to Cicerone et al. (2011), cognitive rehabilitation is effective in helping clients compensate for residual cognitive limitations and improve underlying cognitive functions. Rehabilitation efforts should be directed at achieving changes that improve function in areas relevant to everyday life (Cicerone et al., 2011). Given the crucial role of cognitive skills, the first phase of occupational therapy intervention is assessment, which should encompass the clientâs occupational performance, cognitive abilities and disabilities, level of awareness, and executive functioning abilities (Averbuch & Katz, 2011).
Hartman-Maeir, Katz, and Baum (2009) suggested a model for assessing a client with suspected cognitive disabilities. The Cognitive Functional Evaluation comprises six stages of evaluation; the third stage includes assessments that examine the functional impact of cognitive deficits and potential deficits in higher order cognition. The Functional Loewenstein Occupational Therapy Cognitive Assessment (FLOTCA; Schwartz, Sagiv, Katz, & Averbuch, 2013), which assesses higher level cognitive abilities in various functional tasks, should be added to this stage because few comprehensive assessments do what the FLOTCA does in a short time.
As part of testing the functional impact of cognitive disabilities, ecological validity is of paramount importance. Ecological validity addresses the functional and predictive relationship between a personâs performance on an assessment and his or her behavior in the real world (Burgess et al., 2006). The ecological validity of cognitive tests has become increasingly important in determining whether an injury has affected a personâs ability to manage his or her affairs, live independently, or return to work (Sbordone, 1996). Burgess et al. (2006) emphasized the importance of assessing clientsâ cognitive abilities while they engage in activities that reflect the abilities they need to perform daily activities.
The purpose of the FLOTCA is to assess integrative cognitive abilities through performance of tasks that require the person to perform multiple steps in sequence and to deal with new and unfamiliar requirements (e.g., planning the order of putting items in a cabinet, checking the time left before needing to move to another task). This kind of assessment may more accurately reflect a personâs functioning in daily activities that require completion of several coordinated steps.
The aim of this study was to assess the psychometric properties, including reliability and validity, of the FLOTCA with a sample of people with TBI compared with matched healthy control participants. The main research question was, Can the FLOTCA accurately differentiate between participants with TBI and healthy control participants, and do results positively correlate with a daily functioning measure?
Method
Participants
Participants included 25 clients (mean [M] age = 25.12 yr, standard deviation [SD] = 7.22) from the TBI inpatient and outpatient units at the Loewenstein Rehabilitation Hospital who were consecutively entered into the study when accepted to the units. Inclusion criteria for all participants were baseline score of 4 or 5 on each item of the Dynamic Loewenstein Occupational Therapy Cognitive Assessment (DLOTCA; Katz, Bar-Haim Erez, Livni, & Averbuch, 2012), age 18â50 yr, â„8 yr of education, and ability to read and write in Hebrew. The DLOTCA addresses 20 items in five areasâ(1) visual perception, (2) spatial perception, (3) praxis, (4) visuomotor organization, and (5) thinking operationsâeach scored on a 5-point scale. This assessment was used to screen participants; only those who scored 4 or 5 on all items were included because we were looking for participants with higher level functioning. A score below 4 on the DLOTCA suggests that the personâs basic cognitive abilities are low and thus that he or she would not be able to perform the higher integrative abilities needed for the FLOTCA.
Glasgow Coma Scale (GCS; Teasdale et al., 2014) scores and number of days unconscious were recorded to reflect injury severity; these indicators are used to denote severity in most TBI units. Clients with any previous neurological or psychiatric illness and those with aphasia or severe visual or hearing impairment were excluded. Twenty-five healthy control participants (M age = 25.28, SD = 6.16) who volunteered for the study were matched with hospitalized clients according to age, gender, and years of education. Both groups included 19 men and 6 women. Table 1 lists participant characteristics.
Participant Demographics
Note. GCS = Glasgow Coma Scale; M = mean, NA = not applicable; SD = standard deviation; TBI = traumatic brain injury.
Measures
Functional Loewenstein Occupational Therapy Cognitive Assessment.
The FLOTCA is used to assess integrative cognitive abilities expressed in task performance. Its goal is to identify deficits in the performance of the following cognitive functions: ability to follow directions, spatial organization and planning, time organization, sequence building, priority setting, divergent thinking, and decision making. The evaluation consists of three tasks: (1) planning a route and navigating on a map, (2) organizing tools in a toolbox, and (3) planning a daily schedule according to a list of activities.
The first task, planning a route and navigating on a map, is preceded by a training component to prepare the respondent to follow the instructions. The task is divided into two parts. In Part 1, the evaluator reads the directions to the respondent one at a time and asks the respondent to draw the correct path on the map. Directions include instructions, such as âturn rightâ and âtake the third exit in the traffic circle,â resembling those provided by a global positioning system device. In Part 2, the evaluator asks the respondent to plan the fastest route from Point A to Point B on a different map, given a different list of street names. A detailed manual specifies how to code the respondentâs performanceâfor example, whether he or she found the starting point and drove in the correct direction, number of times the instructions were repeated, and time to completion.
In the second task, organizing tools in a toolbox, the respondent is presented with an empty toolbox and several tools scattered on the table (e.g., a screwdriver, screws, lightbulbs). The respondent is instructed to safely organize all the tools so that the box closes with everything in it. Scoring addresses issues such as whether small parts were separated, whether delicate pieces were safely stored, and time to task completion.
The third task is planning a daily schedule according to a list of activities and the business hours of several locations to be visited. The respondent is asked to organize the tasks into a reasonable and doable schedule, specifying the exact time for each task without overlooking any of them. Scoring takes into account the respondentâs reasoning and ability to set priorities.
Administration of the entire FLOTCA battery takes 30â60 min. A detailed manual describes how to administer and score the assessment. Examples of items scored are number of mistakes that needed correction, number of times the evaluator repeated directions, whether the respondent paid attention to the direction of traffic, whether the toolbox closed, time to completion, and whether the respondent planned enough time for each task and took the business hours into account (Averbuch & Katz, 2011; Sagiv, 2009; Schwartz, 2009; Schwartz et al., 2013). Each item is scored 0, 1, or 2 points; a higher score represents better performance. The items of each task are then averaged. Thus, the different number of items in each task is controlled.
As part of the development of the FLOTCA, content validity was determined by expert opinion as to whether the instrument, including all three tasks, covers the concept it purports to measure. We introduced the assessment to 24 occupational therapists with 3â34 yr of experience in the cognitive rehabilitation field from several rehabilitation centers throughout Israel and asked them to determine which cognitive components were represented in at least one of the evaluation tasks and to rate each task as involving primarily spatial components or abstract thinking components. The results of this survey supported the instrument construction and ensured that each component was represented in at least one of the assessment tasks. The experts pointed out that the skills of attention, executive functions, initiation, and planning are required in all tasks, just as they are required in daily functions.
A one-way analysis of variance with post hoc ScheffĂ© multiple comparisons and homogeneous subsets was performed using the occupational therapistsâ ratings of each task as involving either spatial components or abstract thinking components. The experts rated navigating on a map (M = 2.5) and organizing a toolbox (M = 2.5) as higher in spatial components than planning a daily schedule (M = 1.4), Fâ(2, 69) = 66.23, p < .01. Likewise, they rated planning a daily schedule (M = 2.8) as higher in abstract thinking components compared with navigating on a map (M = 1.9) and organizing a toolbox (M = 2.1), Fâ(2, 69) = 30.71, p < .01.
FIMâą and Functional Assessment Measure.
The FIM (Uniform Data System for Medical Rehabilitation, 1997) and the Functional Assessment Measure (FAM; developed by clinicians at the Santa Clara Valley Medical Center, San Jose, CA) are used in combination (FIM+FAM) to assess the degree of disability in daily function of people with brain injury (Corrigan, Smith-Knapp, & Granger, 1997; Hawley, Taylor, Hellawell, & Pentland, 1999). The combined measure consists of 30 items, 18 from the FIM and 12 from the FAM that were added to enhance the sensitivity of the measure for the TBI population. The FIM+FAM assesses motor function in self-care (e.g., sphincter control, mobility and locomotion) and cognitive abilities, including communication, psychosocial adjustment, and cognitive function. The cognitive abilities section includes 14 items; only these items were used in this study because of the FLOTCAâs focus on cognitive abilities and the desire to avoid bias attributable to the motor limitations of some participants. Each item is scored from 1 = total assistance required to complete task to 7 = complete independence (Nichol et al., 2011; Wright, 2000).
Procedure
All participants in both groups underwent assessment with the FLOTCA. The assessment was administered in a quiet room by one of two experienced occupational therapists (authors Schwartz and Sagiv) who took part in developing the FLOTCA as the other therapist observed; the administering therapists scored the tests independently. For the hospitalized participants, we used the cognitive and behavioral scores from the FIM+FAM as reported during staff meetings. The ethics committee of the Loewenstein Rehabilitation Hospital approved this study and waived written consent.
Data Analysis
The study design was a two-group comparison. Data were analyzed using IBM SPSS Statistics (Version 17; IBM Corp., Armonk, NY). The data were normally distributed on all the outcome variables; hence, all variables were analyzed with parametric statistics. The analysis included descriptive statistics (means and standard deviations) of all variables. Reliability of the FLOTCA was tested in three ways: intraclass correlation (ICC) to test interrater reliability, Cronbachâs α to test internal consistency reliability of the items, and an exploratory factor analysis to test the structure of the instrument. Next, Pearson correlations were performed to test the relationships of the FLOTCA item scores with FIM+FAM scores and with injury severity. Finally, to compare the study groups, two-sample t tests with confidence intervals were performed.
Results
To evaluate the relationships between the injury severity variables and total FLOTCA scores in the hospitalized participant group, a Pearson correlation was performed. A significant positive correlation between GCS score and total FLOTCA score was found, râ(17) = .63, p < .01, as was a significant negative correlation between duration of unconsciousness and total FLOTCA score, râ(12) = â.69, p < .01. Correlations with the demographic variables of gender, age, and years of education were not significant.
Reliability
Interrater reliability was very high (ICC = .996, p < .000). Internal consistency reliability with Cronbachâs α was calculated for each task separately and for the total score. The tasks of navigating on a map and planning a daily schedule were found to have acceptable internal consistency, with αs of .75 and .78, respectively. The task of organizing a toolbox showed low internal consistency (α = .48); the toolbox included only three scored items. The total score had good internal consistency (α = .82). All items for each task had positive correlations with the total score.
Exploratory Factor Analysis
To further validate the construction of the instrument into spatial and abstract thinking components, an exploratory factor analysis was performed (although the sample was small). Factor 1 comprised the tasks of organizing a toolbox and planning a daily schedule, and Factor 2 consisted mainly of the task of navigating on a map. The first factor explained 26% of the variance and the second 12%; the two factors together explained 38% of the variance (Table 2).
Exploratory Factor Analysis
Note. Factor 1 explains 26% of the variance and Factor 2, 12%; both together explain 38% of the variance. Empty cells indicate that the task component did not load on the factor.
Moderate significance (>.400).
Construct Validity: Comparison Between Groups
Two-sample t tests and effect sizes (Cohenâs d) were used to compare results between groups on the FLOTCA. Significant differences were found between the groups for total score, t(48) = â5.48, d = 1.52, and for scores on each task: navigating on a map, t = â2.60, d = 0.76; organizing a toolbox, tâ(48) = â3.16, d = 0.89; and planning a daily schedule, tâ(48) = 5.22, d = 1.49. All comparisons were significant at p < .01 (Table 3).
FLOTCA Scores: Group Comparison
Note. FLOTCA = Functional Loewenstein Occupational Therapy Cognitive Assessment; M = mean; SD = standard deviation; TBI = traumatic brain injury.
p < .01.
Ecological Validity
Within the TBI group, we examined relationships between FLOTCA scores and the cognitive scores on the FIM+FAM. We found significant moderate correlations between total FLOTCA score and total FIM+FAM cognitive abilities score, râ(19) = .44, p < .05; between navigating on a map and the attention and safety management items and total cognitive abilities score on the FIM+FAM, râ(19) = .49, p < .05; and between planning a daily schedule and the writing, safety management, comprehension, and memory items and total cognitive abilities score on the FIM+FAM, r(19) = .49â.63, p < .05.
Discussion
The results of this study support the use of the FLOTCA to assess cognitive functional skills. The cognitive capabilities measured by the FLOTCA go beyond the basic abilities of spatial components and abstract thinking to include executive functions and cognitive abilities expressed in task performance. The toolbox task involves sorting objects by shape, size, and use, and the experts considered this task also to include thinking operations such as categorization and flexibility in thinking. The daily schedule task requires estimating time and understanding the sequence of events, which are based on sorting, sequencing, and flexibility in thinking. Navigating on a map requires attention, spatial perception, sense of direction, spatial memory, and ability to reevaluate when making mistakes.
Because of the complexity of analyzing the cognitive components of functional tasks, we performed an exploratory factor analysis. The sample was small, so the results should be interpreted as a tendency that needs further assessment. The toolbox and daily schedule tasks loaded on one factor, whereas the map task for the most part constituted the second factor. The connection between the first two tasks might be a result of the organizational abilities needed in both. In addition, on both tasks, trial and error was allowed because it typically occurs during task performance, and corrected mistakes were not reflected in scoring. However, in the map task, trial and error mistakes and the need for repeated instructions were reflected in the task score; this task involves remembering and manipulating the information when needed.
The findings of the current study provide support for the discriminant validity of the FLOTCA; the scores distinguished between the hospitalized participants with TBI and healthy control participants. Participants with TBI achieved significantly lower total and task scores on the FLOTCA than the control group. These findings are consistent with the literature regarding the impact of TBI on cognitive function (Mazmanian et al., 1993; Toglia, 2011). Moreover, the variables indicating severity of injury (GCS score and duration of unconsciousness) were more strongly correlated with FLOTCA scores than the demographic variables, a finding consistent with findings of a negative relationship between severity of head injury and cognitive function (Masson et al., 1997). The impact of injury severity on FLOTCA scores is important to take into account when interpreting results of the FLOTCA.
Ecological Validity
A significant relationship was found between the FLOTCA total score and the total cognitive abilities score of the FIM+FAM in hospitalized participants. Because FIM+FAM scores reflect ability to function in everyday life (Corrigan et al., 1997), this finding supports the ecological validity of the FLOTCA. FIM+FAM items reflect everyday function but are based on therapist observation, whereas the FLOTCA enables a self-reported assessment of the personâs function.
The FAM item safety judgment correlated with the map and daily schedule tasks; this item reflects abilities such as orientation to oneâs situation, awareness of oneâs deficits and their implications, ability to plan ahead, ability to recognize potential danger and identify risks, and freedom from impulsivity (Wright, 2000). As Vanderploeg, Belanger, Duchnick, and Curtiss (2007) noted, lack of awareness among people with cognitive impairments might cause a safety risk. This item may correlate with ability to perform the map and daily schedule tasks because these tasks involve exercising discretion, which influences personal safety. The attention item on the FAM correlated with the map task; attentional abilities are required while engaged in various occupations (Averbuch & Katz, 2011). The task of navigating on a map requires the person to be fully aware of the different instructions and to be able to perform them in real time.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The FLOTCA is a useful tool that can contribute to the overall occupational therapy evaluation and assist in setting relevant therapeutic goals for treatment.
The FLOTCA is based on everyday functions; therefore, clients can relate the different tasks to their daily activities. Difficulties in performing the tasks may increase clientsâ awareness of their functional problems.
Years of education has no influence on performance on the FLOTCA, so the tool can be used with people who have various levels of education.
Limitations and Future Directions
The study sample of 25 participants in each group is small, but it allowed us to explore initial discriminant validity between the participants with TBI and healthy participants. However, not all items on the FLOTCA significantly discriminated between the groups. It is important to continue to explore this assessment in a larger sample and with a variety of populations beyond people with TBI. In addition, it is important to further validate the FLOTCA with other tools that assess integrative and cognitive abilities and function.
Conclusion
The FLOTCA is a complex, integrative, cognitive assessment with high internal consistency reliability and ecological validity reflecting functioning in daily activities and supported by a moderate correlation with the cognitive abilities items of the FIM+FAM. Occupational therapy practitioners and their clients can benefit from use of the FLOTCA in evaluation of cognitive deficits during performance of complex functional activities. Future research should test use of the FLOTCA with people with acquired brain injuries and mild cognitive impairment and with other populations experiencing higher level cognitive decline.
Footnotes
Acknowledgments
We thank all the occupational therapists at the Loewenstein Rehabilitation Hospital and other rehabilitation centers in Israel who contributed to the development of the FLOTCA. We also thank all the participants who took part in this study.