Abstract
The EFPT can be used to assess performance of ADLs without the need to use tests from disciplines other than occupational therapy.
Substance addiction is a public health and personal problem that has generated intense research activity. Currently, no unitary concept of the nature of substance addiction exists, given that some believe it is a brain disorder caused by exposure to substances (Volkow et al., 2016), whereas others view it as a set of overlearned habits that cannot be considered a disease (Lewis, 2017). The fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association [APA], 2013) does not address addiction per se; rather, it is limited to classifying substance use disorders. All current neuropsychological models share the idea that a “shutdown” occurs in the prefrontal cortex that eliminates superior control over addictive behavior (Ruiz-Sánchez de León & Pedrero-Pérez, 2019).
The functional and structural changes in the prefrontal cortex that accompany addiction have a crucial effect on occupational performance in daily life (Baum, 2017; Koechlin, 2016). In particular, the executive functions (EFs)—which are associated with the prefrontal cortex—refer to a family of top-down mental processes needed when a person has to concentrate and pay attention, that is, when relying on instinct or intuition would be ill advised, insufficient, or impossible. The general consensus is that three main EFs exist: (1) self-control or inhibitory control, (2) working memory, and (3) cognitive flexibility (Connor & Maeir, 2011). Other, higher order EFs—such as reasoning, problem solving, planning, setting goals and action strategies, self-monitoring, and modifying strategies when goals are not being achieved—are based on these main functions. These skills are essential for mental and physical health; success at school and work; and cognitive, emotional, and social development (Diamond, 2013).
Much work has been done to connect EF deficits with the maintenance of addictive behaviors (Ellis et al., 2016; Pedrero-Pérez et al., 2015). The effects of EF dysfunction on activities of daily living (ADLs) have been studied, especially in relation to cognitive decline and dementia (e.g., Garrett et al., 2019; Pride et al., 2017). Most tests designed to assess EF are neuropsychological in nature (Chan et al., 2008).
Functional cognition is defined as the ability to use and integrate thinking and performance skills to accomplish complex everyday activities (Giles et al., 2017). The performance demands associated with EF can be observed as a person carries out ADLs that require initiation, organization, sequencing, judgment, and completion (Baum, 2017). Neuropsychological assessment of EF takes place from the bottom up—considering each function as an isolated element of the person’s overall functioning. In real life, these functions never operate separately. Overall functioning is the central interest of occupational therapy, the assessments for which are typically formulated with a top-down approach—from real activity to the basic functions into which it can be subdivided (Rojo-Mota et al., 2009). Occupational therapy is concerned with measuring functional cognition as cognitive performance during complex real-life tasks.
The Executive Function Performance Test (EFPT; Baum et al., 2008) is used to assess the performance of daily tasks, the execution of which are methodically analyzed by an occupational therapist with respect to the EFs involved. The test has been widely used to assess people with diverse pathologies and brain disorders, including stroke (Baum et al., 2008), schizophrenia (Katz et al., 2007), and dementia (Kim et al., 2017). No previous work has used this test with a population of people engaging in addictive behaviors.
The aim of this work was to administer the EFPT to a sample of people beginning treatment for substance addiction and determine its internal consistency, structural validity, and convergent and discriminant validity with other occupational and neuropsychological tests. No validated Spanish version of the EFPT exists, so one was created for this study.
Method
Participants and Procedure
A cross-sectional observational study was designed. Recruitment was carried out between January and June 2019. We had previously determined that a sample size of 50 was needed, and 52 people were recruited in case of possible reporting errors. A convenience sample consisting of the first 52 people referred to the Addiction Treatment Centre (CAD) San Blas Occupational Therapy Department for evaluation and intervention were recruited to participate. CAD San Blas is operated by the Institute of Addictions, Madrid Salud, a health care authority under the auspices of the Madrid City Council (Madrid, Spain). It is a free, public outpatient center that specializes in the interdisciplinary treatment of people engaging in addictive behaviors who voluntarily request treatment. After a personal interview, clients are seen within the subsequent week for independent assessment by different professionals (doctor, nurse, social worker, occupational therapist, and psychologist).
Participants who met the criteria for substance abuse or dependence according to the Diagnostic and Statistical Manual of Mental Disorders (4th ed., text revision [DSM–IV–TR]; APA, 2000) were included. As previously stated, the DSM–5 does not address addiction as such. In addition, many countries do not use the DSM–5 for various reasons, instead retaining use of the previous edition. In our case, the DSM–IV–TR was more useful for the classification of patients. Participants were excluded if they had motor or cognitive difficulties that would hinder performance of the EFPT tasks; language difficulties that would prevent a clear understanding of the instructions; any serious psychopathological process (major depression, active psychotic disorder, cognitive decline) at the time of assessment; a history of brain damage of any kind; prescribed medication that could interfere with the understanding or execution of tasks; or narcotic substance use at the time of assessment (as determined by urine toxicology screening and exhaled breath testing). The final sample included participants addicted to heroin (n = 3), cocaine (n = 21), alcohol (n = 19), and cannabis (n = 9). Each participant was informed of the dual objective of the test administration—to design their individual treatment and to conduct research—and signed an informed consent form. The study was authorized by the Madrid Salud Training and Research Unit and Research Committee.
Instruments
Occupational Tools
The EFPT was adapted to the Spanish language through back translation with independent native English speakers.
The EFPT requires the participant to complete four tasks. The simple cooking task consists of preparing oatmeal by following written instructions. The telephone use task involves looking up a local grocery store in the official telephone book, then calling the store and asking whether they deliver groceries. For the medication management task, dosage and distribution instructions are provided along with distractors. For the bill payment task, certain conditions are stipulated, and the participant is required to make decisions with regard to bill payment and maintenance of an account balance. Pretest questions administered before the participant starts the tasks are based on a brief interview included in the test manual and aim to determine whether the participant is familiar with the tasks and whether the participant performs them relatively independently or needs help. This step enables the assessor to determine whether the participant has any deficits in problem awareness after the test is completed.
The four tasks are used to assess three components of EF: (1) initiation (the start of the motor activity that begins a task), (2) execution (consisting of three scales—Organization, Sequencing, Judgment, and Safety—that assess planning, movement, error correction, and safety), and (3) completion (deciding and recognizing when a task is finished). Participants’ performance for each component is rated on a 5-point scale (0 = does not require help, 1 = requires verbal guidance, 2 = requires gestural guidance, 3 = requires verbal direct instruction, 4 = requires physical assistance, 5 = requires the task be done for the participant). Participants’ score on each task reflects the EF capacity observed while they perform the task. Scores range from 0 to 5 for each EF scale and thus from 0 to 25 for each task, and the total score ranges from 0 to 100. A higher score reflects a greater need for help and indicates the most severe EF deficits.
The Allen Cognitive Level Screen–5 (ACLS–5; Allen et al., 2007) classifies participants on the basis of their performance of a task that includes cognitive aspects (understanding and following instructions, error detection and correction) and motor aspects (the actual execution of the task). The assessment kit consists of a prepunched leather rectangle, two types of needle, leather laces, and a shoelace. The assessment allows the clinician to evaluate the person’s performance on three different leather-lacing tasks of varying levels of challenge. To complete the three tasks successfully, the person must pay attention to the verbal instructions and demonstration, understand and use sensory signals from the materials (leather, laces, and needles), and effectively use feedback from the motor actions during practice with the lacing materials. This test has previously been used with participants in treatment for addiction (Rojo-Mota, Pedrero-Pérez, Huertas-Hoyas, et al., 2017).
The Loewenstein Occupational Therapy Cognitive Assessment battery (LOTCA; Katz et al., 1989), a performance test, contains 20 items divided into four areas: Orientation (2 items), Perception (6 items), Visuomotor Organization (7 items), and Thinking Operations (5 items). Each item is scored on a Likert-type scale ranging from 1 to 4 or 1 to 5, according to the standard criteria listed in the test manual (Katz, 1997). The higher the score is, the better the performance. This test has previously been used with participants being treated for addiction (Rojo-Mota, Pedrero-Pérez, Ruiz-Sánchez de León, et al., 2017).
Neuropsychological Tools
The Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005) is a screening test used to assess 10 cognitive domains. The total possible score is 30 points, with a suggested cutoff score of 26. Scores below the cutoff suggest mild cognitive impairment or early dementia. The original Spanish version was validated with Spanish-speaking people with addiction (Rojo-Mota et al., 2013).
Neuropsychological tests known to assess higher order cognition were also administered: the Rey–Osterrieth Complex Figure Test (RCFT; Osterrieth, 1943; Rey, 1941), the Wechsler Memory Scale Fourth Edition Symbol Span test (Wechsler, 2009), the Wechsler Adult Intelligence Scale–Fourth Edition (WAIS–IV; Wechsler, 2008) Digit Span and Cancellation tests, the F-A-S Test for phonemic verbal fluency (Carlesimo et al., 1996), and the Five Digits test (Sedó, 2007; for a description of the tests and the composition of the battery, see Ruiz-Sánchez de León et al., 2011).
Data Analysis
Descriptive statistics were calculated for the EFPT scales and total score. Corrected item–test correlations were used to estimate internal consistency at the item level. To study structural validity, optimal implementation of parallel analysis based on minimum rank factor analysis was done to delimit the number of dimensions to be retained. Closeness to unidimensionality was estimated through values of unidimensional congruence (UniCo; Ferrando & Lorenzo-Seva, 2017); values greater than 0.95 suggest that the data can be treated as essentially unidimensional. A semiconfirmatory factor analysis was also performed on the sample itself. Goodness-of-fit indexes were applied to the solution obtained. Internal consistency was examined with McDonald’s (1999) ω. Zero-order and partial correlation tests were applied to determine convergent and discriminant validity using Bonferroni correction to prevent Type I errors. The same correlation coefficient was used to estimate effect size; .10 to .30 indicates a small effect size; .30 to .50, an average effect size; and ≥.50, a large effect size (Cohen, 1992). Statistical analyses were performed using IBM SPSS Statistics (Version 17.0; IBM Corp., Armonk, NY) and the FACTOR 10.8 program (Lorenzo-Seva & Ferrando, 2013). We then compared our results with those of previous studies with other clinical populations.
Results
Sample Descriptive Statistics
The sample comprised 42 men (80.8%) and 10 women (19.2%), which is consistent with the habitual proportion used for clinical populations in neighboring countries (European Monitoring Centre for Drugs and Drug Addiction, 2018). The mean age of the sample was 39.3 yr (SD = 13.2; men, M = 37.1, SD = 12.9; women, M = 48.5, SD = 10.5). Of the sample, 26.9% had completed elementary school or less (men, 26.2%; women, 30.0%); 30.8, mandatory secondary school (men, 33.2%; women, 20.0%); and 36.5%, advanced secondary school (men, 38.1%; women, 30.0%), and 5.8% had a university education (men, 2.4%; women, 20.0%). Participants’ addictions were as follows: cocaine (total, 40.4%; men, 42.9%; women, 30.0%), alcohol (total, 36.5%; men, 28.6%; women, 70%), cannabis (total, 17.3%; men, 21.4%; women, 0%); and heroin (total, 5.8%; men, 7.1%; women, 0%).
Internal Consistency and Structural Validity
Table 1 shows the correlations between participants’ scores on the EFPT tasks and on the EF scales. As can be seen, the correlations between the EFPT tasks are high, which indicates that they all evaluate different aspects of the same activity. This is also the case with the correlations between the EF scales for each of the tasks.
Correlations Among EFPT Tasks and Scales
Note. All correlations were significant after Bonferroni correction. EFPT = Executive Function Performance Test.
*p < .01. **p < .001.
The EFPT scales made a consistent contribution to the full scale (corrected item–test correlation = .67 < r it < .81). The distribution of the items showed great kurtosis on the Initiation (k 0 = 20.2) and Completion (k 0 = 8.8) scales, thereby rejecting multivariate normality when using Mardia criteria (p < .05). In view of this, we created a polychoric correlation matrix. The criteria for factorial analysis were adequate (Bartlett’s statistic = 127.90, p < .001; Kaiser–Meyer–Olkin test = .76; Bartlett, 1937; Kaiser, 1970). Parallel analysis offered an unequivocally unifactorial solution. The goodness-of-fit estimators for this unifactorial solution were adequate (UniCo = .99, explained common variance = .94, mean of item residual absolute loadings = .19, goodness-of-fit index = .99). Overall, the scale showed excellent internal consistency (ω = .91) and generated few residuals (Kelly’s [1935] criterion; root mean square residual [RMSR] observed = .03; RMSR expected = .14).
Correlational Analysis
Executive Function Performance Test and Occupational Tools
Table 2 shows the correlations observed between the EFPT scales and the LOTCA scales, ACLS–5, and MoCA. Broad correlations were observed for the EFPT execution component and LOTCA scores, with a moderate effect size in some cases. Almost all correlations were negative, reflecting that higher scores on the EFPT scales indicate worse performance, whereas the reverse is true for the LOTCA. The magnitude of the partial correlations controlling for sex, age, and education level showed very little variation. Scores obtained on the ACLS–5 significantly correlated with the EFPT Judgment and Safety scale and especially with the Sequencing scale (see Table 2). The magnitude of the correlations barely changed when controlling for sex, age, and education level. Scores on the EFPT scales also showed moderate correlations with the MoCA, and their magnitude also did not change when controlling for sex, age and education level.
Correlations Between the EFPT Scales and the LOTCA Scales, ACLS–5, and MoCA
Note. Partial correlations controlling for sex, age, and educational level are in parentheses; significant correlations after Bonferroni correction are in bold. ACLS–5 = Allen Cognitive Level Screen–5; EFPT = Executive Function Performance Test; LOTCA = Loewenstein Occupational Therapy Cognitive Assessment; MoCA = Montreal Cognitive Assessment.
*p < .05. **p < .01. ***p < .001.
Executive Function Performance Test and Neuropsychological Tests
Table 3 shows the correlations between the EFPT scales and the neuropsychological tests. The most consistent correlations with a moderate effect size were found for the Five Digits test. This was also found, sometimes with a high effect size, for the EF components evaluated with other tests. For example, the highest EFPT scores (reflecting poor execution) strongly and positively correlated with RCFT Copy and Copy time. This was also found for the neuropsychological tests that measure EF components such as working memory and flexibility.
Correlations Between EFPT Scales and Neuropsychological Tests
Note. Significant correlations after Bonferroni correction are in bold. EF = executive function; EFPT = Executive Function Performance Test; RCFT = Rey–Osterrieth Complex Figure Test; WAIS–IV = Wechsler Adult Intelligence Scale; WMS–IV = Wechsler Memory Scale Fourth Edition.
*p < .05. **p < .01. ***p < .001.
Distribution of the Total Executive Function Performance Test Score
Table 4 shows a comparison of the mean and dispersion scores obtained in this study and those obtained in prior studies with clinical populations with other health problems and healthy control participants.
Comparison of Mean Scores Obtained in Different Studies That Used the EFPT
Note. SDs are in parentheses. EFPT = Executive Function Performance Test.
Discussion
We administered the Spanish version of the EFPT to a sample of people in treatment for substance addiction and determined that it has adequate internal consistency. On the basis of the polychoric correlation matrix, we obtained internal consistency indicators >.90, as have studies with other populations (Baum et al., 2008; Raad & Moore, 2012). The measurement scales applied to the proposed tasks had moderate or high correlations with each other, thus confirming that they measure related aspects. The semiconfirmatory factor analysis also supported the unidimensional nature of the scale with adequate fit indicators for the unifactorial solution.
We examined correlations between the scores on the EFPT tasks and scales and scores on the other occupational and neuropsychological tests to determine the construct validity of the Spanish version of the EFPT. We found significant correlations between the LOTCA and the EFPT scales, which confirm a close relationship between the LOTCA and the EFPT, although each uses different methods to measure similar constructs. After limiting Type I error, the only significant correlation between the EFPT and the ACLS–5 was that with the EFPT Sequencing scale. We expected the relationship between the EFPT scales and the ACLS–5 to be stronger; however, the ACLS–5 has very particular characteristics. Our results are likely a result of the ACLS–5’s strong motor component; the EFPT does not include a motor component. The MoCA classifies participants as having normal or deficient cognitive performance on the basis of a preestablished cutoff score. We found a strong correlation between the MoCA and the three EFPT components.
The construct of EF comes from neuropsychology and refers to a family of cognitive processes that are necessary for behavior control (Diamond, 2013). Therefore, to affirm that the EFPT measures what it says it does, it should have a significant correlation and a certain effect size with tests that have been proven useful in measuring the construct in neuropsychology. The set of neuropsychological assessments administered in this study included the Five Digits test, which measures EF by means of a Stroop-type task (Sedó, 2007). This study’s results show a broad and consistent correlational pattern between the EFPT and the Five Digits test. The magnitude of the correlations is between moderate and high in almost all cases and occurs with both the most automatic of processes (Reading and Counting) and those requiring a greater use of cognitive resources (Choosing and Alternating). After Bonferroni correction, the EFPT Sequencing scale was most highly related to the Five Digit test scores. In other words, both tests require adequate attention to the coordination and ordering of the steps involved in a task.
We also observed correlations of great magnitude between the EFPT and Copy time on the RCFT. Copying requires visual–spatial and motor organization to reproduce the different elements of an image. However, the complexity is greatly increased when working memory must be used. Poor performance (high scores) on the EFPT correlated in almost all cases with longer time to recall and copy the RCFT figure. We also found a relationship between these tasks and the EFPT’s Initiation and Completion scales, with a high magnitude of effect for Immediate Recall, meaning there is a direct relationship between starting and stopping the motor activity and the time taken to finish copying either directly or from working memory.
Verbal fluency is an EF defined as the skill used to evoke appropriate responses to a certain stimulant in a specific amount of time. It requires the use of other cognitive processes, such as processing speed, vocabulary growth, semantic memory, working memory, inhibition, and sustained attention. It is also consistently related to the functions measured by the EFPT. We found no relationship between the EFPT and sustained attention as measured with the WAIS–IV Cancellation task, perhaps because it is substantially automatic and does not require any additional processes.
We also examined partial correlations for all measures, controlling for variables that affect the relationships studied. Nonetheless, controlling for sex, age, and education level did not provide results that were different from those obtained with the zero-order correlations, meaning these variables are not involved in the relationships between the tests.
To summarize, the EFPT and, in particular, the EFPT scales show convergent validity with the neuropsychological tests that evaluate complex functions, yet less so when more isolated functions are measured. This result is consistent with the fact that the EFPT actually evaluates the combined use of all EFs. When performing a complex task, participants must use their EF skills to achieve the preset goal. This is the sole purpose of the EFPT and similar tests, such as the Assessment of Motor and Process Skills (Rojo-Mota et al., 2014): to measure occupational performance of ADLs from a brain function perspective. Some researchers see the need to assess occupational performance with tests that are strictly occupational, given the quality of performance in many cases and, at best, a secondary goal of assessing the effectiveness of interventions (Radomski et al., 2016). Tests such as the EFPT are essential when the decision-making process and other EFs are closely related to maintenance, as is the case with addiction behaviors (Redish et al., 2008). The EFPT allows therapists to plan interventions with an awareness of the participant’s skill level, autonomy in the performance of ADLs, and supports needed for any such plan (Rojo-Mota, Pedrero-Pérez, & Huertas-Hoyas, 2017).
When we compared the scores obtained by participants in this study with those of people with other problems, we noted that our participants scored significantly higher than healthy participants, which clearly reflects worse EF performance. Their scores were also worse than those among people with multiple sclerosis, for whom cognitive problems are more rare and motor impairments, which are not assessed by the EFPT, are predominant. However, scores obtained in the sample of people with addictions were better than those observed among homeless participants, in whom other mental health problems and psychosocial stress are added to substance abuse. The observed scores are also lower than those obtained by people with acute schizophrenia and quite lower than those obtained by people with chronic schizophrenia. Addiction would be ranked in the middle of these problems with respect to cognitive deficits, yet it has a critical difference: The deficits resulting from addiction, in most cases, are quickly reversible, both structurally and functionally, when consumption stops (Parvaz et al., 2017). Occupational therapy can play an immensely important role in formulating and implementing functional cognitive rehabilitation programs.
This study has some limitations. Convenience sampling was used to guarantee at least 50 participants, given the complex nature of the assessment protocol, which required two assessment sessions in a short time frame. The characteristics of the treatment center prevented excessive dedication to this dual parallel assessment process (occupational and neuropsychological) because of the risk of hindering the necessary care process. Because the study is correlational, the results cannot be generalized to the population of people with addictive behaviors and must be viewed as clinical validity indices, requiring new studies with a higher degree of representativity.
Implications for Occupational Therapy Practice
The results of this study have the following implication for occupational therapy practice:
The EFPT allows the evaluation of executive performance in daily life tasks, that is, of functional cognition.
The EFPT is a reliable and valid test for use with people being treated for addiction. Its use eliminates dependence on neuropsychological tests and provides an occupational evaluation with a greater degree of ecological validity.
Conclusion
The Spanish version of the EFPT is a useful test for measuring occupational performance of ADLs among people in treatment for addiction. Its use does not require much investment (it is available from the authors free of charge), an excessive amount of time, complex spatial conditions, or extensive professional training. This study found evidence of its internal consistency, construct validity, and convergent and discriminant validity with occupational and neuropsychological tests. Moreover, the EFPT complies with the scientific requirement to measure occupational performance from a strictly occupational perspective in order to avoid having to resort to tests from other disciplines and to place functional cognition as the central variable of occupational work. Future studies should examine the EFPT’s clinical validity in determining the effectiveness of occupational interventions for the population of people in treatment for addictive behaviors.
Footnotes
Acknowledgments
The study was conducted at the CAD San Blas, Institute of Addictions of Madrid Salud (Madrid, Spain). The study did not receive funding of any kind, and the authors have no conflict of interest to declare.