Abstract
This study highlights the convergent validity, discriminative validity, and test–retest reliability of two tasks, stitching and copying, in the ACLS–6 for community-dwelling people living with schizophrenia.
Schizophrenia is a serious mental illness that affects 4.0% of people throughout their life (Saha et al., 2005). Impaired cognitive functions, including attention, memory, and executive functions, are a core feature of schizophrenia (Lin et al., 2019; Oertel et al., 2019; Thai et al., 2019). Cognitive impairment has negative impacts on functional outcomes such as activities of daily living, work, and social interaction in people living with schizophrenia (Rajji et al., 2014), and cognitive functions can be predictors of functional performance (Higuchi et al., 2017; Wright et al., 2020). Therefore, assessment of the cognitive functions of people living with schizophrenia is critical in order for clinicians and researchers to understand their cognitive status and select appropriate interventions.
The Allen Cognitive Level Screen (ACLS) is a measure of cognitive functions that is commonly used with people living with schizophrenia (Stewart et al., 2019; Su et al., 2011; Yeung & Chan, 2006). Occupational therapy practitioners use the ACLS stitching task to assess a client’s cognitive functions by observing their visuomotor performance. The ACLS was developed on the basis of the Allen Cognitive Disabilities Model, which consists of six levels of functional cognition (Allen, 1985, 2000): (1) automatic actions (e.g., attention to external stimuli), (2) postural actions (e.g., gross motor responses to proprioceptive stimuli), (3) manual actions (e.g., performance of daily grooming tasks), (4) goal-directed actions (e.g., meal preparation), (5) exploratory actions (e.g., use of trial and error to solve a problem), and (6) planned actions (e.g., anticipation and prevention of errors; Kang & Tadi, 2021).
The Allen Cognitive Level Screen–Sixth Edition (ACLS–6; Allen, 2016) is the newest version of the ACLS. Previous editions contained only the stitching task, accomplished using a leather board and a leather lace in one color. The ACLS–6 retains the stitching task with new materials and has eight new tasks: sensing, moving, simple coloring, frog–flower–pot, diamond coloring, copying, peanut butter, and stenciling. The stitching materials in the ACLS–6 are a white plastic board and a plastic lace in two colors that allow examinees to observe errors (e.g., a twist in the lace).
Convergent validity, discriminative validity, and test–retest reliability are essential psychometric properties of an observational measure. Convergent validity is the extent to which constructs that theoretically should be associated are actually related (Chiu et al., 2019). Regarding the convergent validity of the ACLS–6, we had two hypotheses: (1) the copying task is strongly correlated with the stitching task, and (2) the two tasks are moderately correlated with three other cognitive measures. Discriminative validity is an instrument’s ability to distinguish different degrees of a condition—in this case, severity of schizophrenia symptoms (Chiu et al., 2018). We hypothesized that the two tasks could distinguish people living with schizophrenia with marginal symptoms and those with mild clinical symptoms. Test–retest reliability is the level of stability of the data recorded in assessment sessions (Chiu et al., 2020). The good test–retest reliability of the two tasks was assumed.
We are unaware of any research on the psychometric properties of the ACLS–6. Therefore, to establish empirical evidence for its psychometric properties, we examined the convergent validity, discriminative validity, and test–retest reliability of two tasks, stitching and copying, for community-dwelling people living with schizophrenia.
Method
Participants
We recruited a convenience sample of people with chronic schizophrenia from the day hospital of a psychiatric center in northern Taiwan between August 2018 and April 2019. Participants met four inclusion criteria: (1) diagnosis of schizophrenia based on criteria in the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; American Psychiatric Association, 2013), (2) ages 20–65 yr, (3) stable psychiatric symptoms, and (4) willingness to sign an informed consent form. We excluded potential participants who had evidence of substance-related disorders, intellectual disability, dementia, or other neurocognitive diagnoses; who obtained a different Clinical Global Impression– Severity (CGI–S) score after a 2-wk interval, indicating unstable symptoms; and who were unable to complete the ACLS–6 because of upper limb or visual impairments. We used a 2-wk interval for test–retest reliability because the stability of psychopathological traits lasts approximately 2–4 wk (Reichenberg et al., 2005). This study was approved by the ethical committees of the local hospital.
A sample size of 104 was estimated as needed for an intraclass correlation coefficient (ICC) of .70 on two assessments at a significance level of .05 (Bonett, 2002). Thus, we recruited more than 104 participants.
Procedure
A repeated-measures design was used for this study. Occupational therapists from the psychiatric center screened potential participants according to the inclusion and exclusion criteria and referred eligible participants to our two research assistants (Li-Yu Chiu and one other person). The research assistants obtained inform consent from the participants and administered the assessments. The research assistants had received at least 6 hr of training to familiarize them with the administration of the measures (e.g., instructions, items, scoring) and to allow them to practice administering the measures to at least two people living with schizophrenia. With expert consultation, we forward and backward translated the task instructions.
In the first assessment session, participants completed the ACLS–6 stitching and copying tasks, the Mini-Mental State Examination (MMSE), the Montreal Cognitive Assessment (MoCA), and the Wisconsin Card Sorting Test (WCST). After 2 wk, participants again completed the two ACLS–6 tasks. All assessments were conducted in a quiet environment to prevent distractions. We gathered participants’ demographic data from their medical records.
Measures
Allen Cognitive Level Screen–Sixth Edition
The ACLS–6 includes one quick observation and nine tasks. The quick observation is a screening tool used to score examinees’ appearance as follows: 1 = examinee is in bed with head support, 2 = examinee sits and holds his or her head up, 3 = examinee uses hands beyond eating finger food, 4 = examinee is dressed but disheveled, and 5 = examinee talks and looks almost presentable. The quick observation takes about 1 min.
Depending on the quick observation score, the practitioner then administers one or two tasks that match the quick observation score (Allen, 2016). The nine tasks were scored according to their criteria. For example, for the copying task, the examinee places ceramic pieces in a line along the edge of a rectangle to form a rectangle, which is scored 3.6. The nine tasks are as follows (matching score ranges are in parentheses): Sensing—tracking moving visual and auditory cues (1.0–1.8) Moving—displaying head and trunk control, arm movement, and ambulation (1.8–3.0) Simple coloring—coloring a sun, tree, and house based on an example of a completed task (3.0–3.8) Stitching—copying three different stitching techniques (3.0–5.8) Frog–flower–pot—putting one frog, eight hearts, one rectangle, and one pot together based on an example (3.6–4.6) Diamond coloring—coloring diamonds based on an example (4.6–5.0) Copying—placing different-colored ceramic pieces based on an example (3.4–5.4) Peanut butter—wiping peanut butter off of foam (5.0–5.2) Stenciling—making a stenciled card (5.0–5.8).
The ACLS has 25 serial classes, in which scores range from 1.0 to 5.8 in intervals of 0.2. Higher scores indicate better cognitive functions (Allen, 2016). Scores correspond to the amount of care a person requires. For instance, a person with a score of 3.4 needs 24-hr nursing care, whereas a person with a score of 4.4 is able to live alone with a daily check (Stewart et al., 2019). The ACLS score helps health care professionals judge a person’s cognitive abilities. In this study, we examined the stitching and copying tasks for two reasons. First, our participants lived in the community and thus could not have scores <3.0. Second, the score range for the copying task is similar to that of the original stitching task and wider than the score ranges for the other new tasks.
Previous editions of the ACLS showed satisfactory psychometric properties: good discriminative validity in people living with mental illness for the first edition (Josman & Katz, 1991), good convergent validity in people living with schizophrenia (Secrest et al., 2000) and good discriminative validity and acceptable test–retest reliability in people living with mental illness for the second edition (Lee et al., 2003; Penny et al., 1995), sufficient convergent validity and discriminative validity in people living with schizophrenia for the third edition (Leung & Man, 2007), acceptable convergent validity and good test–retest reliability in people living with mental illness for the fourth edition (Wu et al., 2007), and acceptable convergent validity in people living with mental illness for the fifth edition (Schubmehl et al., 2018).
Clinical Global Impression–Severity
Clinicians use the CGI–S to assess the severity of clinical symptoms on the basis of their own observations of the client’s behaviors and the client’s self-reported clinical symptoms at the time of the interview (Guy, 1976). The CGI–S has been widely used with people living with schizophrenia (Fernández-Miranda et al., 2021; Iyo et al., 2021; S.-C. Lee et al., 2021). Clinical symptom severity is scored on a seven-point scale: 1 = no symptoms, 2 = marginal symptoms, 3 = mild symptoms, 4 = moderate symptoms, 5 = marked symptoms, 6 = severe symptoms, and 7 = extremely severe symptoms. The CGI–S has satisfactory convergent validity in people living with schizophrenia (Turkoz et al., 2013). We used the CGI–S to evaluate the discriminative validity of the ACLS–6.
Mini-Mental State Examination
The MMSE assesses general cognitive functions, including orientation, memory, attention, language, and visuospatial skills (Y.-C. Lee et al., 2021). Higher scores reflect better general cognitive functions. The MMSE has acceptable psychometric properties for people living with schizophrenia (de Leon et al., 1993; Rademeyer & Joubert, 2016).
Montreal Cognitive Assessment
The MoCA is a screening test to detect cognitive impairments (i.e., orientation, memory, attention, executive functions, visuospatial skills, and language; Nasreddine et al., 2005). Higher scores indicate better cognitive functions. The MoCA has adequate psychometric properties for people living with schizophrenia (Rademeyer & Joubert, 2016; Yang et al., 2018).
Wisconsin Card Sorting Test
Clinicians use the WCST to assess examinees’ executive functions by having them sort 128 response cards, which depict three types of figures with varying shapes, numbers, and colors (Prentice et al., 2008). We used five WCST indexes as the criteria for convergent validity: total number correct, perseverative responses, perseverative errors, conceptual-level responses, and number of categories completed (Bell et al., 1997; Chiu & Lee, 2021). The psychometric properties (e.g., construct validity and test–retest reliability) of the WCST have been examined for people living with schizophrenia (Bell et al., 1997; Chiu & Lee, 2021).
Data Analysis
To examine convergent validity, we used Pearson’s r to estimate the correlations between the copying and stitching tasks and between these two tasks and each cognitive measure (i.e., MMSE, MoCA, and WCST). Correlations <.30 are considered low; .30–.70, moderate; and >.70, strong (Chiu et al., 2018).
To examine discriminative validity, we checked for floor and ceiling effects and conducted independent t tests. Floor effect and ceiling effect were defined as >15% of participants showing the lowest and highest possible scores, respectively, on the two tasks (Chiu et al., 2019). We used independent t tests (α = .05) to determine whether scores on the two tasks could distinguish between participants with two levels of clinical symptom severity—marginal and mild.
For test–retest reliability, we computed ICCs based on a two-way random effect with absolute agreement. ICCs ≤.39 indicate poor reliability; .40–.59, moderate reliability; .60–.79, good reliability; and ≥.80, excellent reliability (Chiu & Lee, 2021). We also calculated the minimal detectable change (MDC) in scores on the two tasks between the two assessment sessions. The MDC is the smallest real difference in scores between repeated assessments that is not attributable to random measurement error at a confidence level generally of 95% (Schreuders et al., 2003). Two formulas were used for computing MDC: (1) standard error of measurement
Results
Of the 295 community-dwelling people living with schizophrenia at the psychiatric center, occupational therapists referred 110 to participate in the study. All 110 participants completed two assessment sessions; there were no missing data points. Participants’ mean age was 45.0 yr, most were female (59.1%), and their clinical symptoms were marginal (CGI–S score = 2; n = 36) or mild (CGI–S score = 3; n = 74). Additional participant characteristics are shown in Table 1.
Participant Characteristics (N = 110)
Note. CGI–S = Clinical Global Impression–Severity.
The correlation (r) between the stitching and copying tasks was .88, indicating strong convergent validity. The two tasks showed moderate correlations with the MMSE, MoCA, and five indexes of the WCST (rs = .32–.52; Table 2). For discriminative validity, the stitching and copying tasks showed no floor effect (0% for both tasks) or ceiling effect (0.0% for the stitching task, 5.5% for the copying task; Table 3). The t tests showed significant differences between participants with marginal symptoms and those with mild symptoms (p < .001). For test–retest reliability, ICCs (MDCs) were .74 (.64) for the stitching task and .71 (.50) for the copying task.
Convergent Validity of the ACLS–6 Stitching and Copying Tasks
Note. ACLS–6 = Allen Cognitive Level Screen–Sixth Edition; MMSE = Mini-Mental State Examination; MoCA = Montreal Cognitive Assessment; WCST = Wisconsin Card Sorting Test.
Discriminative Validity and Test–Retest Reliability of the ACLS–6 Stitching and Copying Tasks
Note. ACLS–6 – Allen Cognitive Level Screen–Sixth Edition; CI = confidence interval; ICC = intraclass correlation coefficient; MDC = minimal detectable change.
Discussion
To the best of our knowledge, this is the first study to examine the psychometric properties of the stitching and copying tasks of the ACLS–6 for community-dwelling people living with schizophrenia. Our results show that the stitching and copying tasks exhibit good convergent validity, sufficient discriminative validity, and good test–retest reliability in this population. The merit of this study is that the two tasks were evaluated with a large sample (N = 110), allowing reliable generalization of the results to other schizophrenic populations who meet our recruitment criteria.
For convergent validity, a strong correlation was found between the stitching and copying tasks, and each task showed moderate correlations with three other cognitive measures (i.e., MMSE, MoCA, and WCST). These results confirm our hypothesis that the two tasks assess cognitive constructs. Given that the copying task was highly correlated with the stitching task and that their score ranges are similar, these results indicate that the copying task can be used as a substitute task for the stitching task. Our findings on convergent validity for the stitching task were consistent with those of previous studies with people living with schizophrenia (moderate correlations with the MMSE and the perseverative errors index of the WCST, rs = .47–.61; Leung & Man, 2007; Secrest et al., 2000). Good convergent validity indicates that the stitching and copying tasks of the ACLS–6 can be used to assess the cognitive constructs of people living with schizophrenia.
No floor or ceiling effect was found for the stitching and copying tasks, indicating that occupational therapy practitioners can use the two tasks to discriminate between schizophrenic people with minimum and maximum scores on cognitive functions. A previous study showed that the stitching task of the third edition of the ACLS could discriminate between people living with and without schizophrenia (Leung & Man, 2007), whereas our study found that the ACLS–6 discriminated between participants with marginal and mild clinical symptoms. Thus, the stitching and copying tasks can discriminate between groups of people with schizophrenia who have marginal and mild symptoms. However, participants in both studies displayed only two distinct levels of psychiatric symptoms; future studies are warranted with more than two participant groups based on level of symptom severity.
The stitching and copying tasks showed good test–retest reliability in our sample. Previous studies with small samples (Ns = 5–11) of people living with mental illness have shown excellent test–retest reliability in people living with mental illness (Penny et al., 1995; Wu et al., 2007). A possible reason for the lower reliability in our study may be the 2-wk reassessment interval. A 2-wk interval may cause unstable test results for schizophrenic people with cognitive impairments. To guide the clinical application of the ACLS–6, we estimated MDC values from the reliability coefficient (i.e., ICC) for the two tasks, which can help practitioners in clinical and research settings determine whether changes in clients’ scores over repeated assessments represent random variations or real change.
This study has two limitations. First, we used a convenience sample of patients from one psychiatric center in Taiwan, and we excluded those with substance-related disorders. Schizophrenia is highly comorbid with substance-related disorders, so this exclusion criterion may limit the external validity of our study. Future studies should examine the psychometric properties of the ACLS–6 stitching and copying tasks in participants living with schizophrenia who also have substance- related disorders, as well as participants from different cities and countries, to cross-validate our findings. Second, we did not evaluate the psychometric properties of the other new tasks in the ACLS–6. Future studies should evaluate the psychometric properties of the other new tasks with participants at various levels of the Allen Cognitive Disabilities Model (e.g., participants at Level 3 for the simple coloring task).
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice: The good psychometric properties of the ACLS–6 stitching and copying tasks indicate that these tasks can be useful in assessing cognitive functions, distinguishing marginal from mild symptom severity, and measuring changes in the cognitive abilities of community-dwelling people living with schizophrenia. The MDC values we have provided for the ACLS–6 stitching and copying tasks can help clinicians and researchers determine real changes between repeated assessments in people living with schizophrenia.
Conclusion
The stitching task using new materials and the copying task of the ACLS–6 have good convergent validity, discriminative validity, and test–retest reliability. The two tasks can be used to assess cognitive functions and monitor the recovery progress of community-dwelling people living with schizophrenia. In addition, the copying task can be considered as a substitute task for the stitching task.
Footnotes
Acknowledgments
We are grateful to all participants for their involvement. This study was supported by a research grant from Taipei City Hospital (TPCH-109-71), Taipei, Taiwan. Shu-Chun Lee and Li-Yu Chiu contributed equally to this work.