Abstract
The CAT–FER provides comprehensive, valid, and gender-unbiased assessments of FER in adults living with schizophrenia and can help occupational therapists identify deficits in recognition of specific basic emotions and plan corresponding interventions to manage the impact on their clients’ social functions.
Facial emotion recognition (FER) is a multidimensional ability that, conceptually, includes the ability to recognize seven basic emotion domains—happiness, sadness, anger, disgust, fear, surprise, and calm—that have been shown to be consistently present in people of different races and cultures and have unique meanings in adapting to environments (Behere, 2015; Ekman, 2008). FER is crucial for effective social interaction because accurate recognition of others’ emotions is a prerequisite for the interpretation of others’ behaviors and the determination of appropriate responses (Behere, 2015; Ekman & Cordaro, 2011). People living with schizophrenia, however, tend to have moderate to severe deficits in FER (Behere, 2015; Cotter et al., 2018), which are correlated with their symptoms of psychosis (e.g., hostile delusions and negative symptoms; Behere, 2015; Green, 2016), social function (e.g., social participation and interpersonal relationships; Behere, 2015; Brekke et al., 2005), and quality of life (Maat et al., 2012). Moreover, interventions that target FER ability (e.g., the Social Cognition and Interaction Training program; Penn et al., 2007) can improve people’s FER performance and even facilitate their social participation (Kurtz & Richardson, 2012). To manage deficits in FER, a comprehensive, reliable, and valid FER measure is needed.
Five measures are commonly used to assess FER deficits: (1) the Bell Lysaker Emotion Recognition Task (BLERT; Bryson et al., 1997), (2) the Facial Expressions of Emotion: Stimuli and Tests (FEEST; Young et al., 2002), (3) the Face Emotion Identification Test (Kerr & Neale, 1993), (4) the Penn Emotion Recognition Test (Kohler et al., 2003), and (5) the Florida Affect Battery (Bowers et al., 1998). Two main problems limit their utility, however. First, these measures cannot comprehensively assess FER deficits in all seven emotion domains, which may lead to underestimation of said deficits. To be specific, all except one of these measures (the BLERT) provide only a total score, which may mask FER deficits because poor performances in some domains may be offset by good performances in others. Moreover, the severity of FER deficits tends to differ across domains (Barkl et al., 2014). For example, a person may have severe deficits in the disgust and fear domains and relatively mild deficits in the happiness and calm domains, but their overall deficit as represented by a total score would be only moderate to severe. None of these measures (with the exception of the FEEST) assesses all seven basic emotion domains which may result in underestimations because the domains in which a person is impaired may not be assessed.
In addition, these measures do not consider the impact of examinee gender on scoring. In particular, previous studies have shown that women tend to have better FER levels than men (Montagne et al., 2005; Mote & Kring, 2016), which may result from gender differences in FER abilities and/or systematic bias in the measures that favors women. Without an examination of potential gender differences in scoring, however, the effect of systematic biases in these measures cannot be ruled out, therefore limiting the examination of gender differences in FER. In sum, the most commonly used measures of FER are incomprehensive and possibly gender biased.
A comprehensive measure (i.e., one that assesses all seven basic emotion domains) is likely to result in a lengthy assessment, though. In particular, reducing an instrument’s length usually leads to low reliability (Kruyen, 2012), making it difficult to create comprehensive, reliable, and efficient assessments. To overcome such difficulties, computerized adaptive testing (CAT) is recommended (Wang & Chen, 2016). CAT comprises three main processes: (1) item selection, in which the items to be administered are selected from the item bank on the basis of an examinee’s estimated ability; (2) ability estimation, in which the examinee’s ability is estimated on the basis of all completed items and the item difficulties; and (3) stopping rules confirmation, in which the decision of whether to stop the assessment is based on the adopted set of stopping rules (e.g., reliability ≥.90). These processes are repeated until the adopted stopping rules are satisfied or all items in the item bank have been answered. By following these three processes, CAT can be used to administer the most informative items (not too easy or too difficult) for examinees to shorten the assessment length without sacrificing reliability (Wang & Chen, 2016). Moreover, CAT is particularly effective when applied to a multidimensional structure (e.g., FER) because it allows information obtained from one domain to be used for estimations in other domains through interdomain correlations (Wang & Chen, 2016). Thus, CAT appears to be a promising means by which to create efficient, reliable, and comprehensive assessments.
The purposes of this study were twofold: (1) to develop a computerized adaptive test of FER, the CAT–FER, for adults with schizophrenia and (2) to examine the psychometric properties of the CAT–FER, including Rasch reliability, construct and concurrent validity, differential item functioning (DIF) of gender, and efficiency (number of items needed for an administration).
Method
Participants
We recruited both adults with schizophrenia and healthy adults to ensure a wide range of FER levels. This study was approved by the ethics committees of the hospitals where the participants were recruited. All participants provided written informed consent.
Adults living with schizophrenia were recruited from three psychiatric hospitals in northern and southern Taiwan. The inclusion criteria were as follows: diagnosis of schizophrenia according to the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; American Psychiatric Association, 2013), ability to follow three-step instructions, and age ≥20 yr. Potential participants were excluded if they had other diagnoses that might affect their cognitive and emotional functions (e.g., substance abuse or brain injury) or if they could not complete all the assessments.
Adults without diagnosed mental illness older than age 20 yr were recruited by means of online advertisements. We excluded adults who had a history of substance abuse, brain injury, or diagnosis of psychosis, or who were unable to finish all the assessments in this study.
Procedure
This study was divided into three phases: (1) development of the FER item bank, (2) validation of the FER item bank and the DIF of gender, and (3) determination of the administration mode for the CAT–FER using data simulations.
Development of the Facial Emotion Recognition Item Bank
The items for the FER item bank were photographs of professional performers’ facial expressions that were selected from a published database (the Behavioral Evaluation Norm for Facial Expressions of Professional Performers; Chen et al., 2013). The database contains 12,393 valid photographs with three views (front, three-quarters, and profile) depicting seven basic emotions (i.e., happiness, sadness, anger, disgust, fear, surprise, and calm) collected from 29 professional performers (15 men and 14 women; Chen et al., 2013). All the photographs were validated on ratings made by at least 50 college students (i.e., 100 students for photographs with front views and three-quarter views and 50 students for photographs with profile views) of the intensity of the seven emotions being depicted (Chen et al., 2013). The face validity of these pictures was thus established. The use of the photographs was approved by the developers of the database.
The candidate items/photographs for the FER item bank were selected according to five criteria. First, the photographs should have a front view because this provides the most information about the performers’ facial expressions compared with the other views. Second, the items in each domain should have appropriate item difficulties (items could be correctly identified by 40%–90% of the students) that are not too easy or too challenging (accuracy ≥90% and ≤40%, respectively). The accuracy was calculated using the college students’ ratings of the photographs, which were extracted from Chen et al.’s (2013) database. Third, the photographs selected should depict equal numbers of male and female performers. Fourth, the photographs should depict performers in different age groups (six age groups were adopted: <20, 20–29, 30–39, 40–49, 50–59, and 60–69). Fifth, the photographs selected should be of different performers, with minimal duplications. If the last four criteria could not be achieved simultaneously, we chose appropriate item difficulty as our first consideration, followed by the performers’ genders, the performers’ ages, and the duplications of performers. The process of item selection was performed by the first author (Shih-Chieh Lee).
Validation of the Facial Emotion Recognition Item Bank and the Differential Item Functioning of Gender
After participants had provided informed consent, demographics and clinical data (the latter for participants living with schizophrenia only, including assessments of cognitive function and clinical severity) were collected by trained research assistants. Then the candidate items were administered to the participants by means of a computer program.
We developed the administration interface using Psychophysics Toolbox Version 3 (MATLAB version; Brainard, 1997), which was used to present the items/photographs and response categories and to collect the participants’ responses. The administration interface showed candidate items (i.e., a photograph of a facial expression) in the center of the screen, one at a time. Eight buttons were displayed at the bottom of the screen, corresponding to the seven emotion domains and an “I don’t know” option, as shown in Figure 1. Participants were asked to determine the performer’s emotion and respond by selecting one of the eight possible responses. Each correct answer was scored 1 point; “I don’t know” responses were scored as incorrect (i.e., 0 points). To lessen fatigue, the participants rested for at least 10 s after every 30 items. No immediate feedback was provided during the administrations; the results of overall performance (i.e., number of correct answers in the seven domains) were provided to participants after all items had been completed.
The administration interface with candidate facial emotion recognition items.
We then performed Rasch analysis to examine the model fits, dimensionality (a kind of construct validity), item difficulties, and impact of examinee gender on scoring (i.e., the DIF of gender) of the candidate items. Items with satisfactory model fits (also indicating examination of the dimensionality) to a multidimensional Rasch model were used to form the FER item bank, and their item difficulties were estimated. The item difficulties of the DIF items were adjusted to control the impact of examinee gender.
Determination of Administration Mode for the CAT–FER Using Data Simulations
We performed simulations to calculate the Rasch reliability and efficiency (number of items needed for an administration) provided by the CAT–FER with different sets of stopping rules. The set that achieved both high reliability (average Rasch reliability ≥ .90 across all seven domains) and efficiency (<70 items) was determined as the ideal administration mode for the CAT–FER (Aaronson et al., 2002). If high reliability and efficiency could not be achieved simultaneously, two suboptimal administration modes (the reliable mode and the efficient mode) were selected for prospective users to focus on either reliability or efficiency. The reliable mode incorporated the set of rules that used the fewest items to achieve reliability similar to that of the full item bank, whereas the efficient mode used the set of rules that included the fewest items to achieve acceptable reliability (≥.70; Aaronson et al., 2002). The simulations were performed only on our target population: adults living with schizophrenia.
The candidate sets of rules for the CAT–FER were proposed according to the stopping rules often used in other CATs (Lee et al., 2018; Lin et al., 2019); specifically, the following two stopping rules were used in this study to deal with the reliability from different perspectives (Choi et al., 2010). First, minimal reliability required (MRR), which focuses on whether reliability is satisfactory, is the lowest reliability needed for an assessment (i.e., ≥.70 for group-level comparisons and ≥.90 for individual-level comparisons). Thus, MRR can be used to stop the assessment when a sufficient level of reliability is achieved (Aaronson et al., 2002). Limited reliability increase (LRI), which focuses on whether it would be worthwhile to continue the assessment, is the increment in reliability after administration of an additional item. Accordingly, LRI can be used when no more informative items can be administered. Considering that MRR and LRI both have their unique advantages in different situations, both sets were adopted.
Sixteen candidate sets of stopping rules were proposed. Among them, six sets of stopping rules were MRR (e.g., MRR ≥ .90, MRR ≥ .80, and MRR ≥ 0.70) and LRI (e.g., LRI < .001, LRI < .005, and LRI < .010) alone, and 9 sets of stopping rules were combinations of LRI and MRR (i.e., three levels of LRI × three levels of MRR), such as MRR ≥ .90 or LRI < .001. The last set of stopping rules had no criteria for MRR and LRI because it was designed to administer all items in the item bank to provide the maximum obtainable reliability of the CAT–FER.
Measures
The Mini-Mental State Examination (MMSE; Folstein et al., 1975) and the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005) were used to evaluate participants’ cognitive function. The MMSE has 11 items across various cognitive domains (e.g., orientation, attention, memory, language, and construction; Folstein et al., 1975). The total score of the MMSE ranges from 0 to 30. It has acceptable psychometric properties in adults living with schizophrenia (Rademeyer & Joubert, 2016).
The MoCA has 30 items that assess a broad area of cognitive functions (i.e., visuospatial skills, language, attention, memory, executive function, abstraction, calculation, and orientation). The total score of the MoCA ranges from 0 to 30. The MoCA also has acceptable psychometric properties in adults with schizophrenia (Rademeyer & Joubert, 2016; Rodríguez-Bores Ramírez et al., 2014).
The Clinical Global Impression–Severity scale (CGI–S) was used to assess symptom severity in the participants with schizophrenia (Haro et al., 2003). The CGI–S scores are ratings, made on a 7-point scale (1 = normal, 7 = the most extreme severity), that are based on observations of the person’s behavior and their self-reported symptom severity across a span of 1 wk. Higher scores indicate more severe symptoms. In general, the CGI–S has acceptable psychometric properties in people living with schizophrenia (Pinna et al., 2015).
Data Analysis
The model fits, unidimensionality of each domain, and DIF values of the candidate items were estimated by Rasch analysis, which was performed using ConQuest (Version 2; Wu et al., 2007). A multidimensional Rasch model was adopted because it can examine the unidimensionality of the seven domains and the relationships among these domains at the same time (Wang & Chen, 2016). Moreover, there are two advantages of the multidimensional Rasch model. First, the reliability and efficiency of the assessments can be further improved because the information from one domain can be shared through interdomain correlations (Wang & Chen, 2016). Second, the examinees’ abilities in all seven domains can be updated simultaneously after the administration of additional items through the shared interdomain correlations (Wang & Chen, 2016).
Misfit items were defined as those with infit or outfit mean squares (MnSqs) >1.4; all items meeting the definition were removed iteratively. The DIF items were defined as those with large DIF values (i.e., differences in item difficulty estimated for male and female participants ≥0.38; Zieky, 2003). The item difficulties of the DIF items were adjusted according to the participant’s gender (Hsueh et al., 2013) by assigning different weights according to that gender.
The simulations were performed using a MATLAB program (Version 2015a; MathWorks, Natick, MA) designed by the authors. The simulations used the responses by the participants with schizophrenia to the FER item bank to simulate the whole processes of the CAT–FER (including item selection, ability estimation, and stopping rules examination) with each of the 16 candidate sets of rules. To be specific, in regard to item selection, the most informative item was initially selected and administered to a participant on the basis of the Fisher information function (Reckase, 2009). Then, for ability estimation, the program estimated the participant’s abilities in all seven domains according to their response to that item using the maximum a posteriori estimation with the Newton–Raphson method (Reckase, 2009). These two processes were repeated until the adopted set of stopping rules was satisfied, and then the average Rasch reliability and the average number of items needed for an administration were calculated for each set of rules and compared to determine the best set for the CAT–FER.
Results
Demographics and Clinical Characteristics
Participants’ demographic and clinical data are shown in Table 1. A total of 452 participants (351 adults living with schizophrenia and a comparison group of 101 adults without diagnosed mental illness) were recruited through convenience sampling. In general, there were slightly more female than male participants (54.7% of participants living with schizophrenia, 63.4% of comparison participants). The average ages were 45.0 ± 10.7 (participants living with schizophrenia) and 23.3 ± 2.3 (comparison group). All participants living with schizophrenia had sufficient cognitive function (average scores on the MMSE or the MoCA = 26.1 and 25.3, respectively) and mild clinical severity (median [Mdn] CGI–S score = 3).
Characteristics of the Participants
Note. — = data not collected or not applicable; CGI–S = Clinical Global Impression–Severity scale; Mdn = median; MMSE = Mini-Mental State Examination; MoCA = Montreal Cognitive Assessment; Q = quartile.
Not reported for 10 patients with schizophrenia.
Development and Validation of the Facial Emotion Recognition Item Bank
A total of 168 items/photographs (i.e., 24 for each domain) were initially selected for the FER item bank. After the misfit items were removed, the remaining 165 items with satisfactory model fits (infit MnSq = 0.81–1.36, outfit MnSq = 0.13–1.24) were used to form the FER item bank (the model fit and item difficulties can be found in Supplemental Table 1; to access supplemental content, go to https://ajot.aota.org, navigate to the top of this article, and click on “supplemental”). Among these items, 39 had severe DIF of gender (20 with DIF values higher than the acceptable range [0.39 to 1.16], and 19 with DIF values lower than the acceptable range [–0.41 to –0.96]); therefore, to control the gender bias, the item difficulties of these DIF items were adjusted according to the participants’ genders by adding different weights (e.g., the item difficulties for the first item in the calm domain were 1.89 and 2.85 for men and women, respectively).
As shown in Figure 2, the distributions of item difficulties in most of the seven domains matched those of the participants with diverse FER levels, except in the calm, happiness, and fear domains; however, the differences between adjacent item difficulties were large (≥.50 logit) for participants with extremely high and/or low FER levels in these domains. Moreover, three domains (calm, happiness, and fear) appeared to lack items for participants with specific FER levels (the calm and happiness domains lacked difficult items, and the fear domain lacked easy items).
Distributions of item difficulties of the FER item banks for each emotion domain and participants’ FER levels.
Determination of Administration Mode for the CAT–FER Using Data Simulations
Table 2 shows the simulated reliability and efficiency provided by the CAT–FER with the candidate sets of stopping rules. Given that none of the 16 candidate sets of stopping rules achieved high reliability and efficiency simultaneously, two suboptimal administration modes (the reliable mode and the efficient mode) were suggested for prospective users.
Simulated Reliability and Efficiency of the FER Item Bank and the CAT-FER With Selected Sets of Stopping Rules
Note. An = anger; Ca = calm; Di = disgust; Fe = fear; Ha = happiness; LRI = limited reliability increase; MRR = minimal required reliability; Sa = sadness; Su = surprise.
The reliable mode was determined to be “Rasch reliability ≥ .80 or LRI < .001” because it needed the fewest items to achieve reliability similar to that of the full item bank (average Rasch reliability = .72–.81 and .73–.84, respectively). With the reliable mode, on average, 128 items (77.6% of the full item bank, about 20 min for assessment) were used. The scores of the CAT–FER with both modes were highly correlated (r = .93–1.00) with those of the whole item bank.
The efficient mode was determined to be “MRR ≥ .70 or LRI < .001” because it used the fewest items to achieve reliability ≥.70. With the efficient mode, the CAT–FER used about 73 items (44.1% of the full item bank, approximately 11 min for administration) to provide generally acceptable reliability (average Rasch reliability = .69–.73) for all emotions.
Discussion
A total of 165 items with satisfactory model fit to the multidimensional model were selected to form the FER item bank. Satisfactory model fit indicates that the participants’ responses to these items meet the expectations of the multidimensional Rasch model and thus supports that the items in each domain are unidimensional. Given that unidimensionality means that the items in each domain are affected by only a single latent trait (i.e., each of the seven FER domains), the selected items serve as valid indicators of FER in each domain. Therefore, these items were selected to provide valid FER assessments.
None of the 16 candidate sets of stopping rules achieved high reliability and efficiency simultaneously, which may have resulted from the lack of items in the item bank. Figure 2 shows that three domains lacked some items with certain item difficulties (the calm and happiness domains lacked difficult items, and the fear domain lacked easy items), which limited the reliability of the CAT–FER for some participants. To compensate for the limited reliability, more items had to be administered to satisfy the criterion of the stopping rules (MRR in this study) and led to decreased efficiency. In future studies, more items with varying item difficulties could be added to improve the quality of the item bank.
Although the ideal mode achieving both high reliability and efficiency could not be realized, two suboptimal sets of stopping rules (the reliable mode and the efficient mode) are recommended for prospective users. For users who require precise assessments, the reliable mode (average Rasch reliability = .73–.84), with optimal reliability and modest efficiency (128 items, about 20 min), is recommended. For those who prioritize administration efficiency, the efficient mode (73 items, about 11 min) can be used to minimize the administrative burden and obtain acceptable reliability (average Rasch reliability = .69–.73). Moreover, regardless of which set is adopted, the scores of the CAT–FER were highly correlated (r = .93–1.00) with those from the full item bank, supporting its concurrent validity. Therefore, prospective users can flexibly select the set of rules to achieve valid assessments with optimized reliability or efficiency, depending on their needs.
The primary advantage of the CAT–FER is that it addresses the seven domains in one assessment, thereby reflecting the comprehensive profiles of respondents’ FER levels. The individual domain scores are helpful for clinicians to identify clients’ FER deficits in specific domains (e.g., anger and fear) and subsequently are useful for setting corresponding intervention goals and programs. For example, knowledge about the features of the faces presented in the domains in which impairment is shown, and training on recognizing these faces, can be provided to improve FER accuracy (Penn et al., 2007). In addition, these scores will allow researchers to investigate the mechanisms of FER, including the factors that affect people’s impairment severity in each domain individually or collectively, as well as the social dysfunction caused by the impairment. Therefore, the CAT–FER can be useful to both clinicians and researchers in measuring severities of FER deficits comprehensively.
Another advantage of the CAT–FER is that the impact of respondent gender on the scores has been controlled; specifically, we found that 39 items had severe DIF of gender, indicating that the participants’ responses to these items were affected by their gender. To control for these biases, we assigned different weights to the DIF items for male and female participants with schizophrenia to yield unbiased estimations of their FER levels. Hence, the CAT–FER can be used to accurately measure FER levels in both men and women.
Our findings also suggest that DIF items may exist in the frequently used FER measures. Given that DIF items produce biased scores, interpretations of previous findings may have been misguided (Mote & Kring, 2016). For example, previous studies have shown that women have higher FER levels than men (Hoffmann et al., 2010; Rotter & Rotter, 1988), which may have been due to the inclusion of some DIF items in the measures used in these studies (Hoffmann et al., 2010). Given that the influence of gender on the FER level in each domain can be confirmed only after the DIF of the FER measure is controlled, examinations of DIF in FER measures are needed.
One additional advantage of the CAT–FER is that the scores have better interpretability than those of the commonly used FER measures; for example, the CAT–FER provides an individualized Rasch reliability for each score, which helps users judge the precision of assessments and allows them to interpret the results with confidence. In addition, the Rasch scores can be presented as T scores, which were calculated on the basis of comparisons of the FER levels of the 452 participants in this study. In particular, the participants consisted of both adults with schizophrenia and adults without diagnosed mental illness, covering the continuum of FER levels from the lowest (adults with schizophrenia) to the highest (comparison group); accordingly, the results of the CAT–FER show high interpretability for both clinicians and researchers.
A primary concern about the CAT–FER is that its items provide no contextual information, which may differ from the reality of daily life. However, assessments of FER may conceptually be independent of context for two reasons. First, using both facial and contextual information to infer others’ emotions appears to be a different ability from FER (Mayer et al., 2003). In particular, combining emotional and contextual information can happen only after the information has been extracted, which is more likely to be an ability found in the advanced phases of emotion processing (e.g., emotion facilitation and emotion understanding; Mayer et al., 2003). Second, combining the emotional and contextual information may lead to underestimations of respondents’ deficits in FER. In particular, respondents may perform well because of their ability to recognize others’ emotions using contextual information (e.g., people are more likely to be happy compared with other emotions when they are in enjoyable situations, such as eating a great meal), which would increase the difficulty of detecting FER deficits. Accordingly, it seems more appropriate to use stimuli without contextual information for FER assessments.
This study has four limitations. First, the participants in this study were all Asian. Second, the clinical severity of the schizophrenia in those participants was relatively mild (Mdn CGI–S score = 3). These two limitations may restrict the generalizability of our findings. Third, the healthy adults in this study were much younger than the adults with schizophrenia (average ages = 23.3 vs. 45.0), which may have resulted in overestimation of the FER scores of the former group (Gonçalves et al., 2018). To form a more rigorous norm, adults with no mental health diagnosis and of varied ages and educational levels can be recruited in future studies (Gonçalves et al., 2018). Fourth, the CAT–FER’s reliability and efficiency were simulated on the basis of data used in the development of the FER item bank, which may differ from those directly provided by the CAT–FER. Further validations are needed.
Implications for Occupational Therapy Practice
The findings from this study have the following implications for occupational therapy practice:
The CAT–FER can comprehensively assess FER functions in seven basic emotion domains.
Given that FER is crucial for people to correctly interpret others’ social behaviors, the CAT–FER can help occupational therapists identify clients’ FER deficits and provide corresponding interventions to improve their social participation and interpersonal relationships.
Conclusion
Our results show that the CAT–FER provides comprehensive, valid, and gender-unbiased assessments of adults’ FER ability with acceptable reliability and efficiency. Therefore, it can be useful in capturing the characteristics and severities of FER deficits in adults with schizophrenia.
Footnotes
Acknowledgments
Chiu and Hsieh equally contributed to this work and serve as corresponding authors. This work was supported by the Ministry of Science and Technology (MOST 105–2314-B-227–010-MY2).