Abstract
Social knowledge (SK; also called social schema) refers to the knowledge required to participate in social environments (Addington & Piskulic, 2011; Charernboon & Patumanond, 2017; Green et al., 2008). SK includes
knowledge of roles, which allows people to identify the characters in specific social situations (e.g., a person wearing a uniform in a restaurant is the host who seats customers);
knowledge of routines within social situations, which defines the appropriateness of people’s behaviors (e.g., customers should wait for the host to seat them); and
knowledge of purposes for engaging in a certain context (e.g., customers’ purpose in going to a restaurant is to enjoy a meal; Corrigan, Silverman, et al., 1996).
SK is crucial to understanding social situations and responding appropriately (Savla et al., 2013). Thus, SK is a prerequisite for effective social interactions (Fett et al., 2011).
People with schizophrenia tend to have moderate to severe deficits in SK (Savla et al., 2013). Research has demonstrated that improvement in SK can have a positive effect on their social function (Reeder et al., 2006), suggesting that SK may be a determinant of functional outcomes. To develop effective interventions to improve participation in occupations for people with schizophrenia, a psychometrically sound measure for assessing SK is necessary.
The Situational Feature Recognition Test (SFRT) is the most commonly used SK measure (Corrigan, Buican, & Toomey, 1996; Corrigan, Silverman, et al., 1996). However, the SFRT has four main problems hampering its utility. First, the SFRT has shown unacceptable reliability (e.g., test–retest reliability = .49; Corrigan, Buican, & Toomey, 1996; Corrigan, Silverman, et al., 1996). Second, the SFRT is burdensome because of its numerous items (126 descriptors from nine situations). Third, the SFRT was developed 20 yr ago; thus, some items and descriptors appear outdated (Corrigan & Addis, 1995; Corrigan, Buican, & Toomey, 1996; Roder & Medalia, 2010). Fourth, the SFRT does not control for the possible bias of respondents’ gender on scoring. Gender bias is likely to be present on measures of SK (Charernboon & Patumanond, 2017; Navarra-Ventura et al., 2018) and affects the validity of scores for respondents of different genders. In summary, the SFRT cannot provide reliable, efficient, and valid assessment of SK, which limits the ability of clinicians and researchers to provide effective intervention.
Computerized adaptive testing (CAT) is used to achieve both reliable and efficient assessments (Cella et al., 2007; Jette & Haley, 2005; Lee et al., 2018). Specifically, CAT is a form of computer-based assessment that selects questions tailored to the person and thereby shortens the test to achieve the level of precision desired by users (Jette & Haley, 2005). Thus, CAT can be a promising solution to the limitations of the SFRT.
The purpose of this study was to develop the Computerized Adaptive Test of Social Knowledge (CAT–SK) for people with schizophrenia and to validate its reliability and validity. Occupational therapists can use the CAT–SK to assess clients’ SK and plan interventions fitted to their needs (Savla et al., 2013). For example, the CAT–SK can be administered to clients before and after an intervention program to efficiently evaluate the program’s effectiveness (the administration interface of the CAT–SK is still in development, and currently the items are only available in Chinese). The research question was, Can the CAT–SK reliably, validly, and efficiently assess levels of SK in people with schizophrenia?
Method
Participants
We recruited participants from sheltered workshops, day care centers, and chronic wards of two psychiatric hospitals in Taiwan by convenience sampling. People ages 20–65 yr diagnosed with schizophrenia according to criteria of the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; American Psychiatric Association, 2013) were included. Potential participants were excluded if they had insufficient cognitive function (Mini-Mental State Examination [MMSE] score <24; Folstein et al., 1975), substance abuse, severe vision impairment, or other neurological deficit (e.g., dementia, mental retardation). This study was approved by the hospitals’ ethics committees. All participants provided written informed consent before data collection.
Procedures
This study consisted of two phases: (1) development and validation of the SK item bank and (2) determination of the best stopping rules for the CAT–SK.
Developing and Validating the Social Knowledge Item Bank
Two occupational therapists working in the field of psychiatry developed candidate SK items that were based on common social settings (e.g., restaurant, home, hospital) and the social roles, rules, and goals in these settings (Corrigan & Green, 1993; Corrigan et al., 1992; Green et al., 2008). All candidate items were designed as multiple-choice questions, each with four response choices.
Four experts (three occupational therapists working in psychiatric settings and one expert in CAT development [Hsieh]) reviewed the candidate items and achieved consensus on inclusion by discussion. After revising the items deemed unsuitable (e.g., inappropriate content or wording), we further examined the readability and comprehensibility of these items in cognitive interviews with 10 people with schizophrenia. After the items were finalized, we administered them to study participants for Rasch analysis. The number of items in each step of development is shown in Figure 1.
Flow diagram of items developed and examined.
To develop a Rasch-calibrated SK item bank, we first examined the data–model fit of the items to the Rasch model (Rasch, 1960; Wright & Masters, 1982). If participants’ responses fit a Rasch model’s expectations well, the model’s assumptions (e.g., unidimensionality) were satisfied and supported the validity of the Rasch-related parameters (Smith et al., 1998; Wright & Masters, 1982). After we removed the misfit items, the remaining items with satisfactory model fit were used to form the SK item bank. To further confirm the unidimensionality of the SK item bank, confirmatory factor analysis (CFA) with the one-factor model was performed. Then we examined the possible bias of participants’ gender on scoring (i.e., differential item functioning [DIF] of gender). DIF analysis was used to examine whether the item difficulty was systematically biased (easier or more difficult) toward male or female participants. To eliminate the impact of such bias, we removed the gender DIF items from the SK item bank. The remaining items were used to form the SK item bank for the CAT–SK.
Determining the Best Sets of Stopping Rules
The reliability and efficiency of assessments have commonly been optimized with two types of stopping rules: the minimal required reliability (MRR) and the limited reliability increase (LRI; Lee et al., 2018). The MRR, the lowest reliability needed for assessments, terminates an assessment when the Rasch person reliability is high enough (i.e., ≥.90). The LRI, the minimal change in Rasch person reliability after an additional item is administered, stops an assessment when no more informative items can be administered. In brief, the MRR focuses on whether Rasch person reliability is satisfactory, whereas the LRI focuses on whether it is worthwhile to continue administering items.
A total of 10 candidate sets of stopping rules were proposed and compared: five sets with the LRI alone (<.001, <.005, <.010, <.015, and <.020) and five sets combining the MRR (Rasch person reliability ≥.90) with each of the LRI criteria; the sets were given alphabetical designations (Sets A–J). We did not use the MRR alone because it can result in lengthy assessments when the Rasch person reliability cannot achieve the preset criterion even if the whole item bank has been used. Thus, we adopted the combination of the LRI and MRR to terminate an assessment when no substantial improvement in Rasch person reliability (e.g., <.020) occurred after an additional item was administered (Choi et al., 2010).
To determine the best set of rules, we used a two-stage screening. First, we selected the candidate sets that provided high reliability (average Rasch person reliability ≥.80; Lohr, 2002). Second, we selected the candidate sets that used fewer items to achieve similar levels of Rasch person reliability. However, if the Rasch person reliabilities provided by these candidate sets were not similar, we alternatively determined the most reliable (providing the highest Rasch person reliability) and the most efficient (using the fewest items) sets as choices for prospective users of the CAT–SK (e.g., clinicians and researchers).
Screening Measure
The MMSE is a widely used screening measure of cognitive function. It consists of six dimensions: orientation, attention, memory, language, praxis, and construction. It contains 11 questions and has a total score of 30. A cutoff score of 24 points is recommended for determining cognitive impairment in literate persons (Folstein et al., 1975).
Data Analysis
We used both infit and outfit mean square (MNSQ) to examine the model–data fit in Rasch analysis (Rasch, 1960; Wright & Masters, 1982). The misfit items (i.e., with either infit or outfit MNSQ exceeding the cutoff points) were deleted iteratively. The cutoffs for both MNSQs for the SK items were calculated on the basis of corrections of sample size (i.e., 
CFA was performed with EQS 6.1 with maximum likelihood estimators (Bentler & Wu, 2005). Both the model fit of the root mean square error of approximation (RMSEA) and the standardized root mean square residual (SRMR) were used. The criteria for good model fit were an RMSEA of ≤.06 (Hu & Bentler, 1999) and an SRMR of ≤.08 (Hu & Bentler, 1998, 1999).
The DIF items were identified when both of the following criteria were met: (1) DIF values (i.e., differences in item difficulties estimated for male and female participants) were large (≥.38 logits), and (2) DIF values achieved significant levels (≥1.96 logits times the corresponding standard error of estimations; Zieky, 2003). Items satisfying both of these criteria indicated substantial DIF of gender and were deleted.
We used the simulations to determine the best set of stopping rules. A MATLAB 2015a program (MathWorks, Natick, MA) was written to simulate administrations of the CAT–SK with every candidate set of stopping rules and estimate the corresponding reliability and efficiency. First, the program selected the most informative item from the item bank according to a participant’s SK level (the initial SK level for each participant was set at 0, indicating the middle SK level); the most informative item was identified by the highest Fisher information (Reckase, 2009). Second, the program used a real response from the participant (obtained from the first phase) to infer his or her SK level and estimate the Rasch person reliability. The estimations of the participants’ SK levels and Rasch person reliabilities were based on maximum a posteriori with an iterative Newton–Raphson process (Segall, 2001). These two steps were repeated until the predetermined sets of stopping rules were achieved. Then the indices of reliability (average Rasch person reliability and percentage of participants with Rasch person reliability ≥.70) and efficiency (average number of items needed) were estimated.
The Rasch person reliability is the reproducibility of participants’ ability levels, which is commonly used in Rasch analysis (Li et al., 2011). We examined the CAT–SK’s reliability at both the group level and the individual level. The average Rasch person reliability was used for group comparisons (e.g., demonstrating intervention effects by observing improvements in mean SK scores) with ≥.70 as an acceptable criterion. The percentages of participants who had individual Rasch person reliability were used for individual comparisons (e.g., monitoring change in social knowledge for each participant; Lohr, 2002) with ≥.90 as a sufficient criterion.
Results
Participant Characteristics
Table 1 shows characteristics of the 236 participants in this study. The participants’ average age was 45.0 yr, and about half were male. Most were unmarried and lived with their family. About half worked in a sheltered workshop, and the others were unemployed. On average, the participants had acceptable cognitive function.
Participant Characteristics (N = 236)
Note. M = mean; MMSE = Mini-Mental State Examination; SD = standard deviation.
Missing data for 29 people.
Social Knowledge Item Bank
A total of 101 multiple-choice questions were analyzed by Rasch analysis. An example item is “Which pairing of a family member and a corresponding duty is typically correct? 1) 2-year-old younger brother: mopping the floor; 2) 10-year-old older brother: raising younger siblings; 3) 80-year-old grandpa: fulfilling filial duty to his children; 4) 50-year-old father: sharing chores.” Participants had to choose the correct response from four choices. Each correct answer was awarded 1 point; thus, responses were recorded as dichotomous data. After Rasch analysis, we deleted 30 items because of insufficient model fit (5 items with infit or outfit MNSQ ≥1.4) or substantial DIF of gender (25 items). The remaining 71 items had acceptable fit indices, DIF values, and scale functions (data are provided in Supplemental Tables 1 and 2; available online at http://otjournal.net; navigate to this article, and click on “Supplemental”) and were used to form the SK item bank. Results of CFA showed that these items had satisfactory model–data fit (RMSEA = .04, SRMR = .06).
Figure 2 shows the distribution of the 71 item difficulties and 236 participant SK levels. The distribution of item difficulties (−2.2–2.4) covered the range of most of the SK levels (−1.2–3.4), with the exception of the range of participants with high SK scores (2.4–3.4). In addition, no obvious gap (i.e., difference of >0.5 in item difficulty between adjacent items) was found in the SK items. Regarding floor and ceiling effects, only two participants obtained the extreme SK scores, one the highest possible and one the lowest possible; for these participants, the Rasch person reliabilities provided by the SK items were acceptable (.81) and high (.93), respectively.
Person–item map of the SK item bank.
Best Sets of Stopping Rules
After the screening for reliability, four sets of rules (Sets A, B, F, and G) were selected because they provided high Rasch person reliability (average = .81–.88; Table 2). Because the Rasch person reliabilities provided by these sets were different, we determined the most reliable set and the most efficient set for prospective users.
Reliability and Efficiency of the CAT–SK With the 10 Sets of Stopping Rules
Note. CAT–SK = Computerized Adaptive Test of Social Knowledge; LRI = limited reliability increase; MRR = minimal required reliability.
Set A was determined to be the most reliable because it used the fewest items (40) to achieve the highest Rasch person reliability (.88). Set B was the most efficient because it used far fewer items (21) and sacrificed only a limited amount of Rasch person reliability (.81). The scores provided by the CAT–SK with these sets were highly correlated with the scores estimated using the entire item bank (r = .98 and .93, respectively). We did not select Set G as the most efficient set, even though it had exactly the same performance as Set B, because Set G adopted the LRI alone, which would not terminate the assessment even when its Rasch person reliability was sufficient for individual comparison (i.e., ≥.90) and thus might result in lengthy assessments.
Discussion
After removal of the misfit and DIF items, 71 SK items showed acceptable model fit to both the Rasch model and the one-factor CFA model, suggesting that these items are unidimensional (Wright et al., 1994). This unidimensionality indicates that these items mainly assess a single latent trait (i.e., SK); thus, participants’ responses on these items could be used to infer their SK level. All items were confirmed by experts, were understandable to participants, and had satisfactory scale functions, supporting their content validity and face validity. Moreover, none of the items had a severe DIF of gender, indicating that the CAT–SK is not affected by respondents’ gender and can provide unbiased estimation of SK regardless of gender. Thus, the items appear to provide valid assessments of respondents’ SK.
Two sets of stopping rules were selected to optimize the reliability and efficiency of the CAT–SK. The changeable sets of stopping rules provide an advantage of the CAT–SK to flexibly fit examiner’s needs. Examiners who need an assessment with high-level reliability can use Set A, with 40 items, to achieve Rasch person reliability of .88. If efficiency is the primary concern, examiners can use Set B, with 21 items, to provide Rasch person reliability of .81. Thus, the CAT–SK can meet varying needs, facilitating its utility in both clinical and research settings.
The CAT–SK is likely to overcome the challenges of achieving both high reliability and high efficiency. Compared with other SK tests, the CAT–SK possesses similar or higher reliability with much better efficiency (Achim et al., 2012; Corrigan, Buican, & Toomey, 1996; Corrigan, Silverman, et al., 1996). However, the CAT–SK appears less efficient than two other CATs, one for activities of daily living (ADL–CAT; Hsueh et al., 2013) and the other for the Fugl-Meyer Assessment (CAT–FM; Hou et al., 2012), because it needs more items to achieve high reliability. Specifically, regarding efficiency, the CAT–SK needs about 56% (Set A) and 30% (Set B) of the full length of the item bank, whereas the ADL–CAT and CAT–FM each need 13% of their item bank. For reliability, the average Rasch person reliabilities (.88 and .81) of the CAT–SK are lower than those of the ADL–CAT (.93) and the CAT–FM (.93). This relatively poorer performance in reliabilities and efficiency may result from the fact that the CAT–SK is a unidimensional assessment and thus cannot obtain information from items in other domains. This problem can be solved by adding sets of items within the same social situations that provide information about examinees’ SK levels or by combining items with items in other social cognition domains (e.g., social perception, which is highly correlated with social knowledge).
The CAT–SK has at least three additional advantages that can be helpful to clinicians: (1) CAT–SK results can be presented as T scores, which are easy to interpret by comparing with the distribution of the 236 participants’ SK levels; (2) CAT–SK scores can be calculated and stored automatically, which can help clinicians immediately interpret and communicate the results; and (3) the CAT–SK can provide reliability for each respondent, which can help clinicians determine the level of precision of each assessment. In addition, the CAT–SK can be used with electronic medical records to decrease time spent on paperwork and increase administrative efficiency. Thus, the CAT–SK shows great potential for increasing the efficiency of data interpretation and management.
The CAT–SK has two shortcomings, however. First, it lacks items for clients who have extremely high SK levels (≥2.6 logits), which may compromise its Rasch person reliability. However, reliability may not be a concern for these clients, because they have relatively intact SK. Second, it has many items with similar item difficulties. However, because the content of these items is different, the meaning of responses to these items is different. Moreover, the CAT–SK does not use all items in the item bank; therefore, items with similar item difficulties are less likely to be used in the same assessment. Use of a pool of items increases the diversity of assessments and thus helps reduce the practice effect.
Some limitations of this study may concern readers. First, we used convenience sampling. Second, the performance of the CAT–SK (e.g., reliability and efficiency of administration) was estimated using simulations, so our findings may differ from results obtained from real testing. These two limitations may restrict the generalizability of the study results. Third, because the CAT–SK has not been applied in the real world, it is not yet possible to determine the time needed to complete it. Fourth, because of the relatively small sample size, we could not examine other kinds of DIF (e.g., age, educational level, setting). Finally, given that the correlations between the CAT–SK and other SK measures remain unknown, further investigations of the construct validity of the CAT–SK are needed.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
The CAT–SK is a flexible measure of SK in people with schizophrenia with two sets of rules that can be used according to examiners’ needs. If examiners need an assessment with high reliability, they can use Set A (40 items, to achieve a Rasch person reliability of .88). If efficiency is the primary concern, they can use Set B (21 items, to achieve a person reliability of .81).
CAT–SK scores can be calculated and stored automatically to help clinicians immediately interpret and communicate the results and thus reduce administrative burden.
Conclusion
Our findings suggest that the CAT–SK appears to be a valid assessment that can be used flexibly to provide reliable or efficient assessments. Further validation of other psychometric properties of the CAT–SK, such as test–retest reliability and responsiveness, is warranted.
Footnotes
Acknowledgments
This work was supported by the Taiwan Ministry of Science and Technology (Grants 104-2314-B-030-008, 106-2314-B-030-007, and 107-2314-B-030-004).