Methodological Concerns Regarding a Meta-Analysis of the Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS)

We share an interest in the subjective experience of cognitive difficulties with White et al., ¹ who recently published a meta-analysis on this topic in the Canadian Journal of Psychiatry. In a publication from last year, we showed that individuals with schizophrenia or bipolar disorder report cognitive complaints, that cognitive complaints do not differ between these 2 groups, and that cognitive complaints are not associated with objective cognition as assessed with neuropsychological tests. ² In that paper, we introduced our novel measure of cognitive complaints, the Self-Assessed Cognitive Complaints Scale (SACCS). Stip et al. ³ kindly commented on our paper, providing us with an opportunity to expand on our work. ⁴ As rightfully pointed out by Stip et al., ³ we did not include a list of individual items, precluding a comparison with their established measure, the Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS). We recently published this list of items. ⁴

White et al.'s ¹ meta-analysis, aimed to reduce the heterogeneity in terms of methodology, that is, the instruments used to assess subjective cognition. As stated in their Plain Language Summary: “All studies included in the meta-analysis used the same questionnaire, the Subjective Scale to Investigate Cognition in Schizophrenia (SSTICS), to measure subjective cognitive complaints.” The aim of only including studies that used the SSTICS represents an improvement over a 2014 meta-analysis (by the same research group), which included other instruments as well. By only including studies using the same instruments, more reliable estimates can be provided. We commend this initiative.

We were therefore surprised to see our study ² —which used the novel SACCS—included along with 24 other papers in the SSTICS-only meta-analysis. Our study even received special mention as the only one to examine bipolar disorder. A closer look at the other 24 papers suggests that studies using a third instrument, the Self-Assessment Scale of Cognitive Complaints in Schizophrenia, SASCCS, were also included. It was used in two studies by Haddad and colleagues and one study by Johnson et al. ⁵ Only by scrutinizing the literature can the unfamiliar reader find that the name “SASCCS” appears to have used been for Arabic versions of SSTICS. Finally, one of the included studies (by Seco) does not appear in the reference list, but we were able to locate it and confirm that it used the (Spanish) SSTICS.

Given that the meta-analysis was not pre-registered, transparency and rigor in the analytic procedure are particularly important. This includes providing sufficient clarity to ensure that results can be easily verified. One paper ² clearly does not satisfy the main inclusion criterion, that is, using SSTICS, and its inclusion in the meta-analysis is therefore not substantiated. Including studies that do not use the SSTICS may bias the estimated effect sizes and thereby undermine the validity of the conclusions drawn about the SSTICS itself.

Unfortunately, due to these methodological limitations, our confidence in the findings of this meta-analysis ¹ is reduced.