Validation of ratings on the six-item Positive and Negative Syndrome Scale obtained via the Simplified Negative and Positive Symptoms Interview among outpatients with schizophrenia

Introduction

Aiming to facilitate measurement-based care in schizophrenia (Correll et al., 2011), a brief six-item version of the Positive and Negative Syndrome Scale (PANSS-6) has been extracted, which has since shown promise as an alternative to the full 30-item version of the PANSS (PANSS-30) in assessing treatment response and remission (Hieronymus et al., 2021; Østergaard et al., 2016, 2018a, 2018b). The PANSS-6 consists of three items from the positive subscale of the PANSS-30 (P1—Delusions, P2—Conceptual Disorganization, and P3—Hallucinatory Behavior) and three items from the negative subscale of PANSS-30 (N1—Blunted Affect, N4—Passive/Apathetic Social Withdrawal, and N6—Lack of Spontaneity and Flow of Conversation) (Østergaard et al., 2016).

To reduce the time required for conducting PANSS-6 ratings, a brief semi-structured interview has been developed, the Simplified Negative and Positive Symptoms Interview (SNAPSI), which takes approximately 15–20 min to administer (Østergaard et al., 2017). A recent study among inpatients examined the clinical validity of the PANSS-6+SNAPSI combination (Kølbæk et al., 2021), using the PANSS-30 (Kay et al., 1987) with the Structured Clinical Interview for the Positive and Negative Syndrome Scale (SCI-PANSS; Kay et al., 1992) as the gold standard reference. Specifically, an intra-class coefficient (ICC) of 0.77 [95% confidence interval (CI) = 0.62–0.85] was observed in this study—indicating an excellent level of agreement (Cicchetti, 1994)—between the PANSS-6_SNAPSI and the PANSS-6_SCI-PANSS total scores. These results emphasize the potential of the PANSS-6_SNAPSI to advance the implementation of measurement-based care in patients with schizophrenia.

In order to ensure the generalizability of the PANSS-6_SNAPSI, additional clinical validation among outpatients is essential. Inpatients and outpatients differ in a number of ways: in general, inpatients are less compliant to their treatment and present with more severe symptoms, lower levels of functioning, more substance abuse, and less insight into their psychiatric condition (Karpov et al., 2018). Furthermore, the two treatment settings are quite different; inpatients at psychiatric wards are frequently observed by staff members, who may serve as informants in relation to rating of psychopathology, for example, on the PANSS-6. Indeed, three of the items of the PANSS-6 are to be rated either solely or partially based on information from informants (Kay et al., 1987). Whereas outpatients, on the other hand, typically meet with their treatment provider for an hour every week or every 2 weeks. Therefore, in many cases, less fine-grained information will be available from an informant to support rating of psychopathology, unless a partner, relative, or friend is available and able to fill this role (Kay et al., 2006).

In the present study, the primary aim was to assess the validity of the PANSS-6_SNAPSI among outpatients with schizophrenia. Validity is often reported to comprise three central aspects, namely, criterion validity, content validity, and construct validity (Mokkink et al., 2010). The evaluation of these different aspects of validity involves a comparison with a gold standard (criterion validity), the assessment of appropriateness of the targeting of the items to reflect the underlying syndrome (content validity), and the testing of hypotheses based on an underlying theory (construct validity). Additionally, responsiveness can be thought of as the validity of changes in scores and refers to the scale’s ability to detect changes over time. In accordance with this model of validity, the present study of outpatients with schizophrenia had the following aims:

Materials and methods

Procedure for obtaining PANSS-6 and PANSS-30 ratings

Figure 1 illustrates the procedure for obtaining the PANSS-6 and PANSS-30 ratings. The SCI-PANSS and SNAPSI interviews of the participants were conducted within 24 h of one another to ensure that both interviews were based on similar reference periods (the past week). The participants were asked to take part in this pair of interviews, separated by an interval of at least two months. This interval was chosen to allow for an examination of responsiveness. At the first time point (baseline), the order of the interviews was quasi-randomized based on the participants’ birth year (even versus odd), and the opposite order was employed at the second time point (endpoint). As a rule, the interviews were conducted face-to-face; however, between April 2020 and October 2020, some interviews were conducted via video calls, due to restrictions relating to the COVID-19 pandemic. All interviews were videotaped to allow for re-rating. Two PANSS-certified raters conducted the SCI-PANSS interviews, each of which was subsequently rated by one of the PANSS-30 raters based on the video recordings of the interviews and the corresponding IQ-PANSS. For each interview, the PANSS-30 raters were asked to note whenever the psychopathology relating to an item was not adequately explored by the interviewer.

OPEN IN VIEWER

Figure 1.

Flowchart of interviews and ratings with and without an informant.

The SNAPSI interviews were conducted and rated by the PANSS-6 raters. Whenever an informant was available, the SNAPSI informant section regarding item N4—Passive/Apathetic Social Withdrawal was conducted and rated by an additional rater (the informant-N4 rater). All raters were blinded to the other raters’ assessments. Furthermore, the PANSS-6 rater and the informant-N4 rater were blinded to the identity of the PANSS-30 rater and vice versa. For more details on the rating procedure, see the Supplementary Material.

Results

Sample characteristics

Table 1 lists the demographic and clinical characteristics of the participants. A total of 81 patients consented to participate in the study, eight of whom subsequently withdrew their consent or cancelled the interviews. Therefore, the final sample comprised 73 outpatients with schizophrenia (mean age 38.3 years, 56% males, 63% with paranoid schizophrenia). Ninety-two percent of the participants were treated with an antipsychotic, 90% with a second-generation antipsychotic, 15% with a first-generation antipsychotic and 58% with two or more antipsychotics. Of patients treated with antipsychotics, 51% were also treated with an antidepressant. The mean PANSS-6_SNAPSI total scores and PANSS-30_SCI-PANSS total scores at baseline were 17.4 (SD = 4.2, range = 7–31) and 62.7 (SD = 14.7, range = 37–101), respectively.

OPEN IN VIEWER

Table 1.

Demographic and clinical characteristics of the 73 participants.

	Total (n = 73)	Male (n = 41)	Female (n = 32)
Age (years), mean ± SD	38.3 ± 14.3	39.6 ± 14.0	36.6 ± 14.8
Diagnosis
Paranoid schizophrenia (F20.0)	46 (63.0%)	27 (65.9%)	19 (59.4%)
Other (F20.X)	27 (37.0%)	14 (34.1%)	13 (40.6%)
First contact to psychiatric services (years), median (interquartile range)	14 (13)	15 (14)	10.5 (11.25)
Illness duration (years), median (interquartile range)	8 (13)	10 (13)	8 (10.5)
First schizophrenia diagnosis
Paranoid (F20.0)	39 (53.4%)	21 (51.2%)	18 (56.2%)
Other (F20.0)	34 (46.6%)	20 (48.8%)	14 (43.8%)
Number of psychiatric contacts, median (interquartile range)
Duration of ⩽1 days	82 (65)	67 (65)	93.5 (76.25)
Duration of 2–6 days	1 (4)	1 (4)	1 (3)
Duration of ⩾7 days	9 (9)	7 (9)	10 (6.25)
Drugs for psychosis
Apriprazole	34 (46.6%)	16 (39%)	18 (56.3%)
Quetiapine	31 (42.5%)	20 (48.8)	11 (34.4%)
Clozapine	18 (24.7%)	8 (19.5%)	10 (31.3%)
Chlorprothixene	8 (11.0%)	—	—
Olanzapine	7 (9.6%)	—	—
Risperidone	6 (8.2%)	—	—
Paliperidone	5 (6.9%)	—	—
Zuclopenthixol	<5*	—	—
Other	<5*	—	—
Drugs for depression
Selective Serotonin Reuptake Inhibitor	23 (31.5%)	14 (34.1%)	9 (28.1%)
Serotonin and Noradrenaline Reuptake Inhibitor	5 (6.9%)	—	—
Tricyclic Antidepressants	<5*	—	—
Mirtazapine	<5*	—	—
Others		—	—
Antiepileptics	20 (27.4%)	7 (17.1%)	13 (40.6%)
Drugs for insomnia
Benzodiazepines	11 (15.1%)	—	—
Benzodiazepine-related drugs**	7 (9.6%)	—	—
Stimulants
Central stimulants or atomoxetine	<5*	—	—

*

Specific numbers cannot be reported due to risk of identification of individual patients.

**

Zolpidem and Zopiclone.

Criterion validity

Table 2 lists the ICCs for the six individual PANSS-6 items with all raters included (n = 123), with the outlying rater excluded (n = 81), and with re-ratings for the interviews by this excluded rater (n = 123). The corresponding ICCs between the PANSS-6_SNAPSI and the PANSS-6_SCI-PANSS total scores were 0.67 [95% CI 0.56–0.76], 0.75 [95% CI 0.63–0.83], and 0.72 [95% CI 0.62–0.79], respectively. The relative mean difference between the PANSS-6_SNAPSI total scores and the PANSS-6_SCI-PANSS total scores was 0.85 ± 3.6 points (n = 123). The ICC between the total scores only for face-to-face interviews with all raters included (n = 94) was 0.68 [95%CI 0.54–0.78].

OPEN IN VIEWER

Table 2.

Criterion validity of PANSS-6_SNAPSI and PANSS-6_SCI-PANSS.

	Mean (SD)	Range	ICC	95% CI	F	p-Value
All raters (n = 123 ratings)
P1—Delusions	4.0 (1.7)	1–7	0.63	0.43–0.76	5.20	<0.001
P2—Conceptual Disorganization	2.4 (1.2)	1–5	0.40	0.20–0.55	2.58	<0.001
P3—Hallucinatory Behavior	3.8 (1.6)	1–7	0.83	0.76–0.88	11.34	<0.001
N1—Blunted Affect	2.9 (1.3)	1–6	0.50	0.36–0.63	3.03	<0.001
N4—Passive/Apathetic Social Withdrawal	2.2 (1.3)	1–6	0.44	0.29–0.58	2.71	<0.001
N6—Lack of Spontaneity and Flow of Conversation	1.6 (1.0)	1–5	0.44	0.29–0.57	2.60	<0.001
PANSS-6 total score	17 (4.2)	6–31	0.67	0.56–0.76	5.28	<0.001
With the outlying rater excluded (n = 81 ratings)
P1—Delusions	3.9 (1.7)	1–7	0.72	0.58–0.81	6.61	<0.001
P2—Conceptual Disorganization	2.2 (1.2)	1–5	0.49	0.30–0.64	3.11	<0.001
P3—Hallucinatory Behavior	3.9 (1.6)	1–7	0.81	0.71–0.87	9.48	<0.001
N1—Blunted Affect	2.9 (1.4)	1–6	0.49	0.31–0.64	2.93	<0.001
N4—Passive/Apathetic Social Withdrawal	2.4 (1.4)	1–6	0.49	0.31–0.64	3.06	<0.001
N6—Lack of Spontaneity and Flow of Conversation	1.3 (0.8)	1–6	0.53	0.24–0.71	4.06	<0.001
PANSS-6 total score	16.6 (4.4)	6–31	0.75	0.63–0.83	6.88	<0.001
With re-rated interviews (n = 123 ratings)
P1—Delusions	3.8 (1.6)	1–6	0.73	0.62–0.80	6.65	<0.001
P2—Conceptual Disorganization	2.2 (1.1)	1–5	0.41	0.25–0.55	2.49	<0.001
P3—Hallucinatory Behavior	3.9 (1.5)	1–7	0.82	0.30–0.61	3.02	<0.001
N1—Blunted Affect	2.7 (1.5)	1–6	0.53	0.39–0.64	3.24	<0.001
N4—Passive/Apathetic Social Withdrawal	2.5 (1.3)	1–6	0.45	0.30–0.58	2.69	<0.001
N6—Lack of Spontaneity and Flow of Conversation	1.5 (1.0)	1–5	0.47	0.30–0.61	3.02	<0.001
PANSS-6 total score	16.7 (4.4)	7–33	0.72	0.62–0.79	6.08	<0.001

CI: confidence interval; ICC: intra-class correlation coefficient; PANSS-6_SCI-PANSS: six-item version of the Positive and Negative Syndrome Scale extracted from the 30-item Positive and Negative Syndrome Scale obtained via the Structured Clinical Interview for the Positive and Negative Syndrome Scale; PANSS-6_SNAPSI: six-item version of the Positive and Negative Syndrome Scale obtained via the Simplified Negative and Positive Symptoms Interview.

Construct validity

Figure 2 shows scatter plots of all correlations between total scores on the PANSS-6_SNAPSI and the PANSS-30_SCI-PANSS as well as their respective subscales. Spearman’s rank correlation coefficients between PANSS-6_SNAPSI and PANSS-6_SCI-PANSS total scores were 0.62 with all raters included, 0.69 with the outlying rater excluded, and 0.66 with re-ratings for the interviews by the excluded rater. The corresponding correlations between total scores on the PANSS-6_SNAPSI and the PANSS-30_SCI-PANSS were 0.67, 0.69, and 0.64, respectively. Correlations between PANSS-6_SNAPSI scores and PANSS-30_SCI-PANSS for the positive and negative subscales ranged from 0.60 (negative subscale, all raters included) to 0.69 (negative subscale, outlying rater excluded).

OPEN IN VIEWER

Figure 2.

Construct validity of PANSS-6_SNAPSI and PANSS-6_SCI-PANSS.

Responsiveness

A total of 50 participants (a priori defined target sample size) were assessed at two time points during the study. The median interval between the baseline and endpoint assessments was 117 days (interquartile interval (IQI) = 73, 159). Figure 3 illustrates the correlations for all baseline-to-endpoint changes. The Spearman’s rank correlation for the changes in total scores from baseline to endpoint for PANSS-6_SNAPSI and PANSS-6_SCI-PANSS resulted in a coefficient of 0.40, p < 0.004 with all raters (n = 50), 0.55, p < 0.018 with the outlying rater excluded (n = 18), and 0.37, p < 0.009 with re-ratings for the interviews by the excluded rater (n = 50). The mean difference in scores between baseline and endpoint was −1.1 (standard deviation = 3.8) points for the PANSS-6_SNAPSI and −0.4 (standard deviation = 3.4) for PANSS-6_SCI-PANSS.

OPEN IN VIEWER

Figure 3.

Responsiveness for baseline–endpoint changes of PANSS-6_SNAPSI and PANSS-6_SCI-PANSS.

Discussion

This study was designed to validate the PANSS-6_SNAPSI among outpatients with schizophrenia, using the PANSS-6_SCI-PANSS as gold standard reference. The main findings from the primary analyses were that criterion, content, and construct validity were all acceptable, while responsiveness (sensitivity to change between baseline and endpoint) was low. All measures of validity and responsiveness improved markedly in post hoc analyses excluding ratings from one specific rater. When the interviews rated by this outlying rater were re-rated by other raters, the ICC for the total score increased from 0.67 to 0.72, which seemed to be driven by an increase in the ICC for item P1—Delusions from 0.63 to 0.73. The scores for the other items only changed slightly. The SNAPSI patient interview, including the patient section relating to item N4—Passive/Apathetic Social Withdrawal, could be completed in 21.9 min with no prior knowledge of the patient, compared to the 52.1 min required for the SCI-PANSS.

From a clinical point of view, this time difference between the two interviews is of utmost importance, as time is mentioned as one of primary barriers to routine use of rating scales (Lewis et al., 2019; Correll et al., 2011). Furthermore, we expect that the SNAPSI can be conducted even faster in a clinical setting, where treatment providers often have some prior knowledge and an established relationship with the patient. In the present study, the rater had prior knowledge of a patient, before the first interview, in only nine cases.

When reducing a rating scale from 30 to 6 items, an even greater time reduction might have been expected. However, there are probably several reasons why the time reduction does not match the reduction in items. First of all, 11 items from the PANSS-30, which are not a part of the PANSS-6, are based solely on observation and/or informant information. Secondly, in the PANSS-30, some items overlap in content (e.g., P1—delusions and P5—Grandiosity) and are thus rated based on some of the same information. Third, a SNAPSI interviewer will have to spend approximately the same amount of time introducing the interview and establishing rapport as a SCI-PANSS interviewer.

As has been illustrated by the results of this study, validity measures may be sensitive to the behavior of individual raters. In order to keep error variance to a minimum, a good level of inter-rater reliability was established for each rater team before interviews were initiated in this study. Additionally, the inter-rater reliability of the PANSS-6 raters was tested every 2–3 months throughout the period of the study. Despite these efforts, one rater produced systematically different ratings to the others, to an extent that the overall results were affected. One possible explanation is that this rater’s ratings drifted over the course of the study. Alternatively, and perhaps more likely, these differences in ratings may have been caused by an inadequately refined interview technique, eliciting insufficient information to produce accurate ratings; this could explain why no systematic differences were apparent in the inter-rater reliability tests, in which all raters rated the same videotaped interviews. Furthermore, this would also explain why the majority of the results for the re-rated interviews did not differ greatly from the results of the primary analyses. A third possible explanation is that the dual focus on interviewing and assessing symptom severity may have affected the accuracy of the assessments. Even with this systematic difference in ratings biasing the results, essentially all measures indicated an acceptable level of validity, underpinning the usability of the PANSS-6_SNAPSI. However, this aspect of the findings also underlines the importance of ongoing training for raters and evaluation of their ratings and interview techniques. This is likely the case for most psychiatric rating scales, but information on rater training, inter-rater reliability, and rater drift is often not reported (Berendsen et al., 2021; Mulsant et al., 2002).

Estimates of agreement for the individual PANSS-6 items varied with item P1—Delusions and P3—Hallucinatory Behavior achieving good and excellent agreement, respectively, and item P2—Conceptual Disorganization and the three negative items each achieving a fair level of agreement. The lower levels of agreement on items N1—Blunted Affect and N6—Lack of Spontaneity and Flow of Conversation are consistent with prior studies of PANSS-30 (Igarashi et al., 1998; von Knorring and Lindström, 1992; Yehya et al., 2016) and the PANSS-6 validation study in inpatients (Kølbæk et al., 2021). In the inter-rater reliability tests for the two rater teams in the present study, the lowest ICCs were also found for these two items (see the Supplementary Material). The lower agreement on these specific items possibly reflect a general tendency for clinicians to be less skilled in the assessment of negative symptoms compared to positive symptoms (Galderisi et al., 2018). Therefore, targeted training could be one way to address this issue. However, as suggested by Kølbæk et al. (2021), it may also be relevant to consider refining the anchor points for these items. Furthermore, lack of variance in symptom severity may also have affected the estimates of validity. Indeed, low variance in symptom severity across participants can result in lower ICC values (Mehta et al., 2018). Accordingly, in the present study, items with low variance in scores (i.e., symptom severity) yielded lower ICC values.

The relatively low level of agreement on item P2 was somewhat surprising, as both teams of raters had performed well on this item in the inter-rater reliability tests (producing ICCs of 0.71 and 0.73). It seems that there was a systematic difference between the two teams in the rating of this item, with the PANSS-6 raters rating higher severity (mean = 2.4) than the PANSS-30 raters (mean = 1.8). The same tendency was the case for item P1—Delusions (Mean score for the PANSS-6 raters and the PANSS-30 raters were 3.4 and 4.0, respectively). The two teams of raters held separate training sessions and, consequently, there was not an opportunity to have inter-team discussions of ratings, which may have led to the two teams interpreting the rating criteria differently. For item P2—Conceptual Disorganization, another contributing factor may be that some raters may have assigned excessive weight to the assignment related to this item (asking the patient to describe a sequence of actions leading him/her from A to B), which is included in the SNAPSI, but not in the SCI-PANSS. The assignment was designed to aid raters in assessing disorganization; however, ratings should reflect the overall impression gained from the entire interview.

Changes in scores between baseline and endpoint were minimal. This fact was not surprising when considering the patient population (outpatients) and the relatively short and arbitrarily defined period between the baseline and endpoint interviews (median = 117 days, IQI = 73–159). However, it may have affected the assessment of responsiveness, as the level of error variance may have approximated the level of true changes in symptom severity. Accordingly, in a longitudinal study of 1,344 outpatients with persistent symptoms of schizophrenia, Haro et al. (2018) found that symptomatology and PANSS scores remained relatively stable over a 12-month period for the majority of patients. Testing the responsiveness of a rating scale is important, but doing so was not the primary focus of the present study, and such a focus would have required a different setup and more time. Consequently, the assessment of responsiveness in the present study is limited by the relatively short and arbitrarily defined period between interviews.

There are limitations to this study that should be considered in the interpretation of the results. First of all, in this study, the main comparison was that of the PANSS-6_SNAPSI versus the PANSS-30_SCI-PANSS, from which the PANSS-6 was originally derived. In future studies, however, it would also be relevant to compare the PANSS-6_SNAPSI to other psychosis rating scales in order to examine the convergent validity of the PANSS-6_SNAPSI. Second, the PANSS-30 raters conducted ratings on the basis of videotaped interviews, whereas the PANSS-6 raters conducted the interviews themselves. Consequently, the PANSS-30 raters did not have the opportunity to ask questions or seek clarification during the interviews. The PANSS-30 raters were, however, asked to note if they lacked sufficient information to produce an accurate rating of an item, which was the case for only one interview. Third and relatedly, the difference in the qualitative experience of watching a videotaped interview compared to that of conducting a live interview may have affected the ratings. This may be particularly true for items that are based primarily on observation (Zarate et al., 1997). Fourth, participants’ diagnoses were not systematically confirmed via structured diagnostic interviews for the purpose of this study. However, schizophrenia diagnoses assigned as part of standard clinical practice at psychiatric hospitals in Denmark have high validity (Jakobsen et al., 2005; Uggerby et al., 2013). Fifth, we observed systematic differences between the PANSS-6 ratings from one specific rater and those from the other raters, affecting the overall results. Therefore, in order to provide the most accurate possible representation of the data, all results are reported both with and without data from this rater and also using data from re-ratings. In future studies and in clinical practice, ongoing training and evaluation of both interview technique and ratings should be prioritized in order to keep rater errors arising from, for example, rater drift and/or systematic differences in ratings, to a minimum. Finally, in validation studies employing more than one team of raters, it may be advisable to provide joint rater training before splitting the raters into separate teams in order to minimize the risk of systematic differences in rating between the teams.

In conclusion, the PANSS-6 obtained via the SNAPSI provides valid measurements of core symptom severity among both in- and outpatients with schizophrenia. Accordingly, the PANSS-6 was recently highlighted as an alternative to longer clinician-rated scales in the practice guideline for the treatment of schizophrenia published by the American Psychiatric Association (The American Psychiatric Association, 2021). Notably, however, assessment of core symptom severity cannot stand alone and should be accompanied by assessment of side effects to treatment as well as other symptom domains, such as cognition, depression and/or anxiety, ideally tailored to the profile of the individual patient.

Supplemental Material