Psychometric Properties of Two Brief Versions of the Hopkins Symptom Checklist: HSCL-5 and HSCL-10

Abstract

The Hopkins Symptom Checklist–25 (HSCL-25) is a widely applied measure of depression and anxiety. The present study examines two of its short forms—the HSCL-5 and HSCL-10, which have been proposed by previous research—in a representative sample of the German general population. To this end, we conducted exploratory and confirmatory analysis on two subsamples (n = 1,246 and n = 1,216). Our results suggest that, compared with the HSCL-25, both short forms represent economical ways of assessing depression and anxiety. Model fit was good and correlations with established measures demonstrate convergent validity. Both HSCL short forms are strongly invariant across sex, and we found evidence for partial strong invariance across age groups. Further analyses showed that differences in HSCL can be partially explained by sociodemographic variables. Finally, we report normative values for usage by researchers and clinicians. We recommend the HSCL-5 and HSCL-10 for clinical and research-oriented application.

Keywords

depression anxiety screening instrument mental health assessment normative data

Since its development in the 1950s (Parloff, Kelman, & Frank, 1954), the Hopkins Symptom Checklist (HSCL) has become established in clinical research. Many HSCL versions have been published and evaluated consisting of 9 to 90 items (Petermann & Brähler, 2013; Prinz et al., 2008). One of the most prominent versions of the HSCL family is the HSCL-25 (Glaesmer et al., 2014; Petermann & Brähler, 2013). The HSCL-25 is a widely applied screening tool for symptoms of depression, anxiety, and psychological distress in general. It consists of 25 items in total: 10 of these capture anxiety symptoms, while the other 15 deal with symptoms of depression. The sum score of the depression and anxiety subscale represents psychological distress.

Both diagnoses—depression and anxiety—are among the most prevalent mental disorders in the general population: More than 10% of all people suffer from either one or both impairments at least once during their lifetime (Bandelow & Michaelis, 2015; Jacobi et al., 2004; Kessler & Bromet, 2013; Wittchen et al., 2011). This can partly be explained by the high comorbidity of not only depression and anxiety disorders (Brady & Kendall, 1992; Roy-Byrne et al., 2000) but also the high comorbidity with other diseases, such as coronary heart disease and diabetes, as well as others (Anderson, Freedland, Clouse, & Lustman, 2001; Rudisch & Nemeroff, 2003).

Therefore, a valid and reliable assessment of symptoms of anxiety and depression is of the utmost importance in clinically applied psychology and in research settings alike. Despite the popularity of the HSCL-25, evaluations of the psychometric properties of the questionnaire pointed out some shortcomings. For instance, Glaesmer et al. (2014) evaluated the HSCL-25’s psychometric properties in a representative sample (N = 2,520; age range: 14-91 years), finding a barely acceptable model fit along with good reliability for the 25-item scale. Additionally, the facets depression and anxiety showed a very high correlation (.78), along with a lack of differential correlations of the subscales with external criteria. In a comprehensive study based on a student population comprising a total data set of N = 13,525 students (Skogen, Øverland, Smith, & Aarø, 2017), the unique variance attributed to the subscales when accounting for the general factor was comparatively low.

Strand, Dalgard, Tambs, and Rognerud (2003) examined two alternative measures: a 5-item and a 10-item version of the HSCL. These scales represent two brief, economical instruments for the assessment of symptoms of anxiety and depression. Considering disadvantaged people in terms of cognitive processing (i.e., older individuals, psychiatric patients) or settings where a brief screening is more desirable (i.e., large scale health surveys, repeated measurement in experimental settings), 25 items appear excessive. Shrout and Yager (1989) argue that the shortening of an initially reliable scale is easily feasible without any substantial losses in specificity or sensitivity, which Strand et al. (2003) empirically demonstrated in their study. However, a more in-depth investigation of psychometric properties—specifically, model fit and measurement invariance—is still lacking. In particular, the factor structure—whether the measurement model is best represented by one or two factors—has not been investigated as of yet.

The present study evaluates two short versions of the HSCL, which respectively consist of 5 and 10 items. We will examine item and scale descriptive statistics, and run exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) to judge model fit and measurement invariance regarding sex and age. Furthermore, we will inspect convergent validity with related measures of depression and anxiety, also comparing the HSCL-5 and HSCL-10 with the 25-item version. Finally, we will present norm values for the German general population.

Method

Participants

The study sample was recruited as part of the annual representative general population survey conducted by the University of Leipzig. With the assistance of a demographic consulting company (USUMA, Berlin, Germany), we selected a representative sample of participants, using multistage sampling. First, 258 sample point regions from all regions in Germany were randomly drawn from the most recent political election register. The second stage consisted of a random selection of households using the random route procedure. The third stage involved a random selection of household respondents using the Kish selection grid. The aim of the sampling procedure was to obtain a sample that was representative of the German population in terms of age, sex, and education. Inclusion criteria for the current study were of age 14 years or older and the ability to read and understand the German language. All participants were visited by a trained study assistant and informed about the investigation. The participants were provided with self-rating questionnaires.

Out of the 4,069 valid addresses, 2,520 participants took part in the present study. In 539 cases, the household refused to give any information, 441 targeted persons were not willing to participate, in 513 cases no person from the target household could be contacted after three attempts, 45 persons were out of town, and 11 persons were sick. We excluded those participants who had missing values on at least one of the HSCL items (n = 58), leading to a final sample of 2,462 participants (equaling a response rate of 60.5%). The sociodemographic makeup of the sample is described in Table 1. Based on the employed random selection of participants, one would expect the present sample to be fairly representative of the German general population (Jacobsen & Richter, 2019). Comparing the study sample with data from the Federal Statistical Office of Germany (2019), it becomes evident that this is the case for participant age and sex. Namely, there is a 5% mismatch for participant sex, a 12.5% mismatch for participant age. The higher mismatch for age groups is likely due to the underrepresentation of younger participants. The approximated median net income in our sample is 1,723€, compared with 1,827€ in the population—a mismatch of 6%.

Table 1.

Sample Description Based on HSCL-5 and HSCL-10.

	n	%	% in pop.	HSCL-5			HSCL-10
	n	%	% in pop.	Anxiety	Depression	Total	Anxiety	Depression	Total
Sex				F(1, 2460) = 30.917, p < .001, η²_p = .012	F(1, 2460) = 9.370, p = .002, η²_p = .004	F(1, 2460) = 18.267, p < .001, η²_p = .007	F(1, 2460) = 29.695, p < .001, η²_p = .012	F(1, 2460) = 17.145, p = .002, η²_p = .007	F(1, 2460) = 24.531, p < .001, η²_p = .010
Female	1,311	53.2	50.7	2.68 (0.97)	4.20 (1.79)	6.87 (2.57)	5.26 (1.64)	8.11 (3.06)	13.37 (4.41)
Male	1,151	46.8	49.3	2.47 (0.81)	3.99 (1.62)	6.46 (2.20)	4.93 (1.37)	7.63 (2.61)	12.56 (3.64)
Age (in years; M = 50.47; SD =18.57)				F(5, 2456) = 5.304, p < .001, η²_p = .011	F(5, 2456) = 4.965, p < .001, η²_p = .010	F(5, 2456) = 5.893, p < .001, η²_p = .012	F(5, 2456) = 10.030, p < .001, η²_p = .020	F(5, 2456) = 11.831, p < .001, η²_p = .024	F(5, 2456) = 12.960, p < .001, η²_p = .026
14-29	406	16.5	20.4	2.46 (0.89)	3.79 (1.57)	6.25 (2.31)	4.90 (1.58)	7.37 (2.82)	12.28 (4.17)
30-39	323	13.1	14.4	2.50 (0.82)	4.05 (1.68)	6.56 (2.24)	4.88 (1.34)	7.55 (2.62)	12.43 (3.59)
40-49	437	17.7	14.8	2.50 (0.86)	4.02 (1.65)	6.52 (2.29)	4.95 (1.40)	7.57 (2.58)	12.52 (3.64)
50-59	430	17.5	18.5	2.63 (0.90)	4.16 (1.77)	6.79 (2.41)	5.09 (1.49)	7.90 (2.80)	12.99 (3.93)
60-69	407	16.5	13.9	2.64 (0.93)	4.20 (1.78)	6.84 (2.51)	5.25 (1.60)	8.13 (2.98)	13.37 (4.28)
≥70	459	18.6	18.0	2.72 (0.98)	4.33 (1.80)	7.05 (2.57)	5.48 (1.64)	8.65 (3.13)	14.13 (4.44)
Education				F(3, 2458) = 14.844, p < .001, η²_p = .018	F(3, 2458) = 10.601, p < .001, η²_p = .013	F(3, 2458) = 13.699, p < .001, η²_p = .016	F(3, 2458) = 18.034, p < .001, η²_p = .022	F(3, 2458) = 17.393, p < .001, η²_p = .021	F(3, 2458) = 20.206, p < .001, η²_p = .024
≤9 Years	1,135	46.1		2.71 (0.99)	4.27 (1.81)	6.99 (2.58)	5.35 (1.66)	8.30 (3.12)	13.65 (4.44)
10 Years	896	36.4		2.49 (0.83)	4.03 (1.72)	6.52 (2.33)	4.92 (1.41)	7.66 (2.77)	12.58 (3.88)
≥11 Years	330	13.4		2.44 (0.75)	3.83 (1.39)	6.27 (1.95)	4.86 (1.22)	7.32 (1.98)	12.18 (2.91)
School students	101	4.1		2.41 (0.83)	3.55 (1.39)	5.96 (2.05)	4.82 (1.53)	7.05 (2.59)	11.87 (3.88)
Employment status				F(4, 2457) = 18.976, p < .001, η²_p = .030	F(4, 2457) = 33.642, p < .001, η²_p = .052	F(4, 2457) = 33.182, p < .001, η²_p = .051	F(4, 2457) = 21.008, p < .001, η²_p = .033	F(4, 2457) = 40.688, p < .001, η²_p = .062	F(4, 2457) = 38.255, p < .001, η²_p = .059
Working full-time	915	37.2		2.43 (0.75)	3.78 (1.33)	6.21 (1.84)	4.82 (1.21)	7.22 (2.05)	12.04 (2.95)
Working part-time	228	9.3		4.60 (0.90)	3.97 (1.65)	6.57 (2.37)	5.07 (1.56)	7.55 (2.57)	12.62 (3.83)
Unemployed	298	12.1		2.86 (1.16)	4.95 (2.39)	7.81 (3.26)	5.48 (1.92)	9.12 (4.00)	14.60 (5.52)
Retired	832	33.8		2.69 (0.95)	4.28 (1.77)	6.97 (2.51)	5.37 (1.63)	8.44 (3.05)	13.81 (4.36)
In training	189	7.7		2.39 (0.77)	3.63 (1.38)	6.02 (2.01)	4.82 (1.46)	7.10 (2.50)	11.92 (3.71)
Monthly net income				F(5, 2456) = 17.740, p < .001, η²_p = .035	F(5, 2456) = 36.085, p < .001, η²_p = .068	F(5, 2456) = 34.623, p < .001, η²_p = .066	F(5, 2456) = 20.834, p < .001, η²_p = .041	F(5, 2456) = 47.700, p < .001, η²_p = .089	F(5, 2456) = 43.195, p < .001, η²_p = .081
<1,000€	275	11.2		2.94 (1.16)	5.08 (2.33)	8.03 (3.22)	5.78 (1.98)	9.80 (4.00)	15.58 (5.56)
<1,500€	544	22.1		2.73 (1.01)	4.45 (1.93)	7.18 (2.69)	5.36 (1.66)	8.48 (3.15)	13.83 (4.43)
<2,000€	515	20.9		2.50 (0.85)	3.86 (1.52)	6.37 (2.17)	4.94 (1.40)	7.48 (2.43)	12.42 (3.60)
<2,500€	442	18.0		2.43 (0.71)	3.74 (1.29)	6.17 (1.77)	4.83 (1.23)	7.13 (2.03)	11.97 (2.94)
≥2,500€	611	24.8		2.46 (0.77)	3.76 (1.33)	6.22 (1.91)	4.90 (1.31)	7.28 (2.18)	12.18 (3.19)
Refused to answer	75	3.0		2.61 (0.97)	4.40 (1.89)	7.01 (2.75)	5.28 (1.83)	8.64 (3.55)	13.92 (5.16)

Note. HSCL = Hopkins Symptom Checklist; % in pop. = population values according to the Federal Statistical Office of Germany (2019).

Ethics Statement

All participants were informed of the study procedures, data collection and anonymization of all personal data. Additionally, a detailed data privacy statement was delivered by the study assistant. The present study posed a low risk to the participants as procedures such as medical treatments, invasive diagnostics or procedures causing psychological, spiritual or social harm were not included in the present study. Therefore, according to the German law, all participants provided verbal informed consent. For underage participants, parental consent was obtained. Furthermore, the study adhered to the guidelines of the ICC/ESOMAR International Code of Marketing and Social Research Practice.

Measures

The HSCL-25 (Glaesmer et al., 2014; Petermann & Brähler, 2013) assesses symptoms of anxiety and depression using 25 items on a 4-point scale, ranging from not at all to extremely. It consists of two subscales, anxiety (10 items; ω = .856) and depression (15 items; ω = .925), which are calculated by summing up the item scores. The two scale scores can then be aggregated to a total (ω = .942) score assessing psychological distress.

The Patient Health Questionnaire (PHQ-4; Kroenke, Spitzer, Williams, & Löwe, 2009; Löwe et al., 2010) is a brief screening instrument. It uses 2 items each to assess depression (ω = .813) and anxiety (ω = .848). A Total score (ω = .869) is calculated by summing up all items. Response options range from 0 (not at all) to 3 (nearly every day).

The Brief Symptom Inventory (BSI-18; Franke et al., 2017; Petrowski, Schmalbach, Jagla, Franke, & Brähler, 2018) measures symptoms of somatization (ω = .821), depression (ω = .870), and anxiety (ω = .831). Six items per subscale inquire into the extent to which participants suffered from relevant symptoms on a 5-point scale from 0 (not at all) to 4 (extremely). The global severity index (ω = .931) is calculated by summing up all 18 items.

The Jenkins Sleep Scale assesses sleep-related disturbances using four items (Jenkins, Stanton, Niemcryk, & Rose, 1988). Participants rate the frequency of experiencing certain difficulties within the duration of 1 month on a 5-point scale, ranging from 0 (never) to 4 (22-31 days). Internal consistency in the present study was ω = .912.

Analysis Plan

First, we split the study sample into two comparable random samples for the purpose of conducting EFA (n = 1,246) and CFA (n = 1,216) on different samples. For the EFA, we employed three methods. First, we used Principal Axis Factoring with Oblimin rotation in SPSS to obtain factor loadings. Second, we used the Minimum Average Partial test (MAP; Velicer, 1976). Third, we employed Parallel Analysis (PA; Hayton, Allen, & Scarpello, 2004; Horn, 1965). The MAP aims to minimize the average partial correlations between components. On the other hand, PA extracts eigenvalues based on random correlation matrices which are parallel to the empirical data. They are then compared for significant differences. O’Connor (2000) provides a syntax for MAP and PA.

Then, we conducted the CFA using R and the packages lavaan and semTools (Rosseel, 2012; semTools Contributors, 2016). Since the HSCL offers only a 4-point response format, we treated the items as ordinal data. Consequently, we used robust diagonally weighted least squares estimation (Li, 2016). To evaluate goodness of fit, we utilized popular fit indices with commonly recommended cutoff criteria for good fit (Hu & Bentler, 1998, 1999; Schermelleh-Engel, Moosbrugger, & Müller, 2003): The χ²-test which should ideally not be significant; χ² divided by degrees of freedom (χ²/df), which should be smaller than 3; the comparative fit index (CFI) and the Tucker–Lewis index (TLI), which should both be larger than .95 to show good, .90 to show acceptable fit, the root mean square error of approximation (RMSEA) and its 90% confidence interval and the standardized root means square residual (SRMR), which should both be smaller than .05 for a good or .08 for an acceptable fit.

Additionally, we examined the questionnaire’s invariance across sex and age groups by comparing increasingly constrained models (Cheung & Rensvold, 2002; Milfont & Fischer, 2010). First, we constrained factor loadings to be equal to establish weak (or metric) invariance. Second, we additionally constrained intercepts to be equal in order to test for strong (or scalar) invariance. Third, we tested strict invariance by comparing the scalar model with a model that also constrains residuals to be equal across tested groups. As recommended by Milfont and Fischer (2010), we evaluated model comparisons using the χ² test as well as differences in CFI and gamma hat (GH; Steiger, 1989). χ² should ideally not be significant, and CFI and GH should not decline more than .01 between models. In cases where full invariance was not given, we tested for partial invariance, by successively releasing constraints for individual indicators.

As per recommendations from Trizano-Hermosilla and Alvarado (2016), we report McDonald’s (1999) ω as a measure of internal consistency. We differentiate between three types of the coefficient: ω_total, which is comparable to the traditional α as a global measure of internal consistency, ω_hierachical, which indicates the share of variance attributable to a general factor, and ω_subscale, which indicates the proportion of variance traceable to the specific subscale (Rodriguez, Reise, & Haviland, 2016). Additionally, we calculated analyses of variance, comparing both HSCL versions in the sociodemographic groups of the sample.

Results

Descriptive Statistics

We report item and scale descriptive statistics in Table 2. The analysis of skewness and kurtosis suggests nonnormal distributions for the majority of the HSCL’s items, when considering the cutoff values (2 for skewness and 4 for excess kurtosis) provided by West, Finch, and Curran (1995). For both, the HSCL-5 and HSCL-10, all three methods of EFA revealed evidence for a single factor (see Table 3). A majority of the variance (or close to it in the case of the HSCL-10) is traceable to the first latent factor. The lowest partial correlations were and eigenvalues exceeding the randomly generated ones were found for the unifactorial solutions. Factor loadings of all items were in excess of .500. The corrected item-total correlations were larger than the strict .500 cutoff for most items (Hair, Black, Babin, & Anderson, 2010).

Table 2.

Descriptive Statistics of the HSCL-5 and HSCL-10 Items and Scales.

	M	SD	γ₁	γ₂	r_it, HSCL-5			r_it, HSCL-10			HSCL-5	HSCL-10
	M	SD	γ₁	γ₂	Anxiety	Depression	Total	Anxiety	Depression	Total	F1	F1
1. Suddenly scared for no reason, HSCL-10	1.18	0.42	2.39	5.37				.492		.470		.540
2. Feeling fearful, HSCL-5, HSCL-10	1.20	0.47	2.44	5.89	.427		.546	.564		.601	.598	.642
3. Faintness, dizziness, or weakness, HSCL-10	1.36	0.60	1.62	2.22				.476		.583		.614
4. Nervousness or shakiness inside, HSCL-5	1.38	0.60	1.44	1.50	. 427		.591				.671
5. Feeling tense or keyed up, HSCL-10	1.37	0.61	1.52	1.81				.433		.508		.530
6. Blaming yourself for things, HSCL-10	1.23	0.53	2.53	6.46					.581	.611		.674
7. Difficulties falling asleep, staying asleep, HSCL-10	1.50	0.76	1.49	1.60					.488	.524		.543
8. Feeling hopeless about the future, HSCL-5, HSCL-10	1.33	0.68	2.14	4.06		.725	.712		.729	.703	.799	.740
9. Feeling blue, HSCL-5, HSCL-10	1.26	0.58	2.43	5.93		.707	.727		.739	.737	.804	.790
10. Worrying too much about things, HSCL-5	1.50	0.73	1.36	1.18		.628	.664				.731
11. Feeling everything is an effort, HSCL-10	1.33	0.63	1.95	3.47					.659	.678		.724
12. Feeling of worthlessness, HSCL-10	1.23	0.57	2.71	7.24					.720	.695		.758
HSCL-10
Anxiety	1.28	0.38	1.82	3.72
Depression	1.31	0.48	2.16	5.04
Total	1.30	0.41	2.05	4.74
HSCL-5
Anxiety	1.29	0.45	1.80	3.46
Depression	1.37	0.57	2.03	4.17
Total	1.34	0.48	1.96	4.07

Note. HSCL = Hopkins Symptom Checklist; γ₁ = skewness; γ₂ = excessive kurtosis; r_it = corrected item-total correlation with respective subscale; F1 = factor loadings in the exploratory factor analysis.

Table 3.

Results for Minimum Average Partial Test and Parallel Analysis for the HSCL-5 and HSCL-10.

Number of factors	MAP		PA
Number of factors	Squared	Power 4	Raw data	% of Variance	Means	95% CI, upper limit
HSCL-5
0	.2748	.0831
1	.0759	.0087	3.084	61.671	1.079	1.114
2	.1915	.0757	0.640	12.805	1.034	1.059
3	.4325	.2915	0.560	11.201	0.995	1.020
4	—	—	—	—	—	—
HSCL-10
0	.1939	.0459
1	.0258	.0011	4.917	49.166	1.142	1.178
2	.0423	.0063	0.907	9.071	1.100	1.127
3	.0567	.0133	0.867	8.673	1.067	1.089
4	—	—	—	—	—	—

Note. HSCL = Hopkins Symptom Checklist; MAP = Minimum Average Partial test; PA = Parallel Analysis; CI = confidence interval. The lowest average partial correlation and the smallest raw data eigenvalue which is still larger than the upper limit of the 95% CI represent the preferred number of factors.

CFA and Reliability Analysis

Based on the findings from the EFA, we first tested a one-factor solution for both scales. Next, we investigated if a two-factor or a bifactor would be more adequate in representing the data. The results of the CFA can be found in Table 4. All models had significant χ² tests, which is to be expected given the large sample size (Bentler & Bonett, 1980). Apart from that, CFI, RMSEA, and SRMR were acceptable—even good—for all models, while TLI indicated slightly worse, but still acceptable fit. Specifically for the HSCL-5, we found that both the one- and two-factor models are viable solutions and roughly equivalent in fit. Additionally, we constructed a bifactor model—which for the HSCL-25 evinced the best fit (Glaesmer et al., 2014). For the HSCL-5, however, the bifactor model is just-identified and thus not informative, in terms of model fit. But even so, it did not converge. For the 10-item HSCL, on the other hand, the two-factor model—and even more so the bifactor model—exhibited markedly improved fit over the one-factor solution. As a more parsimonious alternative to the bifactor model, we tested a one-factor model that allowed for the errors of Items 1 and 2 to correlate. When comparing the phrasing of all anxiety items, it becomes clear that Items 1 and 2 additionally address aspects of affect, which are not included in the remaining items. This model evinced acceptable fit across all indices, which speaks to the validity of a unifactorial solution. Thus, overall model fit showed a substantial improvement over the 25-item HSCL. We report the factor loadings from all models in Table 5. All loadings were significant, except for some subscale items in the bifactor model. Internal consistency was satisfactory for the one-factor and two-factor models (see Table 6). Merely, the anxiety subscale exhibited a mediocre coefficient. The bifactor model had very good internal consistency for the total score and the depression subscale but also showed moderate reliability for the anxiety subscale. Additionally, the hierarchical and subscale coefficients reveal that more than two thirds of variance can be attributed to a general factor, as opposed to the specific factors, for both subscales, depression (69.6%) and anxiety (70.3%).

Table 4.

CFA Results.

	Model	χ² (df)^a	p	χ²/df	CFI	TLI	RMSEA [90% CI]	SRMR
HSCL-5	One factor	23.915 (5)	<.001	4.783	.980	.961	.056 [.040, .072]	.027
HSCL-5	Two correlated factors	23.825 (4)	<.001	5.956	.979	.949	.064 [.047, .083]	.025
HSCL-10	One factor	212.660 (35)	<.001	6.076	.943	.927	.065 [.048, .082]	.053
	One factor^a	145.537 (34)	<.001	4.281	.964	.953	.052 [.035, .070]	.042
	Two correlated factors	188.069 (34)	<.001	5.531	.950	.934	.061 [.044, .079]	.047
	Bifactor	85.170 (25)	<.001	3.407	.981	.965	.045 [.023, .069]	.028

Note. CFA = confirmatory factor analysis; HSCL = Hopkins Symptom Checklist; df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root means square residual. The bifactor model did not converge for the HSCL-5.

This model allowed for the correlation of the error terms of Items 1 and 2.

Table 5.

Standardized Factor Loadings From the Confirmatory Factor Analyses of All Tested Models.

Items	One factor		Two correlated factors				Bifactor
	HSCL-5	HSCL-10	HSCL-5		HSCL-10		HSCL-10
	G	G	Anx	Dep	Anx	Dep	Anx	Dep	G
1		.457			.488		.459*		.442
2	.604	.650	.622		.693		.547*		.632
3		.614			.646		.039*		.633
4	.618		.641
5		.551			.582		.062*		.564
6		.639				.645		.170	.613
7		.565				.569		−.038*	.602
8	.790	.763		.796		.779		.455	.700
9	.818	.800		.824		.814		.363	.744
10	.737			.740
11		.721				.733		.164	.705
12		.725				.741		.528	.653

Note. HSCL = Hopkins Symptom Checklist; Anx = anxiety factor; Dep = depression factor; G = general factor. The bifactor model did not converge for the HSCL-5.

Factor loading was not significant (at p < .05).

Table 6.

Reliability Coefficients of HSCL-5 and HSCL-10.

	Total	Depression	Anxiety
One-factor model, HSCL-5
α	.836
ω	.846
Two-factor model, HSCL-5
α	.836	.827	.556
ω_total	.850	.825	.568
One-factor model, HSCL-10
α	.879
ω	.882
Two-factor model, HSCL-10
α	.879	.858	.690
ω_total	.887	.858	.696
Bi-factor model, HSCL-10
ω_total	.895	.867	.683
ω_h	.668	.603	.460
ω_s	.197	.264	.222

Note. HSCL = Hopkins Symptom Checklist; ω_h = hierarchical omega; ω_s = subscale omega.

Measurement Invariance

Next, we tested measurement invariance of the two-factor model for both, the HSCL-5 and HSCL-10. To obtain comparable group sizes for the multigroup analysis across age groups, we combined the groups of 14- to 19-year-olds and 20- to 29-year-olds (see the appendix for more information). Tables 7 and 8 presents the results of these analyses. We found evidence for strict invariance across sex groups, but only partial strict invariance across age groups. More precisely, the HSCL-5 can be reasonably expected to be strictly invariant across age groups, while we had to release multiple constraints for the HSCL-10 to exhibit acceptable fit in the more complex models.

Table 7.

Tests of Measurement Invariance of the HSCL-5.

Model	χ² (df)	Δχ²	Δdf	p	CFI	ΔCFI	GH	ΔGH
Sex groups
Configural invariance	60.083 (8)				.974		.992
Female	45.088 (4)				.967		.988
Male	16.973 (4)				.982		.996
Metric invariance	59.034 (11)	1.049	3	.789	.976	.002	.992	.000
Scalar invariance	71.203 (14)	12.169	3	.007	.971	.005	.991	.001
Strict invariance	107.286 (19)	36.083	5	<.001	.955	.016	.986	.005
Partial strict invariance^a	75.641 (18)	4.438	4	.350	.971	.000	.991	.000
Age groups
Configural invariance, years	78.587 (24)				.973		.991
14-29	16.237 (4)				.948		.988
30-39	24.660 (4)				.901		.975
40-49	7.515 (4)				.988		.997
50-59	9.895 (4)				.980		.995
60-69	11.623 (4)				.980		.993
≥70	6.392 (4)				.996		.998
Metric invariance	91.079 (39)	12.492	15	.641	.974	.001	.992	.001
Scalar invariance	171.032 (54)	79.953	15	<.001	.941	.033	.981	.011
Partial scalar invariance^b	108.473 (44)	17.394	5	.004	.968	.006	.990	.002
Strict invariance^b	189.169 (69)	80.696	25	<.001	.940	.028	.981	.009
Partial strict invariance^b,c	136.413 (59)	27.940	15	.022	.961	.007	.988	.002

Note. HSCL = Hopkins Symptom Checklist; df = degrees of freedom; CFI = comparative fit index; GH = gamma hat.

The residual of Item 4 was freed to vary between groups. ^bThe intercepts of Items 4 and 9 were freed to vary between groups. ^cThe residuals of Items 2 and 9 were freed to vary between groups.

Table 8.

Tests of Measurement Invariance of the HSCL-10.

Model	χ² (df)	Δχ²	Δdf	p	CFI	ΔCFI	GH	ΔGH
Sex groups
Configural invariance	417.073 (68)				.946		.972
Female	232.026 (34)				.953		.971
Male	189.526 (34)				.932		.974
Metric invariance	310.980 (76)	106.093	8	<.001	.964	.018	.981	.009
Scalar invariance	361.866 (84)	50.886	8	<.001	.957	.007	.978	.003
Strict invariance	424.968 (94)	63.102	10	<.001	.949	.008	.974	.004
Age groups
Configural invariance, years	676.790 (204)				.927		.963
14-29	63.950 (34)				.965		985
30-39	81.072 (34)				.926		.972
40-49	45.247 (34)				.987		.995
50-59	104.700 (34)				.926		.968
60-69	76.347 (34)				.968		.980
≥70	248.991 (34)				.882		.914
Metric invariance	557.274 (244)	119.516	40	<.001	.952	.025	.975	.012
Scalar invariance	1007.194 (284)	449.92	40	<.001	.888	.064	.945	.030
Partial scalar invariance^a	637.807 (264)	80.533	20	<.001	.942	.010	.971	.004
Strict invariance^a	904.965 (314)	267.158	50	<.001	.909	.033	.954	.017
Partial strict invariance^a,b	729.240 (299)	91.433	35	<.001	.934	.008	.966	.005

Note. HSCL = Hopkins Symptom Checklist; df = degrees of freedom; CFI = comparative fit index; GH = gamma hat.

The intercepts of Items 1, 3, 7, and 9 were freed to vary between groups. ^bThe residuals of Items 1, 3, and 7 were freed to vary between groups.

Convergent Validity.

We correlated the HSCL-5 and HSCL-10 with the full HSCL-25 and other related measures to demonstrate convergent validity (see Table 9). We found very high correlations—in excess of .80 and .90—with the 25-item version, as well as high correlations with the respective subscales of the PHQ-4 and the BSI-18. Additionally, all HSCL versions and subscales had moderately high associations with sleep disturbances as measured by the Jenkins Sleep Scale–4.

Table 9.

Correlation Matrix of Different HSCL-5, HSCL-10, and HSCL-25 With Other Distress Measures.

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17
1. HSCL-5 Anxiety	1	.651	.840	.821	.676	.782	.841	.706	.795	.511	.585	.589	.473	.590	.659	.650	.424
2. HSCL-5 Depression		1	.958	.642	.893	.868	.683	.889	.860	.625	.651	.686	.477	.763	.666	.729	.469
3. HSCL-5 Total			1	.767	.891	.913	.804	.900	.913	.638	.684	.711	.518	.767	.723	.765	.494
4. HSCL-10 Anxiety				1	.696	.863	.915	.734	.841	.503	.552	.567	.563	.584	.670	.682	.451
5. HSCL-10 Depression					1	.963	.742	.959	.930	.626	.642	.682	.556	.779	.706	.777	.572
6. HSCL-10 Total						1	.864	.948	.968	.628	.658	.691	.601	.765	.747	.801	.571
7. HSCL-25 Anxiety							1	.780	.904	.538	.584	.603	.632	.615	.715	.735	.496
8. HSCL-25 Depression								1	.972	.637	.640	.687	.582	.787	.715	.793	.567
9. HSCL-25 Total									1	.635	.654	.693	.632	.766	.754	.815	.571
10. PHQ-4 Depression										1	.728	.932	.510	.721	.630	.710	.439
11. PHQ-4 Anxiety											1	.928	.472	.705	.719	.720	.377
12. PHQ-4 Total												1	.528	.767	.725	.769	.439
13. BSI-18 Somatization													1	.579	.642	.822	.492
14. BSI-18 Depression														1	.793	.911	.431
15. BSI-18 Anxiety															1	.916	.403
16. BSI-18 GSI																1	.498
17. JSS-4																	1

Note. HSCL = Hopkins Symptom Checklist; PHQ-4 = Patient Health Questionnaire–4; BSI-18 = Brief Symptom Inventory–18; JSS-4 = Jenkins Sleep Scale–4: GSI = global severity index. All correlations are significant at the p < .001 level.

Sociodemographic Influences

We tested for differences in the HSCL-5 and HSCL-10’s subscales and total with regard to sociodemographic variables (see Table 1). All comparisons were significant, which is not surprising given the sample size. Sex had only a very small effect in explaining differences in anxiety, depression, and the total score. Age, education, and employment status, on the other hand, exhibited slightly larger effect sizes. The largest effect, however, was traceable to groups of household income, which explained close to 10% of the HSCL-10’s variance.

Norm Values

In Tables 10 to 13, we report percentile ranks partitioned by sex and by age groups.

Table 10.

Percentile Ranks for the HSCL-5 Scales (Female).

Age group	14-29 Years (n = 210)			30-39 Years (n = 165)			40-49 Years (n = 243)			50-59 Years (n = 229)			60-69 Years (n = 209)			≥70 Years (n = 255)
Scale	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total
2	67			60			64			54			53			53
3	83	60		83	55		86	54		87	54		79	47		77	46
4	93	75		92	72		95	77		95	73		93	71		94	62
5	98	86	53	99	81	45	98	87	43	99	83	41	98	82	37	97	76	38
6	100	92	67	100	92	62	100	90	67	100	89	61	100	90	54	100	87	53
7		94	77		93	73		92	82	100	93	73		93	73		93	65
8		97	84		96	81		95	84		96	81		94	80		95	73
9		98	88		98	87		98	88		97	88		98	86		98	80
10		98	92		99	92		99	91		97	91		99	91		99	88
11		99	94		100	93		100	93		100	95		100	92		100	93
12		100	95			94		100	95			95		100	94			95
13			97			95			98			96			96			97
14			98			98			98			97			98			98
15			98			99			99			98			98			98
16			99			100			100			100			99			99
17			99						100			100			100			100
18			100												100
19			100
20			100

Note. HSCL = Hopkins Symptom Checklist; Anx = anxiety factor; Dep = depression factor.

Table 11.

Percentile Ranks for the HSCL-10 Scales (Female).

Age group	14-29 Years (n = 210)			30-39 Years (n = 165)			40-49 Years (n = 243)			50-59 Years (n = 229)			60-69 Years (n = 209)			≥70 Years (n = 255)
Scale	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total
4	54			52			50			46			39			34
5	73			73			76			66			63			58
6	83	50		87	56		86	47		87	45		79	35		73	31
7	89	68		91	71		91	67		92	59		89	55		84	44
8	93	79		95	78		96	77		96	71		94	71		93	56
9	96	86		98	83		97	84		97	79		98	78		97	68
10	99	89	41	100	85	43	99	88	36	99	85	31	98	84	25	99	75	22
11	100	91	56		90	58	100	91	52	99	90	48	99	88	40	99	81	33
12	100	92	65		92	67	100	93	64	100	93	58	100	91	54	100	84	46
13	100	93	75		93	75		95	75		93	67	100	93	62		89	53
14	100	94	80		95	79		95	81		94	74		94	70		93	60
15		95	84		95	82		97	84		97	79		95	77		93	66
16		97	86		96	84		98	87		97	86		96	82		95	71
17		97	88		98	90		98	89		98	89		97	86		97	75
18		98	90		99	92		99	91		99	90		99	88		98	82
19		98	92		99	93		100	91		99	92		100	90		99	84
20		99	93		100	93		100	92		99	93		100	92		100	88
21		99	94			93			95		100	94			92			90
22		99	94			94			96		100	95			94			93
23		99	95			95			98			97			95			93
24		100	96			95			98			97			95			95
25			96			97			98			97			97			96
26			97			98			98			98			98			97
27			97			99			99			98			98			97
28			99			100			99			99			99			99
29			99						99			99			99			100
30			99						100			99			100			100
31			99									99			100
32			99									99
33			99									100
34			99
35			100
36			100
37			100
38			100

Note. HSCL = Hopkins Symptom Checklist; Anx = anxiety factor; Dep = depression factor.

Table 12.

Percentile Ranks for the HSCL-5 Scales (Male).

Age group	14-29 Years (n = 196)			30-39 Years (n = 158)			40-49 Years (n = 194)			50-59 Years (n = 201)			60-69 Years (n = 198)			≥70 Years (n = 204)
Scale	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total
2	78			72			70			61			67			58
3	93	75		95	57		92	61		87	50		85	55		87	53
4	97	89		99	79		98	75		96	73		98	73		95	71
5	100	92	66	100	87	49	99	87	54	100	85	40	100	82	50	99	82	44
6	100	95	83		93	71	100	92	67	100	92	63	100	92	63	100	88	61
7		96	89		96	80	100	94	80	100	95	75		94	75	100	95	73
8		98	92		97	88		99	87	100	97	81		96	81		96	81
9		99	95		98	92		99	92		98	89		98	89		99	85
10		100	96		99	96		100	94		99	93		99	93		99	91
11			97		99	98		100	97		100	97		100	96		100	94
12			98		100	98			98		100	97		100	98		100	96
13			99			99			98			99			98			97
14			99			100			98			99			99			98
15			100						100			100			100			100
16			100						100			100			100			100
17									100			100			100
18												100
19												100
20

Note. HSCL = Hopkins Symptom Checklist; Anx = anxiety factor; Dep = depression factor.

Table 13.

Percentile Ranks for the HSCL-10 Scales (Male).

Age group	14-29 Years (n = 196)			30-39 Years (n = 158)			40-49 Years (n = 194)			50-59 Years (n = 201)			60-69 Years (n = 198)			≥70 Years (n = 204)
Scale	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total	Anx	Dep	Total
4	66			59			55			52			52			39
5	87			84			78			75			73			65
6	93	70		92	53		91	55		88	47		84	46		83	37
7	96	83		96	70		96	69		94	63		92	62		92	56
8	97	90		98	78		98	78		98	75		96	75		96	66
9	98	92		99	86		99	88		98	85		99	79		97	77
10	98	93	55	100	91	45	99	92	42	99	88	35	99	83	35	100	81	27
11	99	93	74		94	58	100	95	56	99	90	52	99	86	50		86	38
12	99	96	82		96	69		96	68	100	94	63	100	91	66		91	55
13	100	97	87		97	75		97	77		95	74		93	71		93	62
14		97	92		98	82		98	86		97	79		94	76		97	72
15		97	93		99	85		98	89		97	82		96	79		98	77
16		97	94		99	92		98	92		98	86		98	82		98	80
17		98	94		99	94		99	93		99	90		98	86		99	84
18		99	95		99	96		99	96		100	93		99	88		100	88
19		99	95		100	97		99	97		100	94		99	91		100	90
20		100	96			98		99	97			95		99	93		100	92
21			97			99		100	97			95		99	94		100	95
22			97			99			98			98		99	96			96
23			97			100			98			99		100	97			97
24			97						99			99			97			98
25			98						99			99			98			99
26			98						99			99			98			99
27			98						99			100			99			100
28			99						99			100			99			100
29			99						99			100			99
30			99						99			100			99
31			99						99						99
32			99						100						99
33			100												99
34															99
35															100
36
37
38

Note. HSCL = Hopkins Symptom Checklist; Anx = anxiety factor; Dep = depression factor.

Discussion

The aim of the present study was to test the merits of two short versions of the HSCL-25: the HSCL-5 and HSCL-10. These two instruments would allow for a more economical assessment of mental health in a variety of contexts. We found good psychometric properties for both versions. The model fit was acceptable, even good. This represents a marked improvement over the 25-item HSCL, which exhibited ambiguous fit (see Glaesmer et al., 2014). Reliability of both scales was comparable to the original, except for the anxiety subscale, which had mediocre internal consistency. Considering that both instruments use just two and four items to measure anxiety, this relatively low reliability comes as no surprise. The hierarchical and subscale ω coefficients for the bifactor model further indicate that a majority of the HSCL-10’s variance can be traced back to a general factor, further justifying the construction of a total score.

The present study found evidence for strict measurement invariance across sex groups and partial strict invariance across age groups. This is an important and novel finding as this level of invariance was not previously shown for the HSCL. For the German HSCL specifically, measurement invariance has not previously been established, which makes this a particularly important finding as meaningful comparisons between groups are not possible without invariance.

The very high correlations with the 25-item version with the shortened HSCL’s point out its capability of adequately capturing anxiety- and depression-related symptoms, even with less than half of the original scale’s items. Correlations between the anxiety and depression subscales were relatively high, albeit slightly lower than for the HSCL-25. Most crucial for evidence of the validity of both HSCL short forms are the high associations of the analogous subscales (short and long HSCL versions) with the PHQ-4 and the BSI-18 demonstrating the measures convergent validity. Finally, we found a moderately high correlation with sleep difficulties, as has been shown previously (Breslau, Roth, Rosenthal, & Andreski, 1996).

As has been previously found by Glaesmer et al. (2014), the HSCL struggles to differentiate between anxiety and depression on a level sufficient for a clinical diagnosis. Regardless of used measures, however, previous research provided sound evidence emphasizing the high comorbidity between the anxiety and depression facets making it almost impossible to differentiate both constructs. Specifically, negative affect has been shown to be the source of shared variance, while bodily hyperarousal is assumed to be specific to anxiety and anhedonia to depression (Renner, Hock, Bergner-Köther, & Laux, 2018). This substantial interplay is also represented in the bifactor model, which exhibits inconsistent factor loadings on the specific factors in addition to explaining a majority of variance by means of the general factor. The adapted one factor model—allowing for a correlation between Items 1 and 2—provides further evidence that the HSCL—at least when the short versions are concerned—actually measures a single characteristic.

In sum, our results indicate that both short versions of the HSCL should mainly be used as brief symptom assessment measures and do not replace an in-depth clinical assessment of anxiety or depression, respectively. The five-item version is recommended as a screening instrument (e.g., in large-scale surveys), and the 10-item questionnaire for a more reliable but still brief assessment of symptoms of depression and anxiety (e.g., experimental setting). As displayed in the norm value tables, younger female individuals (<30 years) reach the maximum score range in HSCL-5 (20 of 20) and almost in HSCL-10 (38 of 40). Lower maximum range scores were reported for the corresponding male population and for all other older female and male age groups. In general, female individuals need to report higher HSCL-scores (for HSCL-5: ≥11, for HSCL-10: ≥21) to be classified to the upper 5% of the distribution compared with male individuals (for HSCL-5: ≥6, for HSCL-10: ≥18).

Comparisons between sociodemographic groups evinced a small to moderate influence of group membership on symptom severity. In particular, employment status and monthly income appeared to be significant predictors of psychological distress. This fits well with previous research demonstrating a link between socioeconomic status and mental health (Barrett & Turner, 2005; Williams, Yu, Jackson, & Anderson, 1997).

Limitations

The present study used a survey that employed the HSCL-25 to test two short versions. A cross-validation using only the respective final 5 and 10 items should complement the present findings.

Additionally, an investigation of the HSCL-5 and HSCL-10 in children, adolescents and young adults is still pending. The analyses of the present study focused on a representative sample of the German adult population, and only a small number of respondents were between the ages of 14 and 19 years (n = 137). Future research should examine the suitability of the HSCL as a psychometric instrument for younger populations.

Conclusion

The HSCL-5 and HSCL-10 are reliable and valid short forms of the HSCL-25. Both should be preferred for the economical assessment of symptoms of depression and anxiety. We recommend the 5-item version as a screening instrument, and the 10-item questionnaire for a more reliable but still brief assessment of symptoms of depression and anxiety.

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Bjarne Schmalbach

Ana Nanette Tibubos

References

Anderson

R. J.

Freedland

K. E.

Clouse

R. E.

Lustman

P. J.

(2001). The prevalence of comorbid depression in adults with diabetes: A meta-analysis. Diabetes Care, 24, 1069-1078.

Bandelow

Michaelis

(2015). Epidemiology of anxiety disorders in the 21st century. Dialogues in Clinical Neuroscience, 17, 327-335.

Barrett

A. E.

Turner

R. J.

(2005). Family structure and mental health: The mediating effects of socioeconomic status, family process, and social stress. Journal of Health and Social Behavior, 46, 156-169.

Bentler

P. M.

Bonett

D. G.

(1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588-606.

Brady

E. U.

Kendall

P. C.

(1992). Comorbidity of anxiety and depression in children and adolescents. Psychological Bulletin, 111, 244-255.

Breslau

Roth

Rosenthal

Andreski

(1996). Sleep disturbance and psychiatric disorders: A longitudinal epidemiological study of young adults. Biological Psychiatry, 39, 411-418.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233-255.

Federal Statistical Office of Germany. (2019). Bevölkerung [Population]. Retrieved from https://www.destatis.de/DE/Themen/Gesellschaft-Umwelt/Bevoelkerung/_inhalt.html

Franke

G. H.

Jaeger

Glaesmer

Barkmann

Petrowski

Braehler

(2017). Psychometric analysis of the brief symptom inventory 18 (BSI-18) in a representative German sample. BMC Medical Research Methodology, 17, 14. doi:10.1186/s12874-016-0283-3

10.

Glaesmer

Braehler

Grande

Hinz

Petermann

Romppel

(2014). The German Version of the Hopkins Symptoms Checklist-25 (HSCL-25): Factorial structure, psychometric properties, and population-based norms. Comprehensive Psychiatry, 55, 396-403.

11.

Hair

Black

Babin

Anderson

(2010). Multivariate data analysis. Upper Saddle River, NJ: Prentice Hall.

12.

Hayton

J. C.

Allen

D. G.

Scarpello

(2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191-205.

13.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179-185.

14.

L. T.

Bentler

P. M.

(1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3, 424-453.

15.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1-55.

16.

Jacobi

Wittchen

H. U.

Hölting

Höfler

Pfister

Müller

Lieb

(2004). Prevalence, co-morbidity and correlates of mental disorders in the general population: Results from the German Health Interview and Examination Survey (GHS). Psychological Medicine, 34, 597-611.

17.

Jacobsen

Richter

(2019). Gibt es repräsentative Umfragen? [Do representative surveys exist?]. Psychotherapie Psychsomatik Medizinische, 69, 203-204. doi:10.1055/a-0859-8631

18.

Jenkins

C. D.

Stanton

B. A.

Niemcryk

S. J.

Rose

R. M.

(1988). A scale for the estimation of sleep problems in clinical research. Journal of Clinical Epidemiology, 41, 313-321.

19.

Kessler

R. C.

Bromet

E. J.

(2013). The epidemiology of depression across cultures. Annual Review of Public Health, 34, 119-138.

20.

Kroenke

Spitzer

R. L.

Williams

J. B.

Löwe

(2009). An ultra-brief screening scale for anxiety and depression: The PHQ-4. Psychosomatics, 50, 613-621.

21.

C. H.

(2016). The performance of ML, DWLS, and ULS estimation with robust corrections in structural equation models with ordinal variables. Psychological Methods, 21, 369-387.

22.

Löwe

Wahl

Rose

Spitzer

Glaesmer

Wingenfeld

. . . Brähler

(2010). A 4-item measure of depression and anxiety: Validation and standardization of the Patient Health Questionnaire-4 (PHQ-4) in the general population. Journal of Affective Disorders, 122, 86-95.

23.

McDonald

(1999). Test theory: A unified treatment. Mahwah, NJ: Lawrence Erlbaum.

24.

Milfont

T. L.

Fischer

(2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111-121.

25.

O’Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, 32, 396-402.

26.

Parloff

M. B.

Kelman

H. C.

Frank

J. D.

(1954). Comfort, effectiveness, and self-awareness as criteria of improvement in psychotherapy. American Journal of Psychiatry, 111, 343-352.

27.

Petermann

Brähler

(2013). Hopkins-Symptom-Checkliste-25: Deutsche version. Göttingen, Germany: Hogrefe.

28.

Petrowski

Schmalbach

Jagla

Franke

G. H.

Brähler

(2018). Norm values and psychometric properties of the brief symptom inventory-18 regarding individuals between the ages of 60 and 95. BMC Medical Research Methodology, 18, 164. doi:10.1186/s12874-018-0631-6

29.

Prinz

Nutzinger

D. O.

Schulz

Petermann

Braukhaus

Andreas

(2008). Die Symptom-Checkliste-90-R und ihre Kurzversionen: Psychometrische Analysen bei Patienten mit psychischen Erkrankungen [The Symptom-Checklist-90-R and its short versions: Psychometric analyses with patients with psychological disorders]. Physikalische Medizin, Rehabilitationsmedizin, Kurortmedizin, 18, 337-343.

30.

Renner

K. H.

Hock

Bergner-Köther

Laux

(2018). Differentiating anxiety and depression: The state-trait anxiety-depression inventory. Cognition & Emotion, 32, 1409-1423.

31.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21, 137-150.

32.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. Retrieved from http://www.jstatsoft.org/v48/i02/

33.

Roy-Byrne

P. P.

Stang

Wittchen

H. U.

Ustun

Walters

E. E.

Kessler

R. C.

(2000). Lifetime panic–depression comorbidity in the National Comorbidity Survey: Association with symptoms, impairment, course and help-seeking. British Journal of Psychiatry, 176, 229-235.

34.

Rudisch

Nemeroff

C. B.

(2003). Epidemiology of comorbid coronary artery disease and depression. Biological Psychiatry, 54, 227-240.

35.

Schermelleh-Engel

Moosbrugger

Müller

(2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23-74.

36.

semTools Contributors. (2016). semTools: Useful tools for structural equation modeling (R package version 0.4-14). Retrieved from https://CRAN.R-project.org/package=semTools

37.

Shrout

P. E.

Yager

T. J.

(1989). Reliability and validity of screening scales: Effect of reducing scale length. Journal of Clinical Epidemiology, 42, 69-78.

38.

Skogen

J. C.

Øverland

Smith

O. R.

Aarø

L. E.

(2017). The factor structure of the Hopkins Symptoms Checklist (HSCL-25) in a student population: A cautionary tale. Scandinavian Journal of Public Health, 45, 357-365.

39.

Steiger

J. H.

(1989). EzPATH: A supplementary module for SYSTAT and SYGRAPH. Evanston, IL: Systat.

40.

Strand

B. H.

Dalgard

O. S.

Tambs

Rognerud

(2003). Measuring the mental health status of the Norwegian population: A comparison of the instruments SCL-25, SCL-10, SCL-5 and MHI-5 (SF-36). Nordic Journal of Psychiatry, 57, 113-118.

41.

Trizano

H. I.

Alvarado

J. M.

(2016). Best alternatives to Cronbach’s alpha reliability in realistic conditions: congeneric and asymmetrical measurements. Frontiers in Psychology, 7, 769. doi: https://doi.org/10.3389/fpsyg.2016.00769

42.

Velicer

W. F.

(1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41, 321-327.

43.

West

S. G.

Finch

J. F.

Curran

P. J.

(1995). Structural equation models with nonnormal variables: Problems and remedies. In Hoyle

R. H.

(Ed.), Structural equation modeling: Concepts, issues and applications (pp. 56-75). Thousand Oaks, CA: Sage.

44.

Williams

D. R.

Jackson

J. S.

Anderson

N. B.

(1997). Racial differences in physical and mental health: Socio-economic status, stress and discrimination. Journal of Health Psychology, 2, 335-351.

45.

Wittchen

H. U.

Jacobi

Rehm

Gustavsson

Svensson

Jönsson

. . . Fratiglioni

(2011). The size and burden of mental disorders and other disorders of the brain in Europe 2010. European Neuropsychopharmacology, 21, 655-679.