Cross-Cultural Validity of the Ruminative Responses Scale in Argentina and the United States

Abstract

Although frequently used in the United States, the Ruminative Response Scale (RRS) has not been extensively studied in cross-cultural samples. The present study evaluated the factor structure of Treynor et al.’s 10-item version of the RRS in samples from Argentina (N = 308) and the United States (N = 371). In addition to testing measurement invariance between the countries, we evaluated whether the maladaptive implications of rumination were weaker for the Argentinians than for the U.S. group. Self-critical perfectionism was the criterion in those tests. Partial scalar invariance supported an 8-item version of the RRS. There were no differences in factor means or factor correlations in RRS dimensions between countries. Brooding and Reflection were positively correlated with self-critical perfectionism in both countries, with no significant differences in the sizes of these relations between the two samples. Results are discussed in terms of psychometric and cross-cultural implications for rumination.

Keywords

rumination cross-cultural measurement invariance perfectionism Argentina

The psychological construct of rumination is considered one of the key processes in the mechanisms of depression (Nolen-Hoeksema, Wisco, & Lyubomirsky, 2008). Defined as an emotional regulation coping style, rumination involves repetitive thinking about symptoms of distress and the meaning of those symptoms (Nolen-Hoeksema, 1991). According to Conway, Csank, Holm, and Blake (2000), ruminative thoughts are not goal-directed and thus do not help solve sadness-related problems. Research has consistently shown that rumination is associated with negative outcomes such as depressive and anxious symptomatology, bulimia, substance abuse, externalizing problems, and posttraumatic stress disorder (e.g., Young & Dietrich, 2015). Indeed, these associations with diverse forms of psychopathology have led authors to propose rumination as a transdiagnostic process (McLaughlin, Wisco, Aldao, & Hilt, 2014; Nolen-Hoeksema & Watkins, 2011). In fact, the transdiagnostic implications of rumination may go beyond strictly psychological phenomena given research finding links between rumination and distal physical outcomes such as migraines (Kokonyei et al., 2016), slowed heart rate recovery (Roger & Jamieson, 1988), and increased blood pressure (Key, Campbell, Bacon, & Gerin, 2008).

The Ruminative Response Scale (RRS) is one of the most widely used measures of rumination (Luminet, 2004). The initial version of the RRS was the Response Styles Questionnaire (RSQ), with 71 items measuring ruminative, distracting, problem-solving, and dangerous activities (Nolen-Hoeksema & Morrow, 1991). The RRS was subsequently developed from the rumination subscale of the RSQ and comes in two forms: a 21-item original form (Butler & Nolen-Hoeksema, 1994) and a 10-item short form (Treynor, Gonzalez, & Nolen-Hoeksema, 2003). This latter shortened version was designed in an attempt to reduce confounding between item content measuring rumination and depression (Treynor et al., 2003). This effort resulted in rumination being separated into dimensions of brooding and reflection. Brooding involves passive repetitive thinking about causes of depression or sadness, whereas reflection focuses on a seemingly constructive way on dealing with causes of sadness (Roberts, Gilboa, & Gotlib, 1998; Schoofs, Hermans, & Raes, 2010; Treynor et al., 2003). Consequently, brooding has consistently related to maladaptive outcomes but findings regarding reflection have been more mixed (Miranda & Nolen-Hoeksema, 2007; Watkins, 2008); for instance, reflection has produced low-magnitude associations with negative outcomes much more often than stronger associations with positive outcomes (e.g., Burwell & Shirk, 2007). The two-factor structure of rumination has been replicated with the RRS (e.g., Whitmer & Gotlib, 2011) as well as with other measures of rumination (e.g., Tanner, Voon, Hasking, & Martin, 2013). For example, Siegle, Moore, and Thase (2004) supported the two-factor structure based on 16 measures of rumination completed by healthy and depressed adults.

There has been preliminary support for the cross-cultural utility of the RSQ and RRS given its successful translation and adaptation into German (Kuehner & Weber, 1999), Turkish (Erdur-Baker & Bugay, 2010), Japanese (Sakamoto, Kambara, & Tanno, 2001), Korean (Lee & Kim, 2014), Dutch (Raes, Hermans, & Eelen, 2003), and Spanish (Chile: Cova, Rincón, & Melipillán, 2007; Spain: Hervas, 2008). However, cross-cultural validity has not been fully tested for the RRS (Erdur-Baker & Bugay, 2010). Indeed, in a broader way, cross-cultural findings in rumination research are scarce and have mostly relied on comparisons between North America/Western Europe and Asian countries as representatives of individualistic versus collectivistic nations. Evidence thus far reveals higher levels of rumination in non-Western or collectivistic cultures compared with Western or individualistic cultures (e.g., Sakamoto et al., 2001). For example, Jose, Kramer, and Hou (2014) have recently found that Chinese adolescents reported more rumination than New Zealand adolescents. Rumination in Asian countries may be partially motivated by a desire to properly function within others (Hong et al., 2010; Jose et al., 2014) and therefore implemented to obtain a goal of interpersonal harmony. Chang, Tsai, and Sanna (2010) demonstrated that Asian Americans ruminate more than European Americans, but results for Asian Americans also revealed weaker associations with measures of psychological functioning, operationalized as affectivity, depressive symptoms, and anxious symptoms. From a cross-national approach, Grossmann and Kross (2010) have found that Russian students exhibited the same pattern of more rumination but less maladjustment (i.e., depressive symptoms and general distress) compared with U.S. students. A rationale for this difference can be found in Cohen, Hoshino-Browne, and Leung (2007), who explain that members of non-Western or collectivistic cultures tend to self-distance when thinking about negative experiences compared with members of Western/individualistic cultures. Kross, Ayduk, and Mischel (2005) hypothesized that people’s attempts to analyze negative experiences fail due to a self-immersed view rather than a self-distanced perspective. Studies consistently indicated that individuals who were cued to adopt a self-distanced perspective tend to display less negative affect in the short term (Gruber, Harvey, & Johnson, 2009; Kross & Ayduk, 2008). Interestingly, psychological distance seems to play a key role in distinguishing between adaptive versus maladaptive forms of self-reflection (i.e., reflection and brooding in the Treynor et al. RRS). Thinking about one’s negative experiences from the perspective of a “fly on the wall” (self-distanced) seems to be far better in terms of adjustment than adopting a self-immersed view (first person, self-immersed perspective). That said, it would be predicted that those in non-Western/more collectivistic/less individualistic nations should exhibit higher levels of reflection or, at least, low levels of brooding compared with those in nations whose people are characterized by high levels of individualism and have difficulties trying to self-distance from negative experiences (Ayduk & Kross, 2010).

Overall, although a link has traditionally been established between Western values and rumination, evidence from non-Western countries and, specifically, a Latin American country such as Argentina, is lacking. There is some evidence positioning Argentina as a less extreme collectivistic nation compared with Asian countries. Chiou (2001) found that Argentine and Taiwanese students were more collectivistic than participants from the United States, but the Argentina sample was more horizontally individualistic than participants in the Taiwan sample. This means that Argentinians may endorse personal goals and independence more so than other collectivistic countries, but at the same time, Argentinians exist in a cultural orientation that emphasizes interdependence, communal over personal goals, and equality over hierarchy. Despite some leanings in the direction of individualism, Argentina seems to be quite far from traditional Western values in the United States (Hofstede, Hofstede, & Minkov, 2010). This middle-range status of Argentina with regard to individualistic/collectivistic dimensions might result in different cross-cultural outcomes compared with other samples. Nevertheless, to our knowledge, no prior study has explicitly evaluated the hypothesis that Argentinians ruminate more but with less maladjustment than those in the United States.

Before we can directly compare means and correlations between countries, fundamental questions regarding the comparability of measurement models must be addressed. This is the reason why we adopted a measurement invariance (MI) approach to compare the factor structure of the RRS across nations. In brief, MI applies increasingly demanding constraints on measurement models to assess whether these models retain their fit under progressively more rigorous demands for parameter estimates to be invariants between groups. If not, if noninvariance is found, inferences based on further comparisons among groups would be statistically compromised (Chen, 2008). Testing MI involves at least three steps. First, a baseline invariance model simply checks that the measurement model with freely estimated parameters within- and between-groups provides a reasonably good representation of the data. Next, metric invariance constrains item-to-factor loadings to be invariant between groups. Metric invariance means that the unit of measurement is the same between groups, and this level of measurement is required when comparing the strengths of association between the construct of interest and some criterion. Scalar invariance adds the constraint that item intercepts are also equal between the groups. Tests of factor mean differences between groups are predicated on support for scalar invariance. For most applications, metric and scalar invariance are sufficient (Brown, 2015).

We expected MI would be supported for the two-factor structure of the RRS between samples and, thus, rigorous support would exist to test subsequent comparisons. Reasons to expect invariance came from evidence of the factor structure of the RRS being replicated in other non-Western cultures (e.g., Sakamoto et al., 2001) and in neighboring countries such as Chile (Cova et al., 2007). Second, considering culture differences, we expected a higher mean value of brooding for Argentine students over U.S. students. Directional hypotheses did not seem plausible for the reflection factor for conceptual and statistical reasons; we knew of no prior work or theory that would lead to an expectation of higher or lower reflection in one of the countries, and the psychometric value and reliability of reflection scores are still in question (Armey et al., 2009; Whitmer & Gotlib, 2011).

Third, a lower magnitude association was expected between brooding and a negative criterion variable for the Argentine students than for the U.S. students. Self-critical perfectionism, in its form of evaluative concerns (Stoeber & Otto, 2006), was selected as a negative criterion because this personality trait has been consistently linked to depression (see Limburg, Watson, Hagger, & Egan, 2017, for a meta-analysis) as well as to rumination (Harris, Pepper, & Maack, 2008; James, Verplanken, & Rimes, 2015). Indeed, several authors have proposed that rumination mediates the link between the perfectionism–depression association (Nepon, Flett, Hewitt, & Molnar, 2011; Van Der Kaap-Deeder et al., 2016). In addition, levels of perfectionistic concerns have reached partial invariance between Argentina and the United States (Arana, Rice, & Ashby, 2018), so future comparisons with rumination factors may be highly reliable.

Summarizing, the first aim of this study was to test the MI of the RRS across Argentine and U.S. students. We expected to find at least a certain degree of scalar invariance (i.e., partial scalar invariance). If so, the second aim was to test the expectation of more rumination but less maladjustment for Argentina as a non-Western/less individualistic country compared with the United States. We expected higher mean levels of brooding in Argentina than in the United States, but we also expected a greater estimated correlation of brooding and perfectionistic concerns (self-critical perfectionism) in the United States than in Argentina.

Method

Participants

The Argentina sample was composed of 308 undergraduate students (235 women, 73 men) with a mean age of 25.42 years (SD = 5.33). These students were recruited from the School of Psychology at a large national university in Argentina. The U.S. sample was composed by 371 students (315 women, 56 men) with a mean age of 19.47 (SD = 1.47). These participants were enrolled in psychology courses at a large, public university in the southeastern United States. There were proportionally more men in the Argentina sample compared with the U.S. sample, χ²(1, N = 679) = 8.10, p < .004. Argentine participants were also older than U.S. participants, t(668) = −20.34, p < .001, Cohen’s d = 1.52. Age and gender were used as covariates in the measurement model to see differences. The race/ethnic breakdown for the Argentine sample was 100% Latin American (97% European heritage). For the U.S. sample, the racial/ethnic distribution was 53% (White, European American), 14% (Black, African American), 14% (Hispanic, Latino), 13% (Asian or Asian American), 5% (multicultural mixed race), and 1% (Pacific Islander).

Measures

Rumination

To measure rumination, we selected the 10-item version of the RRS proposed by Treynor et al. (2003). The scale presents two subscales, named Brooding and Reflection, with five self-report items each, with responses on a 4-point scale (1 = almost never through 5 = almost always). A sample Reflection item is, “Write down what you are thinking and analyze it,” whereas a sample Brooding item is, “Why do I have problems other people don’t have?” Cronbach’s coefficients alpha were .72 for the Reflection score and.77 for the Brooding score (Treynor et al., 2003). In the current study, the original RRS was used in the U.S. sample, whereas the Argentine students completed an adaptation of the Spanish versions of the RRS for use in the Argentina context. The adaptation was created following Hambleton and Zenisky’s (2011) suggestions of translating and adapting psychological tests. Specifically, we compared the Chilean and Spanish versions to a new translated version from the original English-language scale. Then, we selected the most accurate wording of each item with regard to the parlance used by young Argentinians.

Self-Critical Perfectionism

To measure self-critical perfectionism, we used the Discrepancy subscale from the Short Almost Perfect Scale (Rice, Richardson, & Tueller, 2014), a brief, four-item measure derived from the longer Almost Perfect Scale–Revised (Slaney, Rice, Mobley, Trippi, & Ashby, 2001; Argentine version: Arana, Keegan, & Rutsztein, 2009). Discrepancy is an indicator of perfectionistic concerns and is specifically intended to measure the perceived gap between actual and ideal standards of performance. Consequently, higher scores on Discrepancy represent higher levels of self-critical perfectionism. Items on the Self-Assessment of Perfectionism Subtypes (SAPS) are responded to using a 7-point scale (1 = strongly disagree through 7 = strongly agree). A sample Discrepancy item is, “Doing my best never seems to be enough.” The SAPS has demonstrated excellent psychometrical properties (Rice et al., 2014), including partial scalar invariance and good reliability in a prior study of U.S. (ρ = .90) and Argentine students (ρ = .92; Arana et al., 2018).

Procedure

Approval from each university’s committee on research with human subjects was obtained and informed consent was required from each participant in both countries. After consent, participants in Argentina completed a brief demographics questionnaire (e.g., age, sex) followed by the SAPS and the RRS. The U.S. sample also completed the SAPS and the RRS, followed by a similar demographic questionnaire. Students received modest course credit (United States) or no credit (Argentina) for participation. Participants in both countries completed the measures in-person, during small group participation sessions held in classrooms.

Data Analyses

A main set of analyses was selected to: (a) compare the factor structure and reliability of the RRS factors between Argentina and U.S. samples, (b) examine sex and age as potential covariates in the measurement model, and (c) compare factor mean levels and criterion-related validity between the samples based on associations between brooding and self-critical perfectionism. Analyses were conducted with IBM SPSS Version 23 (2015) and Mplus Version 7.4 (Muthén & Muthén, 1998-2015) using a robust maximum likelihood estimator. Full information maximum likelihood is used in Mplus to address missing data and generate unbiased parameter estimates. We used some general guidelines to interpret fit indices but attempted to view those results critically in light of other model results. For example, acceptable fit could be reflected in values for the comparative fit index (CFI) in the .90 range (Byrne, 1998), root mean square error of approximation (RMSEA) values less than .08 (Browne & Cudeck, 1993), and standardized root mean square residual (SRMR) values of .08 or less (Hu & Bentler, 1999). As Brown (2015) points out, it is best to consider optimal fit across different indices and “in tandem with the particular aspects of the analytic situation” (p. 74), such as sample size, number of indicators, and interpretability of model results. Assessment of cross-sectional MI between the Argentina and U.S. samples involves a series of nested model comparisons to determine if imposing invariance constraints significantly worsen model fit over allowing parameters to be freely estimated between groups. Significant differences in Yuan–Bentler scale-corrected chi-square values suggest noninvariance, although chi-squares are affected by sample size. Thus, other fit indices were also examined. Failed invariance can be signaled by decreases of more than .01 in CFIs between models (Chen, 2008; Cheung & Rensvold, 2002), differences greater than .015 in the RMSEA, and differences between .01 and .03 in the SRMR (Chen, 2008). In addition, a multiple indicator multiple causes model was used to evaluate the effects of sex and age in the measurement model.

Results

Cross-Sectional Measurement Invariance

We began by testing a traditional confirmatory factor analysis (CFA) of the 2-factor, 10-item version of the RRS separately for each country. Fit indices for the U.S. sample were χ²(34, N = 363) = 147.98, p < .0001, CFI = .911, RMSEA = .096 (95% CI [.081, .112]), SRMR = .051. In turn, model fit for the Argentine sample was χ²(34, N = 308) = 107.49, p < .0001, CFI = .871, RMSEA = .084 (95% CI [.066, .102]), SRMR = .061. Similarly, the two-sample CFA of this configural model based on 10 RRS items produced less than desired fit (e.g., RMSEA = .91; see Table 1 for fit results and model comparisons). Modification indices suggested (i.e., χ² values higher than 3.84) that error variances for Items 02 (“Analyze recent events to try to understand why you are depressed”) and 09 (“Analyze your personality to try to understand why you are depressed”) could be correlated. Although correlating item errors seemed reasonable from a psychometric view, the depression content of these items suggested a clearer representation of rumination factors would be obtained by excluding the items (Armey, et al., 2009). Thus, we proceeded by excluding Items 02 and 09 and subsequently analyzed an eight-item solution. The revised configural model was estimated with five items loading on the Brooding factor and three items loading on the Reflection factor. As shown in Table 1, the two-sample CFA for that model produced fit statistics within traditional benchmarks (CFI > .90, RMSEA < .08, SRMR < .08), indicating that metric invariance could be tested next.

Table 1.

Comparing the Argentina and U.S. Samples on the Two-Factor Ruminative Response Scale Invariance Models.

Model	χ²	df	CFI	RMSEA [90% CI]	SRMR	ΔCFI	ΔRMSEA	Adjustments
M0 (10-item) (configural)	251.71	68	.900	.091 [.079, .103]	.056
M1 (8-item) (configural)	85.41	38	.961	.061 [.044-.078]	.042			Drop Items 02 and 09
M2 (metric)	106.93	44	.949	.065 [.050, .081]	.057	−.012	.004
M2p (partial metric)	97.96	43	.955	.062 [.046, .078]	.052	−.006	.001	Free loading Item 01
M3 (scalar)	176.58	49	.896	.088 [.074, .102]	.064	−.059	.026
M3p (partial scalar)	120.22	47	.940	.068 [.053, .083]	.050	−.015	.006	Free intercepts Items 01 and 03

Note. df = degrees of freedom; CFI = comparative fit index; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual. All χ² were significant, p < .0001.

For metric invariance, factor loadings were constrained to be invariant between the samples. Although model fit was acceptable, constraining factor loadings substantially worsened fit compared with the configural model (e.g., ΔCFI = −.012). We tested each of the RRS items and determined that allowing the loading for Item 01 (“Think ‘What am I doing to deserve this?’”) to be freely estimated, while constraining the remaining item loadings to be invariant, produced a negligible change in fit and therefore supported partial metric invariance (see Table 1). Scalar invariance was then tested against the partial metric invariance model. The inclusion of constraints to the intercepts of the items substantially worsened the measurement model (e.g., ΔCFI = −.059), indicating that some items had different averages in each country. Again, several modifications were explored. At best, a partial scalar invariance model could be derived by allowing intercepts for Item 01 and Item 03 (Think “Why do I always react this way?”) to be freely estimated between the countries (these items also had the pronounced intercept differences based on the configural model; see Table 2).

Table 2.

Sample Comparisons of 8-Item Ruminative Response Scale and Discrepancy Subscale Factor Loadings and Intercepts Based on Configural Invariance Model.

Factor/items	USA				Argentina
Factor/items	B	SE	β	Intercepts	B	SE	β	Intercepts
Ruminative Response Scale
Brooding
RR01	1.00	0.00	0.57	1.90	1.00	0.00	0.30	1.59
RR03	1.25	0.13	0.74	2.00	2.86	0.68	0.67	2.32
RR06	1.11	0.12	0.70	2.57	1.73	0.38	0.44	2.34
RR07	1.25	0.12	0.70	1.90	1.85	0.47	0.48	1.75
RR08	1.42	0.16	0.80	2.23	3.23	0.78	0.79	2.31
Reflection
RR04	1.00	0.00	0.96	2.11	1.00	0.00	0.88	2.04
RR05	0.32	0.06	0.38	1.51	0.29	0.11	0.31	1.46
RR10	0.75	0.06	0.77	2.08	0.58	0.14	0.53	2.04
Short Almost Perfect Scale
Discrepancy
APS11	1.00	0.00	0.78	3.41	1.00	0.00	0.68	4.02
APS16	0.98	0.06	0.86	3.42	1.67	0.19	0.81	3.64
APS20	0.93	0.06	0.85	3.08	1.15	0.14	0.63	4.32
APS23	0.78	0.06	0.66	4.18	1.48	0.15	0.72	3.88

Note. SE = standard error; B = unstandardized factor loadings; β = standardized factor loadings. All loadings were significant, p < .0001.

To examine structural invariance, the partial scalar model was compared with a model that constrained factor variances and the covariance between RRS factors to be invariant between the countries. That model produced these fit indices: χ²(50, N = 671) = 126.67, p < .0001, CFI = .938, RMSEA = .068 (95% CI [.053, .082]), SRMR = .070. Results indicated no differences in factor variances and covariance between the countries, Δχ²(3, N = 671) = 6.20, p = .102, ΔCFI = −.002, ΔRMSEA = .000, ΔSRMR = .020. Factor mean differences for Brooding and Reflection were based on comparing the estimated factor means of the Argentina sample to the U.S. reference group factor means (M = 0). The Argentina sample had a slightly lower Brooding factor mean (−.057) that was not significantly different from the U.S. sample, p = .360. The same pattern was followed by the Reflection factor mean (−.072) indicating no statistically significant differences between countries, p = .321. Next, we calculated internal consistency following Raykov’s (2009) structural modeling approach which, unlike Cronbach’s coefficient alpha, does not require the assumption of tau equivalence or equal loadings for all indicators of a factor (MacDougall, 2011; Yang & Green, 2011). Reliability for Brooding was ρ = .830 (95% CI [.800, .860]) for the U.S. sample and ρ = .690 (95% CI [.634, .747]) for the Argentina sample. Reliability for Reflection was ρ = .792 (95% CI [.756, .829]) for the U.S. sample and ρ = .643 (95% CI [.569, .718]) for the Argentina sample.

Covariate Effects on the Measurement Model

Because the two samples differed in terms of age and sex, we used a multiple indicator multiple causes model to assess their effects as covariates on item responses (i.e., possible bias) in the measurement model. Following Brown’s (2015) procedures, we regressed the RRS factors and their item indicators on age and sex, and stipulated that the associations among the items and the covariates be constrained to zero. The modification indices could then be examined to determine if the model would be significantly improved by relaxing one or more of those constrained associations (see Brown, 2015, for a worked example). The addition of age and sex to the partial scalar measurement model produced these overall fit indices: χ²(73, N = 671) = 158.50, p < .0001, CFI = .936, RMSEA = .059 (95% CI [.047, .072]), SRMR = .049. Modification indices suggested Items 06 (“Think about a recent situation, wishing it had gone better”) and 08 (“Think ‘Why can’t I handle things better?’”) might be related with age in both samples. Nevertheless, when we released the constraints on those items, only a negligible albeit significant association between age and Item 08 was found (.009, p = .004), and only in the Argentina group. There were no other indications of potential bias in item responses. In terms of associations with the factors, in the Argentina sample, a significant association was found between sex and Brooding (−.28, p = .003); the same effect in the U.S. sample was not significant (p = .968). No other significant associations were observed. Additionally, we ran an independent t test separately on each country, which confirmed Brooding differed by sex in Argentina, t(301) = 3.05, p = .003, Cohen’s d = 0.41, but not in the United States, t (357) = 0.89, p = 375, Cohen’s d = 0.15 (the United States). Thus, women in Argentina endorsed higher levels of Brooding than men but otherwise sex and age did not appear to affect the items or factors.

Criterion-Related Validity

Before we compared the correlations between rumination factors and perfectionism, we evaluated MI for the SAPS Discrepancy subscale. As can be seen in Table 1, partial metric invariance was reached by releasing the factor loading of one of the four items (11: Doing my best never seems to be enough). Because at least metric invariance is needed to compare factor correlations, next we compared the correlations between the RRS factors and the Discrepancy factor by simply including the partial metric invariance model for Discrepancy within the partial scalar model for the RRS. That addition produced these fit statistics: χ²(117, N = 671) = 268.16, p < .0001, CFI = .936, RMSEA = .062, SRMR = .056. The estimated correlation between Brooding and Discrepancy was .45 for the U.S. group and .32 for the Argentina group, and no statistical difference emerged when constraining the correlation to be invariant between countries, Δχ²(1, N = 671) = 2.62, p = .106. In addition, the estimated correlation between Reflection and Discrepancy was nearly identical between countries (the US = .22, Argentina = .23), and this slight difference was not significant between the samples, Δχ²(1, N = 671) = 0.01, p = .929.

Discussion

The current study aimed to test invariance of the factor structure of the RRS between Argentine and U.S. students and, if invariance was supported, to evaluate the substantive question as to whether Argentinians had higher levels of rumination compared with their U.S. counterparts, and whether elevations in rumination were associated with maladaptive personality characteristics in one country but not the other. With regard to the first hypothesis, our findings support the factor structure of brooding and reflection of the RRS as relatively invariant between Argentine and U.S. students. Recall that, to reach that conclusion, two Reflection items (02 and 09) referring specifically to depression had to be dropped. Those items also have been problematic indicators in other research. For instance, Whitmer and Gotlib (2011) found that the same two items had cross-loadings with both the Brooding and Reflection factors, and they removed them from their measurement model. Although Treynor et al. (2003) refined the RRS by eliminating items from the original depression subscale of the RRS (see the appendix for more details regarding item content), these two items were retained despite their emphasis on “depression.” That decision seemed reasonable at the time because the original scale development work on the RRS was focused on depression and they considered those items as more directly measuring rumination than depressive mood. Nevertheless, as Armey et al. (2009, p. 318) argued, “these two items seem as likely candidates for elimination if adopting a strict, conservative, interpretation of the rule” (referring to the rational elimination rule described by Treynor et al., 2003). In our case, eliminating these items resulted in a significant improvement of model fit, supporting other authors’ recommendations regarding dropping them from the RRS (Armey et al., 2009; Lee & Kim, 2014). Another option would have been to retain the items but allow error variances for those items to correlate (Cova et al., 2007; Schoofs et al., 2010). We reasoned that a more parsimonious and theoretically consistent approach was to remove the items and retain a less ambiguous RRS factor structure.

Based on the reduced item set, partial scalar invariance between the countries was found after minor adjustments to the measurement. Item 01 (“Think ‘What am I doing to deserve this?’”) probably warrants additional scrutiny and discussion because it received the lowest factor loadings on the Brooding subscale in both countries and showed differences in intercepts between countries. Perhaps the phrasing of this item makes it less tenable as an indicator of rumination in both countries, and weaker in specific Argentina–U.S. comparisons. From a cross-cultural perspective, it seems that this item may be representing a more general complaint rather than a specific problematic aspect of rumination, perhaps especially for Argentine ruminators. As Clyne (1994) suggested, general complaints could be viewed more as a prosocial way to maintain a sense of connection with others, and less as a way to deal with feeling upset. Perhaps it is those positive social implications that help explain why response tendencies indicated the item was rated differently between the two samples. Although we retained this item with freely estimated intercepts between countries, Sass and Schmitt (2013) point out that “an observed variable with a smaller factor loading is arguably better than an observed variable than functions differently (i.e., noninvariant) across groups” (p. 320). Therefore, in the case of the RRS, our results suggest that in the context of similar cross-cultural comparisons, future work might consider excluding Item 01 to obtain a more accurate representation of rumination.

Another interesting caveat involves reliability of the factors. Reliability estimates for factors derived from the U.S. sample were in an acceptable range (.79 reflection, .83 brooding), but factor reliability was lower for the Argentina sample (.64 reflection, .69 brooding). Recall that original reliabilities reported by Treynor et al. (2003) based on measured indicators were not substantially different from results obtained in our U.S. sample (i.e., .75 for reflection and .78 for brooding). Although reliability estimates were relatively lower for the Argentina sample, other researchers have also encountered lower than desired reliability estimates for the RRS. For instance, Hong et al. (2010), Armey et al. (2009), and Whitmer and Gotlib (2011) reported reliability coefficients of <.70 for RRS subscale scores. Consistent with Cova, Rincón, and Melipillán (2009), one way to improve reliability is to add items, preferably without depression content, to the RRS subscales. Another implication is that users of the RRS might be well-advised to address measurement error by using CFA or structural equation modeling approaches in data analyses.

The measurement model findings permitted testing our other hypotheses. Contrary to expectations, we found that factor mean levels of Brooding were similar and comparable between countries; although we did not have a directional hypothesis for Reflection, we found factor levels for that dimension were also comparable between countries. Perhaps the absence of rumination differences is consistent with evidence that rates of depression are similar between the United States and Argentina (Ferrari et al., 2013), and with rumination being a known mechanism of depression (Nolen-Hoeksema et al., 2008).

The test of our third hypothesis compared correlations between the rumination factors and self-critical perfectionism. Although the U.S. students had a slightly higher correlation between Brooding and self-critical perfectionism than the Argentina sample, there was not a statistical difference between the correlations for the two groups. These findings suggest that students in both countries differed neither in terms of rumination levels nor in terms of the maladaptive, self-critical perfectionistic implications of rumination.

The absence of anticipated differences in the association between rumination and self-critical perfectionism has several implications for cross-cultural research on rumination. These results are likely consistent with Argentina being more similar to other Western countries, such as the United States, than it is to Asian countries, despite some similarities with Asian countries in terms of collectivistic leanings. In addition, the findings in this study may also reflect one of its limitations with a focus only on college students. University students may represent a unique subtype of individualistic cultural orientations, regardless of their countries of origin. Students often are evaluated individually more than they are collectively, and therefore have to prioritize individualistic assignments and aims over interpersonal or social activities. As a result, students, regardless of larger cultural context, may be “trained” in environments that emphasize self-immersing rather than in self-distancing. Given that self-distance is a mechanism that could allow reflecting without becoming distressed or depressed, more research is needed to capture how Western/non-Western people perform in certain situations where they were cued to self-reflect. Experimental designs or longitudinal research may help further address whether self-distancing might interact with cultural values and orientations. Our current findings suggest the RRS and its subscales can be reliably used in future cross-cultural studies aiming at this hypothesis.

Another consideration in the present study is reflected in MI being a growing but still relatively new practice in applied psychological research. Our comparisons were essentially different from previous work in which differences were only theoretically suggested, examined with instruments other than those used in the present study, or interpreted without evaluating measurement models in each culture before proceeding to formal tests. Moreover, although rumination has been a popular topic of research in the United States, little is known about how Latinos ruminate, and even scarcer is the evidence of rumination in Argentine students as representatives of Latino culture. To our knowledge, no other study has compared rumination in a South American country with a North American country. That said, future work is needed to disentangle some of the similarities and measurement implications reported in the current study. For now, we can only speculate that Western/individualistic values may have less of an effect on rumination than anticipated, at least in culturally proximal countries examined in the current study, but it should also be noted that Argentina is perhaps not as representative of a non-Western country as other Latin American countries might be. For example, in terms of individualism, Chile is more similar to Taiwan and Hong Kong than to Argentina (Hofstede et al., 2010). Similarly, although cultural concepts such as individualism and collectivism were raised, distinctions between Argentina and the United States on those dimensions were based on previous research (e.g., Chiou, 2001; Hofstede et al., 2010) and not directly measured in the present study. This limitation must temper greater confidence in individualistic and collectivistic inferences until future studies can directly evaluate those constructs.

There were several other limitations to the current study. The use of only one measure to evaluate the cross-cultural criterion validity of the RRS is a limitation that can be remedied in future studies with multiple measures of conceptually relevant constructs. For instance, criterion-related validity could be strengthened with future research that includes measures of worry, state and trait anxiety, posttraumatic stress disorder, physiological stress reactivity, and health indicators, which have been associated with rumination processes (e.g., Denson, Spanovic, & Miller, 2009; Lackner & Fresco, 2016; Michael, Halligan, Clark, & Ehlers, 2007).

Theoretically different measures of rumination are also needed to further test convergent and concurrent validity across countries. Alternative measures of rumination that tap a broader context of self-regulation would add knowledge to cross-cultural comparisons. For example, goal progress theory (Martin, Shrira, & Startup, 2004) proposes that the failure to progress toward higher order goals initiates rumination. Such perceived or actual failure to make goal-related progress has clear implications for self-critical perfectionism, so future work building on the current study might include goal progress theory measurement of rumination in cross-cultural comparison research. Of course, cross-cultural studies should ideally use criterion measures with some established measurement equivalence, similar to what was done in the present study with the measure of self-critical perfectionism (Arana et al., 2018).

Another limitation in order is that college student samples were obtained from both countries, which, without further invariance testing, preclude these findings being generalized to other groups in either country. The samples were also different in terms of age and gender distribution, although those variables did not alter the measurement findings. Future work with more broadly representative samples might go further by examining not only potential intersectionalities between country, gender, and age but also measurement variations attributable to cultural values represented within different countries. Indeed, in the present study, Argentine female students scored higher in Brooding than Argentine male students, in line with meta-analytical findings about gender and rumination (Johnson & Whisman, 2013). Although it is possible that the absence of this effect in the U.S. sample could be explained by the relatively weaker ratio of men to women compared with the Argentine sample, the size of the gender effect was small and unlikely to improve much with a larger subgroup of men in the analysis. Thus, future research might examine culturally informed gender issues that have implications for rumination and related psychological outcomes. For example, with 47% of the U.S. sample in the current study represented by racial/ethnic minority groups, this sample seems more diverse than is typically the case in studies of rumination, although race/ethnicity does not seem to be formally addressed in research summarizing gender differences in rumination (e.g., Johnson & Whisman, 2013). Thus, one specific focus of intersectionality might involve Gender × Race/Ethnic (or perhaps more important, cultural) interactions in the prediction of rumination.

Overall, our findings indicate that Argentinians and Americans ruminate at similar levels and with similar implications for maladjustment, insofar as the criterion of self-critical perfectionism serves as an indicator of maladjustment. The current study could be considered as a first part of a two-part process where more data would be collected to measure cross-cultural differences and to establish several kind of validities. Considering the preliminary nature of the present work in this area, we recommend future cross-cultural studies focusing on concurrent validity (i.e., compare negative outcomes with alternative measures of rumination), discriminant validity (i.e., compare positive outcomes with negative rumination and vice versa; e.g., self-esteem and brooding, sadness and reflection), and criterion validity studies of RRS scores. Relationship science could also benefit from cross-cultural comparisons of rumination and interpersonal problems. As some researchers garnered differential effects for brooding and reflection on interpersonal stressors such as romantic breakups (del Palacio-González, Clark, & & O’Sullivan, 2017; Saffrey & Ehrenberg, 2007), to test whether this pattern is maintained across cultures would be an interesting idea. Summarizing, the implementation of more precise measures of cultural values, or conducting stronger, longitudinal or experimental tests of patterns observed in the present study, added to further testing of construct validities, will be attractive avenues to follow for future cross-cultural research in rumination.

Footnotes

Appendix

Description of Items and Spanish Translation of the Short Ruminative Response Scale.

Items	21-Item version^a (71-item version^b)	Description [Spanish translation]	Dimension
RR01	RR05 (RR14)	Think “What am I doing to deserve this?” [Pensás “¿Qué hice para merecer esto?”]	Brooding
RR02	RR07 (RR18)	Analyze recent events to try to understand why you are depressed [Analizás acontecimientos recientes para tratar de entender por qué estás deprimido/a]	Reflection
RR03	RR10 (RR22)	Think “Why do I always react this way?” [Pensás “¿Por qué siempre reacciono de esta manera?”]	Brooding
RR04	RR11 (RR25)	Go away by yourself and think about why you feel this way [Te aislás y pensás sobre por qué te sentís de esa manera]	Reflection
RR05	RR12 (RR28)	Write down what you are thinking and analyze it [Escribís lo que estás pensando y lo analizás]	Reflection
RR06	RR13 (RR30)	Think about a recent situation, wishing it had gone better [Pensás sobre una situación reciente, deseando que hubiera salido mejor]	Brooding
RR07	RR15 (RR40)	Think “Why do I have problems other people don’t have?” [Pensás “¿Por qué tengo problemas que otras personas no tienen?”]	Brooding
RR08	RR16 (RR42)	Think “Why can’t I handle things better?” [Pensás “¿Por qué no puedo manejar mejor las cosas?”]	Brooding
RR09	RR20 (RR53)	Analyze your personality to try to understand why you are depressed [Analizás tu personalidad para tratar de entender por qué estás deprimido/a]	Reflection
RR10	RR21 (RR56)	Go someplace alone to think about your feelings [Te vas a algún lugar a solas para pensar en tus sentimientos]	Reflection

Butler and Nolen-Hoeksema (1994). ^bNolen-Hoeksema and Morrow (1991).

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Arana

F. G.

Keegan

E. G.

Rutsztein

(2009). Adaptación de una medida multidimensional de perfeccionismo: La Almost Perfect Scale-Revised (APS-R). Un estudio preliminar sobre sus propiedades psicométricas en una muestra de estudiantes universitarios argentinos [Adaptation of a measure of multidimensional perfectionism: The Almost Perfect Scale-Revised. A preliminary study about its psychometrical properties on a sample of Argentine university students]. Evaluar, 9, 35-53.

Arana

F. G.

Rice

K. G.

Ashby

J. S.

(2018). Perfectionism in Argentina and the United States: Measurement structure, invariance, and implications for depression. Journal of Personality Assessment, 100(2), 219-230. doi:10.1080/00223891.2017.1296845

Armey

M. F.

Fresco

D. M.

Moore

M. T.

Mennin

D. S.

Turk

C. L.

Heimberg

R. G.

. . . Alloy

L. B.

(2009). Brooding and pondering: Isolating the active ingredients of depressive rumination with exploratory factor analysis and structural equation modeling. Assessment, 16, 315-327.

Ayduk

Ö.

Kross

. (2010). From a distance: Implications of spontaneous self-distancing for adaptive self-reflection. Journal of Personality and Social Psychology, 98, 809-829. doi:10.1037/a0019205

Brown

T. A.

(2015). Confirmatory factor analysis for applied research (2nd ed.). New York, NY: Guilford Press.

Browne

M. W.

Cudeck

(1993). Alternative ways of assessing model fit. In Bollen

Long

(Eds.), Testing structural equation models (pp. 136-162.). Newbury Park, CA: Sage.

Burwell

R. A.

Shirk

S. R.

(2007). Subtypes of rumination in adolescence: Associations between brooding, reflection, depressive symptoms, and coping. Journal of Clinical Child & Adolescent Psychology, 36, 56-65.

Butler

L. D.

Nolen-Hoeksema

(1994). Gender differences in responses to depressed mood in a college sample. Sex Roles, 30, 331-346.

Byrne

B. M.

(1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications, and programming. London, England: Lawrence Erlbaum.

10.

Chang

E. C.

Tsai

Sanna

L. J.

(2010). Examining the relations between rumination and adjustment: Do ethnic differences exist between Asian and European Americans? Asian American Journal of Psychology, 1, 46-56.

11.

Chen

(2008). What happens if we compare chopsticks with forks? The impact of making inappropriate comparisons in cross-cultural research. Journal of Personality and Social Psychology, 95, 1005-1018.

12.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.

13.

Chiou

J. S.

(2001). Horizontal and vertical individualism and collectivism among college students in the United States, Taiwan and Argentina. Journal of Social Psychology, 141, 667-678.

14.

Clyne

(1994). Intercultural communication at work: Cultural values in discourse. Cambridge, England: Cambridge University Press.

15.

Cohen

Hoshino-Browne

Leung

A.K.-y.

(2007). Culture and the structure of personal experience: Insider and outsider phenomenologies of the self and social world. In Zanna

M. P.

(Ed.), Advances in experimental social psychology (Vol. 39, pp. 1-67). San Diego, CA: Academic Press.

16.

Conway

Csank

P. A. R.

Holm

S. L.

Blake

C. K.

(2000). On assessing individual differences in rumination on sadness. Journal of Personality Assessment, 75, 404-425.

17.

Cova

Rincón

Melipillán

(2007). Rumiación y presencia de sintomatología ansiosa y depresiva en adolescentes [Rumination and presence of anxious and depressive symptomatology in adolescents]. Revista Mexicana de Psicología, 24, 175-183.

18.

Cova

Rincón

Melipillán

(2009). Reflexión, rumiación negativa y desarrollo de sintomatología depresiva en adolescentes de sexo femenino [Reflection, brooding and development of depressive symptomatology in female adolescents]. Terapia Psicológica, 27, 155-160.

19.

del Palacio-González

Clark

D. A.

O’Sullivan

L. F

. (2017). Cognitive processing in the aftermath of relationship dissolution: Associations with concurrent and prospective distress and posttraumatic growth. Stress & Health, 33(5), 540-548. doi:10.1002/smi.2738

20.

Denson

T. F.

Spanovic

Miller

(2009). Cognitive appraisals and emotions predict cortisol and immune responses: A meta-analysis of acute laboratory social stressors and emotion inductions. Psychological Bulletin, 135, 823-853. doi:10.1037/a0016909

21.

Erdur-Baker

Ö.

Bugay

. (2010). The short version of ruminative response scale: Reliability, validity and its relation to psychological symptoms. Procedia—Social and Behavioral Sciences, 5, 2178-2181.

22.

Ferrari

A. J.

Charlson

F. J.

Norman

R. E.

Patten

S. B.

Freedman

Murray

C. J. L.

. . . Whiteford

H. A.

(2013). Burden of depressive disorders by country, sex, age, and year: Findings from the Global Burden of Disease Study 2010. PLoS Medicine, 10, e1001547. doi:10.1371/journal.pmed.1001547

23.

Grossmann

Kross

(2010). The impact of culture on adaptive versus maladaptive self-reflection. Psychological Science, 21, 1150-1157.

24.

Gruber

Harvey

Johnson

(2009). Reflective and ruminative processing of positive emotional memories in bipolar disorder and healthy controls. Behavior Research and Therapy, 47, 697-704.

25.

Hambleton

R. K.

Zenisky

A. L.

(2011). Translating and adapting tests for cross-cultural assessments. In Matsumoto

van de Vijver

F. J. R.

(Eds.), Cross-cultural research methods in psychology (pp. 46-70). New York, NY: Cambridge University Press.

26.

Harris

Pepper

Maack

(2008). The relationship between maladaptive perfection- ism and depressive symptoms: The mediating role of rumination. Personality & Individual Differences, 44, 150-160. doi:10.1016/j.paid.2007.07.011

27.

Hervas

(2008). Adaptación al castellano de un instrumento para evaluar el estilo rumiativo: La escala de respuestas rumiativas [Spanish adaptation of an instrument to assess ruminative style: Ruminative responses scale]. Revista de Psicopatología y Psicología Clínica, 13, 111-121.

28.

Hofstede

G. J.

Minkov

(2010). Cultures and organizations: Software of the mind (3rd ed.). New York, NY: McGraw-Hill.

29.

Hong

Abela

J. R.

Cohen

J. R.

Sheshko

D. M.

Shi

X. T.

Hamel

A. V.

Starrs

(2010). Rumination as a vulnerability factor to depression in adolescents in mainland China: Lifetime history of clinically significant depressive episodes. Journal of Clinical Child & Adolescent Psychology, 39, 849-857.

30.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

31.

James

Verplanken

Rimes

K. A.

(2015). Self-criticism as a mediator in the relationship between unhealthy perfectionism and distress. Personality and Individual Differences, 79, 123-128. doi:10.1016/j.paid.2015.01.030.

32.

Johnson

D. P.

Whisman

M. A.

(2013). Gender differences in rumination: A meta-analysis. Personality and Individual Differences, 55, 367-374.

33.

Jose

P. E.

Kramer

Hou

(2014). Does brooding rumination moderate the stress to depression relationship similarly for Chinese and New Zealand adolescents? Journal of Educational and Developmental Psychology, 4, 114-127. doi:10.5539/jedp.v4n1p114

34.

Key

B. L.

Campbell

T. S.

Bacon

S. L.

Gerin

(2008). The influence of trait and state rumination on cardiovascular recovery from a negative emotional stressor. Journal of Behavioral Medicine, 31, 237-248.

35.

Kokonyei

Szabo

Kocsel

Edes

Eszlari

Pap

. . . Anderson

I. M.

(2016). Rumination in migraine: Mediating effects of brooding and reflection between migraine and psychological distress. Psychology & Health, 31, 1481-1497.

36.

Kross

Ayduk

. (2008). Facilitating adaptive emotional analysis: Distinguishing distanced analysis of depressive experiences from immersed-analysis and distraction. Personality and Psychology Bulletin, 34, 924-938.

37.

Kross

Ayduk

Mischel

(2005). When asking “why” does not hurt: Distinguishing rumination from reflective processing of negative emotions. Psychological Science, 16, 709-715.

38.

Kuehner

Weber

(1999). Responses to depression in unipolar depressed patients: An investigation of Nolen-Hoeksema’s response styles theory. Psychological Medicine, 29, 1323-1333.

39.

Lackner

R. J.

Fresco

D. M.

(2016). Interaction effect of brooding rumination and interoceptive awareness on depression and anxiety symptoms. Behaviour Research and Therapy, 85, 43-52. doi:10.1016/j.brat.2016.08.007

40.

Lee

Kim

(2014). Cross-cultural adaptation, reliability, and validity of the revised Korean version of ruminative response scale. Psychiatry Investigation, 11, 59-64.

41.

Limburg

Watson

H. J.

Hagger

M. S.

Egan

S. J.

(2017). The relationship between perfectionism and psychopathology: A meta-analysis. Journal of Clinical Psychology, 73(10), 1301-1326. doi:10.1002/jclp.22435

42.

Luminet

(2004). Measurement of depressive rumination and associated constructs. In Papageorgiou

Wells

(Eds.), Depressive rumination: Nature, theory and treatment (pp. 187-215). New York, NY: Wiley.

43.

MacDougall

(2011). Moving beyond the nuts and bolts of score reliability in medical education: Some valuable lessons from measurement theory. Advances and Applications in Statistical Sciences, 6, 643-664.

44.

Martin

L. L.

Shrira

I. L. A. N.

Startup

H. M.

(2004). Rumination as a function of goal progress, stop rules, and cerebral lateralization. In Papageorgiou

Wells

(Eds.), Depressive rumination: Nature, theory and treatment (pp. 153-175). London, England: Wiley.

45.

McLaughlin

K. A.

Wisco

B. E.

Aldao

Hilt

L. M.

(2014). Rumination as a transdiagnostic factor underlying transitions between internalizing symptoms and aggressive behavior in early adolescents. Journal of Abnormal Psychology, 123, 13-23. doi:10.1037/a0035358

46.

Michael

Halligan

S. L.

Clark

D. M.

Ehlers

(2007). Rumination in posttraumatic stress disorder. Depression and Anxiety, 24, 307-317. doi:10.1002/da.20228

47.

Miranda

Nolen-Hoeksema

(2007). Brooding and reflection: Rumination predicts suicidality at one-year follow up in a community sample. Behavior Research and Therapy, 45, 3088-3095.

48.

Muthén

L. K.

Muthén

B. O.

(1998-2015). Mplus Version 7.4. Los Angeles, CA: Muthén & Muthén.

49.

Nepon

Flett

G. L.

Hewitt

P. L.

Molnar

D. S.

(2011). Perfectionism, negative social feedback, and interpersonal rumination in depression and social anxiety. Canadian Journal of Behavioural Science, 43, 297-308. doi:10.1037/a0025032

50.

Nolen-Hoeksema

(1991). Responses to depression and their effects on the duration of depressive episodes. Journal of Abnormal Psychology, 100, 569-582.

51.

Nolen-Hoeksema

Morrow

(1991). A prospective study of depression and posttraumatic stress symptoms after a natural disaster: The 1989 Loma Prieta earthquake. Journal of Personality and Social Psychology, 61, 115-121.

52.

Nolen-Hoeksema

Watkins

E. R.

(2011). A heuristic for developing transdiagnostic models of psychopathology explaining multifinality and divergent trajectories. Perspectives on Psychological Science, 6, 589-609.

53.

Nolen-Hoeksema

Wisco

B. E.

Lyubomirsky

(2008). Rethinking rumination. Perspectives on Psychological Science, 3, 400-424.

54.

Raes

Hermans

Eelen

(2003). Kort instrumenteel De Nederlandstalige versie van de Ruminative Response Scale (RRS-NL) en de Rumination on Sadness Scale (RSS-NL). Gedragstherapie, 36, 97-104.

55.

Raykov

(2009). Evaluation of scale reliability for unidimensional measures using latent variable modeling. Measurement and Evaluation in Counseling and Development, 42, 223-232.

56.

Rice

K. G.

Richardson

C. M. E.

Tueller

(2014). The Short Form of the Revised Almost Perfect Scale. Journal of Personality Assessment, 96, 368-379. doi:10.1080/00223891.2013.838172

57.

Roberts

J. E.

Gilboa

Gotlib

I. H.

(1998). Ruminative response style and vulnerability to episodes of dysphoria: Gender, neuroticism, and episode duration. Cognitive Therapy and Research, 22, 401-423.

58.

Roger

Jamieson

(1988). Individual differences in delayed heart-rate recovery following stress: The role of extraversion, neuroticism and emotional control. Personality and Individual differences, 9, 721-726.

59.

Saffrey

Ehrenberg

(2007). When thinking hurts: Attachment, rumination, and postrelationship adjustment. Personal Relationships, 14, 351-368. doi:10.1111/j.1475-6811.2007.00160.x

60.

Sakamoto

Kambara

Tanno

(2001). Response styles and cognitive and affective symptoms of depression. Personality and Individual Differences, 31, 1053-1065.

61.

Sass

D. A.

Schmitt

T. A.

(2013). Testing measurement and structural invariance. In Teo

(Ed.), Handbook of quantitative methods for educational research (pp. 315-345). Rotterdam, Netherlands: Sense.

62.

Schoofs

Hermans

Raes

(2010). Brooding and reflection as subtypes of rumination: Evidence from confirmatory factor analysis in nonclinical samples using the Dutch Ruminative Response Scale. Journal of Psychopathology and Behavioral Assessment, 32, 609-617.

63.

Siegle

G. J.

Moore

P. M.

Thase

M. E.

(2004). Rumination: One construct, many features in healthy individuals, depressed individuals, and individuals with lupus. Cognitive Therapy and Research, 28, 645-668.

64.

Slaney

R. B.

Rice

K. G.

Mobley

Trippi

Ashby

J. S.

(2001). The revised Almost Perfect Scale. Measurement and Evaluation in Counseling and Development, 34, 130-145.

65.

Stoeber

Otto

(2006). Positive conceptions of perfectionism: Approaches, evidence, and challenges. Personality and Social Psychology Review, 10, 295-319. doi:10.1207/s15327957pspr1004_2

66.

Tanner

Voon

Hasking

Martin

(2013). Underlying structure of ruminative thinking: Factor analysis of the ruminative thought style questionnaire. Cognitive Therapy and Research, 37, 633-646.

67.

Treynor

Gonzalez

Nolen-Hoeksema

(2003). Rumination reconsidered: A psychometric analysis. Cognitive Therapy and Research, 27, 247-259.

68.

Van der Kaap-Deeder

Soenens

Boone

Vandenkerckhove

Stemgée

Vansteenkiste

. (2016). Evaluative concerns perfectionism and coping with failure: Effects on rumination, avoidance, and acceptance. Personality and Individual Differences, 101, 114-119.

69.

Watkins

(2008). Constructive and unconstructive repetitive thought. Psychological Bulletin, 134, 163-206.

70.

Whitmer

Gotlib

I. H.

(2011). Brooding and reflection reconsidered: A factor analytic examination of rumination in currently depressed, formerly depressed, and never depressed individuals. Cognitive Therapy and Research, 35, 99-107.

71.

Yang

Green

S. B.

(2011). Coefficient alpha: A reliability coefficient for the 21st century? Journal of Psychoeducational Assessment, 29, 377-392. doi:10.1177/0734282911406668

72.

Young

C. C.

Dietrich

M. S.

(2015). Stressful life events, worry, and rumination predict depressive and anxiety symptoms in young adolescents. Journal of Child and Adolescent Psychiatric Nursing, 28, 35-42. doi:10.1111/jcap.12102