A Novel Measure of Real-Time Perseverative Thought

Abstract

Perseverative thinking (PT), or repetitive negative thinking, has historically been measured using global self-report scales. New methods of assessment are needed to advance understanding of this inherently temporal process. We developed an intensive longitudinal method for assessing PT. A mixed sample of 77 individuals ranging widely in trait PT, including persons with PT-related disorders (generalized anxiety disorder, major depression) and persons without psychopathology, used a joystick to provide continuous ratings of thought valence and intensity following exposure to scenarios of differing valence. Joystick responses were robustly predicted by trait PT, clinical status, and stimulus valence. Higher trait perseverators exhibited more extreme joystick values overall, greater stability in values following threatening and ambiguous stimuli, weaker stability in values following positive stimuli, and greater inertia in values following ambiguous stimuli. The joystick method is a promising measure with the potential to shed new light on the dynamics and precipitants of perseverative thinking.

Keywords

perseverative thinking repetitive negative thinking cognitive processes major depressive disorder generalized anxiety disorder transdiagnostic intensive longitudinal methods open materials

Perseverative thinking (PT), or repetitive negative thinking, is a cognitive process involving difficulty disengaging from negative thoughts. Two common forms of PT are worry, or negative, difficult-to-control thinking about future events whose outcomes are uncertain (Borkovec et al., 1983), and rumination, or negative thinking about one’s feelings or past events, particularly past failures (Nolen-Hoeksema et al., 2008). PT is dimensionally distributed in the population (Olatunji et al., 2010; Ruscio et al., 2001) and has been linked to a wide range of adverse psychological and physical outcomes (Brosschot et al., 2006; Nolen-Hoeksema et al., 2008; Olatunji et al., 2010; Watkins, 2008). It is increasingly recognized as a transdiagnostic process that robustly predicts the onset and maintenance of anxiety, depressive, and related disorders (Ehring & Watkins, 2008; Harvey et al., 2004; Lyubomirsky & Nolen-Hoeksema, 1993; Nolen-Hoeksema, 1991, 2000; Thielsch et al., 2015).

Previous Measurement of PT

Despite its importance, understanding of PT has been impeded by the tools that are available to measure it. PT is fundamentally a temporal process, yet available methods typically measure it as a static trait (see Samtani & Moulds, 2017). Traditionally, researchers have used global trait questionnaires to measure particular forms of PT, such as worry (Meyer et al., 1990; Tallis et al., 1992) or depressive rumination (Trapnell & Campbell, 1999; Treynor et al., 2003). As evidence has accumulated for overlapping features across different forms of PT (Ehring & Watkins, 2008; McEvoy & Brans, 2013), new trait questionnaires that assess PT as a transdiagnostic construct have emerged (Ehring et al., 2011; McEvoy et al., 2010). Both the traditional and transdiagnostic questionnaires have excellent reliability, and their ease of use has helped amass a large literature on PT and its correlates.

Unfortunately, trait questionnaires also have significant limitations for revealing the nature of PT. Global reporting of symptoms is prone to recall errors and biases, to which individuals with emotional disorders may be especially susceptible (Mineka et al., 2003). Moreover, trait measures provide little insight into dynamic (i.e., time-dependent) characteristics of PT, such as its persistence, recurrence, and intrusiveness. Global scales capture individuals’ general impressions of these characteristics but fail to measure the characteristics as they unfold over time in response to changing environmental conditions.

Recognizing the limitations of global trait questionnaires, researchers have developed state measures of PT. The focused-breathing task (FBT) has been used to assess PT over short periods in the laboratory (Borkovec et al., 1983; Hirsch et al., 2009; McLaughlin et al., 2007; Ruscio & Borkovec, 2004; Ruscio et al., 2011). In the FBT, participants are instructed to focus on their breath for 5 min. At random intervals spaced an average of 60 s to 90 s apart, participants are signaled to report whether, at the moment of the signal, they were focused on their breath or distracted by negative, positive, or neutral thoughts. Supporting the task’s validity as a measure of PT, distraction by negative thoughts during the FBT is associated with the frequency and severity of trait worry and rumination (Borkovec et al., 1983; Ruscio et al., 2011) and with the presence and severity of generalized anxiety disorder (GAD) and major depressive disorder (MDD; Ruscio et al., 2011), disorders in which PT is a prominent clinical feature.

Researchers also use periodic thought ratings as a state measure of PT in daily life. In GAD research, daily diaries (self-monitoring forms), completed once or a few times per day, have been used to assess the frequency and severity of worry (Borkovec et al., 2002; Dupuy et al., 2001). Increasingly, ecological momentary assessment (EMA) has been used to sample worried or ruminative thoughts at more frequent intervals, typically eight times over a 12-hr day (Kircanski et al., 2015; Moberly & Watkins, 2008; Ruscio et al., 2015). Thought ratings collected using these sampling methods have been shown to correlate with trait worry and rumination scores (e.g., Kircanski et al., 2015).

EMA and daily diary methods take thought samples at widely spaced intervals of hours or days. This reveals broad patterns of PT and its situational contexts but lacks the resolution to detect the fine-grained details of an episode of negative thought. Laboratory-based methods such as the FBT sample thoughts over minutes, capturing PT in relation to proximal, standardized stimuli. Although this allows for a context-controlled understanding of PT, directing attention toward or away from thoughts and interrupting thoughts to ask about their content risks disrupting the natural thought progression. In addition, periodic sampling may miss negative-thought intrusions that occur between samples.

Measuring PT in Real Time and High Resolution

A new type of measure is needed that can capture a “bout” of PT as it unfolds moment to moment in real time. Such a measure would provide a window into what happens when people ruminate or worry. In particular, it would enable investigation of the temporal characteristics of PT, which would illuminate what it means for thoughts to be repetitive, intrusive, and uncontrollable. By identifying dynamic features of thoughts that are associated with specific clinical problems or poor functional outcomes, this work could set the stage for more targeted interventions to disrupt the cycle of repetitive negative thinking.

The internal, private nature of cognition poses methodological challenges to assessing PT in real time. However, affective science provides models for measuring covert processes “on line” in a moment-to-moment fashion, such as using a dial to continuously rate one’s degree of amusement or sadness while watching films (Mauss et al., 2005). Affective science also provides a useful framework for describing the temporal dynamics of an internal response to a stimulus, such as the peak or amplitude of the response, the rise time to reach that peak, and the recovery time or duration of the response (Davidson, 1998). Methods from the growing field of emotion dynamics, which seeks to describe how and why feelings change over time (Kuppens & Verduyn, 2017), could profitably be applied to study PT. In particular, two dynamic metrics seem promising given clinical accounts of PT: stability and inertia. Stability indexes the magnitude of change from moment to moment, with higher stability reflecting less change (fewer or weaker ups and downs) over time. Inertia indexes self-predictiveness, with greater inertia reflecting cognitive states that linger and show less homeostatic return to baseline over time (Houben et al., 2015). Testing whether these metrics characterize perseverators’ thinking patterns requires a new measurement approach that has sufficient temporal resolution to detect these patterns.

Contexts That Cue and Perpetuate PT

A real-time measure has the potential to reveal not only the temporal dynamics of PT but also the contexts that trigger PT and the stimulus features that predict a more intense, sustained PT response. Research using trait questionnaires sheds limited light on these questions, but useful leads are suggested by studies that examined cognitive processing of valenced stimuli in clinical populations characterized by high PT (e.g., GAD and MDD). Those studies have generally revealed heightened responding to negative stimuli in anxious and depressed individuals (Mathews & MacLeod, 2005; but see Bylsma et al., 2008). In particular, threats have been identified as an important stimulus class. GAD is associated with attentional bias toward threatening stimuli and a tendency to interpret ambiguous events as threatening; there is preliminary evidence that the same biases occur in MDD (see Craske et al., 2009). How preferential processing of potential threats influences thought patterns is less well understood, although claims that PT prolongs representations of stressors in the mind (Brosschot et al., 2006) suggest that individuals with higher trait PT may experience a more prolonged period of negative thoughts after threat exposure. In line with this idea, greater baseline PT predicts more negative-thought intrusions following a laboratory stressor (Ruscio et al., 2011), and inducing a ruminative thinking style during recall of a failure experience results in more negative-thought intrusions later on (Watkins, 2004).

There is an especially pressing need to characterize PT responses outside of negative contexts. Virtually no research has measured PT in the wake of positive stimuli. A few studies have used a trait questionnaire (Feldman et al., 2008) to investigate cognitive responses to positive affect in bipolar disorder and have yielded mixed results (Gilbert et al., 2013; Gruber et al., 2011; Johnson et al., 2008). Likewise, mixed results have emerged from research on affective responses to positive stimuli in MDD; some studies have shown blunted responding to positive experiences (Bylsma et al., 2008; Dichter, 2010), and others have shown heightened responding (Bylsma et al., 2011; Khazanov et al., 2019) to positive experiences in this disorder. More research has considered ambiguous stimuli. The tendency to interpret ambiguous information in a negative manner has been identified as a causal factor for PT, and targeting this bias reduces intrusive thoughts (Hirsch et al., 2016). Learning how trait perseverators respond to ambiguous and positive stimuli is important for determining the specificity of reactions to negative stimuli. It is also a necessary step toward understanding how PT is experienced in complex natural environments involving stimuli of varying valence, intensity, and ambiguity.

In sum, notable gaps in the literature prevent us from drawing conclusions about how perseverative thinking behaves in different contexts. Most prior studies have focused on the magnitude (mean level) of responding rather than the time course of responding, on negative stimuli rather than positive stimuli, and on affective responses rather than cognitive responses. In addition, most studies have used stimuli that simply reflect the presence or absence of threat or reward rather than taking a more nuanced approach by including varying degrees of threat and reward or by including stimuli whose valence is ambiguous. Investigating the time course of cognitive responses to systematically varying stimuli would shed much needed light on the contexts that cue and perpetuate PT.

The Present Study

We developed a novel, real-time measure of intrusive thought using a joystick device. Participants used the joystick to provide 30 s of continuous ratings of the intensity and valence of their thoughts following scenarios representing differing levels of threat and reward. As a local-state measure of PT rather than global-trait measure of PT, the joystick method avoids many limitations of trait questionnaires. By rating current thoughts, it eliminates the risk of errors in recalling and averaging across memories. By collecting a simple rating of thought valence, it reduces self-report biases relative to traditional questionnaires, which require multiple, more complex ratings of thought patterns, appraisals, and beliefs. As a continuous measure, the joystick method precludes the possibility of thoughts being missed between samples. Although it requires thought monitoring, ongoing monitoring using the joystick should be less intrusive than tasks like the FBT, which interrupt thoughts abruptly and repeatedly so that participants can read and respond to rating scales.

Our aims in this study were (a) to validate the joystick method as a novel thought-sampling approach and (b) to use the joystick to reveal patterns of perseverative responding to evocative stimuli. We used trait PT, clinical status, and stimulus category to predict the mean levels and temporal dynamics of the joystick ratings. We expected to find evidence for the joystick measure as a valid and context-sensitive measure of PT, as reflected in associations with trait PT, elevations in disorders characterized by high PT, and sensitivity to the valence and emotional salience of the preceding stimulus. We also expected that higher trait PT would be associated with higher levels of stability and inertia in the time series of joystick ratings. Across all analyses, we hypothesized that negative and ambiguous stimuli would be especially revealing of individual differences in PT, although we lacked a strong basis for predicting whether those differences would extend to positive stimuli.

Method

Participants

Participants were adults recruited from the Philadelphia community via advertisements on Craigslist, in local newspapers, in mental health and family medicine clinics, and on community posting boards. To enroll a sample that varied widely in PT, we recruited a mixed sample of clinical and nonclinical participants. Clinical participants were required to have a current, principal (most severe) diagnosis of either GAD or MDD. Psychotropic medications were permitted at a stable dosage, but individuals who reported current suicidal intent, acute psychosis, or substance-related disorders (other than tobacco) within the past month were excluded. Nonclinical participants were required to have no current or past psychopathology.

A total of 77 participants provided joystick data. These participants were predominantly female (56%) and ranged in age from 18 to 65 (M = 30.16 years, SD = 9.91). The sample was racially diverse: 61% of participants identified as White, 22% identified as Black or African American, 7% identified as Asian, and 10% identified as “other”; 3% of participants reported their ethnicity as Hispanic or Latinx. Most participants were single (66%) and had at least a bachelor’s degree (55%). The majority of participants (57%) were employed, 18% were unemployed, and 25% were students. Annual household income ranged from $0 to $120,000 (M = $31,572, SD = 25,813).

The clinical (n = 56) and nonclinical (n = 21) participants did not differ in age or household income (both ts < 0.63, both ps > .536) or in sex, race, ethnicity, marital or employment status, or educational attainment (all χ²s < 4.32, all ps > .173). As expected, clinical participants scored higher than nonclinical participants on all clinician-assessed measures of depression and anxiety and all trait PT measures (all ts > 5.29, all ps < .001).

Measures

Clinician-assessed measures

The Anxiety and Related Disorders Interview Schedule for DSM-5–Lifetime Version (ADIS-5L; Brown & Barlow, 2014) was used to assess the presence and severity of mental disorders. The ADIS-5L is a semistructured diagnostic interview designed to diagnose anxiety, mood, obsessive-compulsive, trauma-related, and substance use disorders according to the criteria of the fifth edition of the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association, 2013). Clinical interviewers were master’s- or bachelor’s-level diagnosticians trained to a high level of reliability by an expert rater. Diagnostic decisions and clinical severity ratings for each participant were finalized in weekly consensus meetings of the assessment team led by the supervising clinical psychologist. Interrater reliability was high for GAD (κ = 1.00) and MDD (κ = .88) diagnoses according to blind, independent ratings of recorded interviews (n = 32) selected at random from ongoing studies with these populations conducted in our lab.

The Hamilton Depression Rating Scale (HAM-D; Hamilton, 1960; intraclass correlation coefficient [ICC] = .97) is a 17-item, clinician-administered scale assessing depressive symptoms experienced in the past week. Eight items are rated on a 5-point scale ranging from 0 (not present) to 4 (severe); the remaining nine items are rated from 0 (absent) to 2 (frequent).

The Hamilton Anxiety Rating Scale (HAM-A; Hamilton, 1959; ICC = .96) is a 14-item, clinician-administered scale assessing symptoms of anxiety. All items are scored on a 5-point scale ranging from 0 (not present) to 4 (severe).

Trait PT measures

The Perseverative Thinking Questionnaire (Ehring et al., 2011) is a 15-item scale that measures PT transdiagnostically. Items on the scale assess key process characteristics of PT, including repetitiveness, intrusiveness, uncontrollability, and unproductiveness. Participants rate how they typically think about negative experiences or problems using a scale from 0 (never) to 4 (almost always).

The Repetitive Thinking Questionnaire (McEvoy et al., 2010) is a transdiagnostic measure of PT. Participants are asked to describe a recent stressful situation they encountered and then rate 10 statements concerning thoughts and images they experienced after that specific situation. Statements are rated on a scale from 1 (not true at all) to 5 (very true).

The Penn State Worry Questionnaire (Meyer et al., 1990) is a 16-item measure of pathological worry. Items assess the frequency, intrusiveness, and pervasiveness of worry, rated on a scale from 1 (not at all typical of me) to 5 (very typical of me).

The Ruminative Responses Scale (Treynor et al., 2003) is a questionnaire that measures rumination. Twenty-two statements describing styles of cognitive responding when depressed are rated on a scale from 1 (almost never) to 4 (almost always).

The Rumination-Reflection Questionnaire (Trapnell & Campbell, 1999) is a 24-item scale that measures the constructs of rumination and reflection. Only the 12 items comprising the rumination subscale were administered. Items assessing the participant’s general tendency to ruminate are rated on a scale from 1 (strongly disagree) to 5 (strongly agree).

Each of the aforementioned PT questionnaires correlated highly with the others (rs = .68–.87). To enhance measurement reliability and reduce the number of analyses, we created a PT composite by summing the standardized scores of the five questionnaires. The PT composite had high internal consistency (Cronbach’s α = .94) and provided excellent coverage of the PT construct by assessing both broad and specific forms of the trait and by measuring PT as both a general cognitive style and a postevent response. Subsequent analyses were conducted using this composite variable.

Experimental measures

Joystick ratings

Participants provided online ratings of their thoughts using a custom-built joystick device, administered using E-Prime behavioral experiment software (Version 2.0.10.242; Psychology Software Tools, Inc., Sharpsburg, PA). The joystick stands on a stationary platform and is moved along a single axis by the participant’s dominant hand. Ratings are represented by the position of the joystick; the “up” position represents positive thoughts, the “down” position represents negative thoughts, and the angle of the joystick in either direction represents thought intensity. When no motion is applied or the joystick is released, the joystick returns to the center position, which represents neutral thoughts. The angle of the joystick is recorded in E-Prime as an integer from −10 (joystick extended fully in the negative, down position) to 10 (joystick extended fully in the positive, up position); neutral is recorded as a value of 0. The joystick’s position, measured using two potentiometers, is sent to E-Prime at a frequency of 4 data points per s.

Emotion ratings

During the experiment, stimuli of varying valence and intensity were presented to participants. After each stimulus was presented, participants used a keypad to rate their emotional valence from 0 (very negative) to 100 (very positive), their emotional arousal from 0 (low arousal) to 100 (high arousal), and their tension from 0 (very relaxed) to 100 (very tense) according to how they were feeling at that moment.

Procedure

Individuals who screened as eligible on an online screening survey were invited to the lab, where they were administered the ADIS-5L, HAM-A, and HAM-D by a clinical interviewer. Participants who met eligibility criteria for the study according to this interview returned to the lab at a later date to complete the experiment.

At the experimental session, participants met individually with a research assistant who guided them through the study. The study took place in a private testing room equipped with a computer, headphones, and the joystick device. Before the experiment began, the research assistant introduced the joystick and guided the participant in using the joystick to rate thoughts evoked by a hypothetical scenario, checking that the participant understood and was following the joystick procedure. Participants then practiced using the joystick independently during two practice trials.

Next, participants completed a “vanilla baseline” in which they sat quietly and listened to a 3-min historical narrative. This procedure was designed to reduce individual differences in baseline emotions and cognitions by focusing participants’ attention on a standardized, emotionally neutral, minimally demanding task (Jennings et al., 1992). Following this baseline period, participants completed the experimental task. This task was based on the well-established fear-reactivity protocol developed by Lang and colleagues (2007) and adapted by our team to elicit threat reactivity in the context of anxiety. Participants completed 36 trials, each with the following structure (for a schematic of the trial structure, see Fig. S1 in the Supplemental Material available online).

Participants sat quietly for 8 s and then listened to an audio-recorded scenario, one sentence in length (for the list of stimuli, see Table S1 in the Supplemental Material). The scenarios were selected in equal number and at random from the following six categories, with a total of six stimuli per category:

Major negative: unambiguous, high-level threat (e.g., “You are fired from your job after five years with the company.”)

Minor negative: unambiguous, low-level threat (e.g., “Your alarm does not go off and you are late to work.”)

Ambiguous: unclear whether threatening or benign (e.g., “A very close friend has some unexpected news to share with you.”)

Major positive: unambiguous, strongly rewarding (e.g., “You are promoted to a more prestigious job with a higher salary.”)

Minor positive: unambiguous, mildly rewarding (e.g., “You run into an old friend whom you haven’t seen in years.”)

Neutral: neither positive nor negative (e.g., “You hang your coat on the coat rack in the entry hall.”)

After listening to each stimulus over headphones, participants were instructed to think as intensely as possible about how they would react if they experienced the situation described. After 12 s, participants rated their emotion. Then, they sat quietly for 30 s while providing continuous online ratings of their thoughts using the joystick device. Participants were explicitly instructed that they could think about whatever they liked during this “rest” period (rather than being directed to focus on thoughts they might have in the situation described) and to use the joystick to reflect how positive or negative their thoughts were at each moment. To emphasize that all thoughts should be rated, the instructions gave multiple examples of task-unrelated positive and negative thoughts. Thus, the joystick ratings provided a real-time measure of naturally occurring thoughts experienced in response to the stimuli. After the 30-s period, participants were asked to remove their hand from the joystick, clear their mind, and sit quietly for 8 s until the next stimulus was presented.

Participants returned to the laboratory approximately 2 to 3 weeks later to complete additional measures, including the trait PT questionnaires. A subset of participants did not complete this additional study phase and consequently did not have a trait PT composite score. Participants with (n = 66) and without (n = 11) the trait PT score did not differ in clinical status (χ² = 0.54, p = .465) or in depression or anxiety severity as measured by the HAM-D and HAM-A, respectively (both ts < 1.23, both ps > .246). Participants received monetary compensation for each lab visit.

Statistical analyses

In preliminary analyses, we examined pilot data on the acceptability of the joystick measure to participants and then inspected mean joystick values by stimulus category and across clinical and nonclinical participants. Next, we conducted validation analyses testing whether joystick value was (a) associated with emotional responses to the stimuli and (b) predicted by trait PT, clinical status, and stimulus valence. Because we expected trait PT to be higher in persons with psychopathology, including clinical status in the models allowed us to test whether joystick values were uniquely related to trait PT rather than simply a proxy for psychopathology. Finally, we used the joystick time series to examine the temporal dynamics of cognitive responses to the stimuli. Specifically, we tested whether individuals with higher trait PT had more stable joystick values (operationalized by the squared successive difference [SSD]) or more inert joystick values (operationalized as autocorrelation), depending on stimulus valence.

Because our data were hierarchical—joystick values were nested within stimuli, which were nested within individuals—we used hierarchical linear models. Individual joystick values (or the relationship between joystick values in SSD analyses) were modeled at Level 1. Participants typically took at least half a second to begin moving the joystick in each trial, so we excluded the first two data points from each time series. For many analyses, we were interested in the effect of the valence of the experimental stimulus (i.e., positive, negative, or ambiguous) on joystick responding. To increase statistical power and reduce the number of tests, we created a single stimulus negativity variable, dummy coded as 2 for major negative stimuli, 1 for minor negative stimuli, and 0 for all other stimuli (neutral, ambiguous, minor positive, major positive). We used an analogous process to create a stimulus positivity variable. We also created a stimulus ambiguity variable, dummy coded as 1 for ambiguous stimuli and 0 for all other stimuli. These variables essentially served as contrasts in our regression equations.

When employing the stimulus valence variables as predictors, we modeled the individual at Level 2. When the stimulus valence variables were not included as predictors, we modeled the stimulus at Level 2 and the individual at Level 3. All analyses were conducted in R software environment (Version 3.6.1; R Core Team, 2019).

We used the lme4 (Version 1.1.25; Bates et al., 2015) and MASS (Version 7.3.53; Venables & Ripley, 2002) packages for model building, specifically the lmer and glmer.nb functions. We used the lrtest and BIC functions from the lmtest package (Version 0.9.38; Zeileis & Hothorn, 2002) to compare model fits. To conduct simple-slopes analyses, we used the sjPlot (Version 2.8.6; Lüdecke, 2020) and interplot (Version 0.2.2; Solt & Hu, 2019) packages, specifically the plot_model and interplot functions, as well as custom functions written for the current study.

Results

Validating the joystick measure

For pilot analyses on the acceptability of the joystick measure, see the Supplemental Material. To validate the joystick measure, we examined joystick ratings as a function of stimulus valence and evocativeness. As detailed below, joystick ratings increased monotonically in the expected pattern according to stimulus valence. In addition, joystick ratings for a stimulus were related to emotional valence and tension ratings for that stimulus.

Effects of stimulus category and valence

We first examined mean joystick values by stimulus category. Mean joystick level differed significantly across stimulus categories, F(5, 456) = 91.89, p < .001, η² = .50 (Fig. 1a). A post hoc Student-Newman-Keuls test revealed significant differences between all six stimulus categories, all ps < .01. As expected, joystick values were reliably larger (greater in magnitude) for major stimuli than minor stimuli and for minor stimuli than neutral stimuli. Joystick values for ambiguous stimuli were slightly negative on average.

After collapsing the major and minor categories into ordinal stimulus negativity and positivity variables, as described above, we inspected mean joystick values by stimulus valence across groups (Fig. 1b). Joystick ratings appeared more negative for the clinical group than the nonclinical group. Note that the mean value for ambiguous stimuli fell in the negative range for the clinical group but was positive for the nonclinical group. We formally tested the interaction of clinical status by stimulus valence in later hierarchical analyses.

Effects of stimulus evocativeness

Next, we tested whether joystick ratings were predicted by emotional responses to the stimuli. We reasoned that scenarios that were especially evocative for the participant, as indexed by that participant’s ratings of emotional valence, arousal, and tension after imagining the scenario, would elicit larger joystick values than scenarios that were less personally salient. We built a multilevel model predicting joystick value from these emotional responses:

\begin{array}{l} J S_{t k i} = π_{000} + β_{1} A f f V a l e n c e_{i} + β_{2} A f f A r o u s a l_{i} \\ + β_{2} T e n s i o n_{i} + e_{t k i} + r_{00 i} + u_{0 k i} . \end{array}

JS_tki represents joystick value t for stimulus k for person i. This outcome was modeled as a function of average intercepts at both the person and stimulus levels; the Level 1 fixed effects of emotional valence, arousal, and tension (β₁, β₂, and β₃); and global intercept π₀₀₀ representing the mean of the time series when emotional valence, arousal, and tension are all zero. The remaining terms represent random intercepts for the individual (r_00i) and the stimulus (u_0ki) and the residual at the level of the individual joystick values (e_tki).

Modeling emotional valence, arousal, and tension as fixed effects significantly improved model fit over the intercept-only model, χ²(3) = 1,533.39, p < .001, Bayesian information criterion = 1,017,979. Joystick values were higher (more positive) for stimuli that evoked higher emotional valence ratings (β = 0.12) and lower tension ratings (β = −0.03), both ts > 7.57, both ps < .001. Joystick values were not reliably predicted by emotional arousal (β = 0.01), likely because high arousal could be evoked by either positive or negative stimuli.

Predicting patterns of perseverative responding

The next set of analyses tested the hypothesis that trait PT and clinical status would predict more negative joystick responses, specifically in the context of threat. As described below, trait PT and clinical status interacted with stimulus valence to predict more extreme joystick responses to all stimulus types.

We began by testing the main effects of stimulus valence, trait PT, and clinical status (dummy coded: nonclinical group = 0, clinical group = 1) on joystick values. Subsequent models tested the hypothesized interaction effects. Given the potential for multicollinearity, we tested interaction effects for PT and clinical status in two separate models (see Table 1, Steps 2a and 2b, respectively). Then, we ran the composite model to evaluate the effects of PT while covarying the effects of clinical status (see Table 1, Step 2c):

\begin{array}{l} J S_{t i} = γ_{00} + γ_{01} P T_{i} + γ_{02} C S_{i} + β_{1} N e g V a l_{t i} + β_{2} P o s V a l_{t i} \\ + β_{3} A m b V a l_{t i} + β_{4} N e g V a l_{t i} \times P T_{i} + β_{5} P o s V a l_{t i} \times P T_{i} \\ + β_{6} A m b V a l_{t i} \times P T_{i} + β_{7} N e g V a l_{t i} \times C S_{i} \\ + β_{8} P o s V a l_{t i} \times C S_{i} + β_{9} A m b V a l_{t i} \times C S_{i} + e_{t i} + u_{0 i}, \end{array}

in which JS_ti is modeled as a function of a random intercept and the fixed effects of stimulus negativity, positivity, and ambiguity (β₁, β₂, and β₃). The remaining fixed coefficients represent person i’s mean joystick value as a function of stimulus valence moderated by person i’s trait PT (β₄, β₅, and β₆) and clinical status (β₇, β₈, and β₉). The random coefficients (γ₀₁ and γ₀₂) represent person i’s mean joystick value as a function of person i’s trait PT and clinical status. The global intercept γ₀₀ is the mean of the time series when the fixed effects—trait PT, clinical status, and stimulus negativity, positivity, and ambiguity—are all zero.

Table 1.

Results of Hierarchical Models Predicting Joystick Value From Stimulus Valence, Clinical Status, and Trait Perseverative Thinking

Step and fixed effect	β	SE	t	p	LR χ² (df)	BIC
1					114,559.56 (5)***	1,518,738
StimNeg	−2.26	0.01	−199.52	< .001
StimPos	1.68	0.01	147.72	< .001
StimAmb	−1.34	0.02	−60.83	< .001
CS	−0.86	0.62	−1.40^a	.167
Trait PT	−0.07	0.06	−1.11^a	.273
2a					8,912.96 (3)***	1,509,863
StimNeg × Trait PT	−0.12	< 0.01	−48.04	< .001
StimPos × Trait PT	0.09	< 0.01	37.04	< .001
StimAmb × Trait PT	−0.13	< 0.01	−26.71	< .001
2b					3,932.69 (3)***	1,514,843
StimNeg × CS	−0.73	0.02	−29.20	< .001
StimPos × CS	0.69	0.02	27.55	< .001
StimAmb × CS	−0.72	0.05	−14.87	< .001
2c					9,967.91 (6)***	1,508,846
StimNeg × Trait PT	−0.20	< 0.01	−43.40	< .001
StimPos × Trait PT	0.12	< 0.01	25.40	< .001
StimAmb × Trait PT	−0.24	0.01	−26.31	< .001
StimNeg × CS	0.97	0.05	20.94	< .001
StimPos × CS	−0.30	0.05	−6.61	< .001
StimAmb × CS	1.27	0.09	14.22	< .001

Note: For Step 1, likelihood ratio (LR) χ² value refers to the comparison with the intercept-only model. For Steps 2a through 2c, LR χ² values refer to the comparison with Step 1. Steps 2a and 2b refer to separate, nonnested models (a PT-only model and a CS-only model). Step 2c refers to the composite model that evaluates the effects of PT while covarying out effects of CS. The degrees of freedom value for the t tests was 280,300 unless otherwise specified. StimNeg = stimulus negativity; StimPos = stimulus positivity; StimAmb = stimulus ambiguity; CS = clinical status; PT = perseverative thinking; BIC = Bayesian information criterion.

df = 60.

***

p < .001.

Effects of stimulus valence

For main effects, see Table 1, Step 1. Stimulus negativity, stimulus positivity, and stimulus ambiguity each predicted joystick value in the expected direction.

Effects of trait PT

There was no main effect of trait PT on joystick values because, as hypothesized, the effects of trait PT depended on stimulus valence. Trait PT interacted with each type of stimulus valence to predict more extreme joystick values, and including these interaction terms significantly improved model fit (Table 1, Step 2a). Simple-slopes analysis revealed that for negative and ambiguous stimuli, individuals with higher trait PT provided more negative joystick ratings than individuals with lower trait PT. For positive stimuli, higher trait PT increased the effect of positive stimuli, which resulted in more positive joystick ratings.

Effects of clinical status

As with trait PT, there was no main effect of clinical status on joystick values. Instead, clinical status moderated the effects of stimulus valence on joystick values, and each interaction term significantly improved model fit (Table 1, Step 2b). Simple-slopes analysis revealed that clinical status (compared with nonclinical status) predicted more extreme responses: more negative joystick values in the context of stimulus negativity and ambiguity and more positive values in the context of stimulus positivity. For additional details of these findings, see the Supplemental Material.

Composite model

Finally, we created a composite model that included the interaction terms for both trait PT and clinical status. Including both sets of interaction terms significantly improved model fit (Table 1, Step 2c). We examined the unique effect of trait PT by estimating the predicted joystick value for negative, positive, and ambiguous stimuli across the full range of possible PT values, controlling for the effect of clinical status. As depicted in Figure 1c, even when clinical status was controlled, increasing trait PT amplified the effects of stimulus valence on predicted joystick values.

Trait PT as a predictor of cognitive dynamics

Cognitive stability

Next, we examined whether trait PT predicts the stability of joystick responses. As described below, the relationship between PT and stability depended on stimulus valence: Higher PT was related to more stable responses after negative and ambiguous stimuli but less stable responses after positive stimuli.

In these analyses, the SSD between observations at occasions t and t – lag quantified the stability between each pair of time points. We calculated SSD for joystick values at lags of 5 s and 10 s before the current time point, which corresponded to lags of 20 and 40 observations, respectively, given our sampling rate of 4 observations per s. We selected these lags a priori to represent a meaningful length of time between observations.

The formula for SSD with a lag of 20 is SSD_ti = (JS_t – JS_t-20)². Because the distribution of SSDs is discrete, nonnegative, and positively skewed, it is well modeled by negative binomial regression (Hilbe, 2011). Thus, we used hierarchical generalized linear models with a negative binomial distribution and a log link function to model SSD:

\begin{array}{l} l n (E (S S D_{t i})) = γ_{0 i} + γ_{1 i} P T_{i} + γ_{2 i} C S_{i} + β_{1} N e g V a l_{t i} \\ + β_{2} P o s V a l_{t i} + β_{3} A m b V a l_{t i} \\ + β_{4} P T_{i} \times N e g V a l_{t i} + β_{5} P T_{i} \times P o s V a l_{t i} \\ + β_{6} P T_{i} \times A m b V a l_{t i} + e_{t i} + u_{0 i} . \end{array}

The outcome is the log of the expected, mean value of SSD_ti (the SSD score of the tth observation for person i), representing stability. This is modeled as a function of the grand mean of stability (the mean of participant means, γ_0i) and coefficients representing the relationship of stability to person i’s trait PT (γ_1i) and clinical status (γ_2i). We included clinical status as a predictor in the cognitive dynamics models to examine the effects of trait PT over and above the influence of clinical status.

Modeling SSD using 5- and 10-s lags produced the same pattern of results (Table 2). In both models, trait PT interacted with each stimulus valence variable to predict SSD. Simple-slopes analyses revealed that as trait PT increased, the marginal effects of stimulus negativity and stimulus ambiguity on SSD decreased. In other words, when the stimulus was negative or ambiguous, participants with higher trait PT showed more stable joystick ratings. In contrast, as trait PT increased, the marginal effect of stimulus positivity on SSD increased, indicating more variable responding to positive stimuli at higher levels of trait PT.

Table 2.

Results of Hierarchical Models Using Stimulus Valence and Trait Perseverative Thinking to Predict the Squared Successive Difference Between Joystick Values

Step and fixed effect	5-s lag						10-s lag
Step and fixed effect	IRR	SE	t	p	LR χ² (df)	BIC	IRR	SE	t	p	LR χ² (df)	BIC
1					1,619.46 (5)***	690,979					1,973.47 (5)***	699,602
Trait PT	1.29	0.06	5.87	< .001			1.32	0.06	5.67	< .001
CS	0.08	0.01	−18.93	< .001			0.07	0.02	−11.59	< .001
StimNeg	1.32	0.01	30.49	< .001			1.39	0.01	36.01	< .001
StimPos	1.43	0.01	39.46	< .001			1.46	0.01	42.61	< .001
StimAmb	1.45	0.03	21.63	< .001			1.49	0.03	23.44	< .001
2					140.93 (3)***	690,875					136.25 (3)***	699,502
Trait PT × StimNeg	0.99	< 0.01	−3.45	.001			0.99	< 0.01	−4.95	< .001
Trait PT × StimPos	1.01	< 0.01	6.57	< .001			1.01	< 0.01	2.93	.003
Trait PT × StimAmb	0.99	< 0.01	−3.10	.002			0.97	< 0.01	−7.11	< .001

Note: For every 1-point change in the predictor, the IRR represents the percentage change in the dependent variable. A 5-s lag represents the SSD between the joystick value at time t and the joystick value at t + 5 s. A 10-s lag represents the SSD between the joystick value at time t and the joystick value at t + 10 s. All models controlled for clinical status. Degrees of freedom for t tests are not reported because of the lack of consensus solutions for computing the approximate degrees of freedom for this modeling approach, as noted by the software developer (Bates et al., 2015). PT = perseverative thinking; CS = clinical status; StimNeg = stimulus negativity; StimPos = stimulus positivity; StimAmb = stimulus ambiguity; IRR = incidence rate ratio; LR = likelihood ratio; BIC = Bayesian information criterion; SSD = squared successive difference.

***

p < .001.

Cognitive inertia

Finally, we tested the effects of trait PT on cognitive inertia, or autocorrelation. As detailed below, joystick responses became less inert as stimulus intensity or ambiguity increased. Moreover, trait PT predicted greater inertia for ambiguous stimuli; higher trait perseverators appeared insensitive to changes in ambiguity.

In this series of models, autocorrelation between current and prior (lagged) joystick values reflected the degree to which previous ratings predicted current ratings, which served as an operationalization of “stuckness” of thought. Higher levels of autoregression indicated that prior values were more predictive of current values. All models included the centered lagged value as a random slope (Koval et al., 2012).

Effects of trait PT and stimulus valence on cognitive inertia

First, we tested the effect of trait PT on cognitive inertia, covarying the effects of stimulus valence. The effects of trait PT and stimulus valence on inertia are captured by the two-way interactions of the lagged joystick value with the trait PT and stimulus valence variables, respectively. The formula for modeling cognitive inertia with a lag of 20 is as follows:

\begin{array}{l} J S_{t i} = β_{0} + γ_{10} J S_{(t - 20) i} + γ_{11} P T_{i} \times J S_{(t - 20) i} \\ + γ_{12} N e g V a l_{t i} \times J S_{(t - 20) i} + γ_{13} P o s V a l_{t i} \times J S_{(t - 20) i} \\ + γ_{14} A m b V a l_{t i} \times J S_{(t - 20) i} + β_{2} C S_{i} + e_{t i} + u_{1 i} . \end{array}

The outcome (JS_ti) was modeled as a function of an intercept (β₀) representing the sample’s mean joystick value, the fixed effect for clinical status (β₂), and a random slope representing the autocorrelation, operationalized as the degree to which person i’s joystick value at the lagged timepoint (JS_t-20) predicted person i’s current joystick value. Autocorrelation was modeled as a function of person i’s trait PT score and the effects of stimulus valence. To facilitate interpretation of within-persons effects, the lagged predictor (JS_t-20) was person-mean centered.

The autoregression models at 5- and 10-s lags produced nearly the same pattern of results (Table 3, Step 2). At both lags, stimulus negativity and ambiguity moderated the relationship between the lagged and current joystick values; at the 10-s lag, stimulus positivity moderated the relationship as well. We probed these interactions using simple-slopes analyses. Higher levels of stimulus negativity, ambiguity, and (in the lag-10 model) positivity decreased the autocorrelation between lagged and current joystick values. In other words, as the intensity or ambiguity of the stimulus increased, joystick responses became less inert. By contrast, trait PT was not a significant moderator of cognitive inertia, although higher trait PT was associated with marginally higher inertia in both the 5-s-lag and 10-s-lag models (see Table 3, Step 2).

Table 3.

Results of Hierarchical Models Predicting Joystick Value From Trait Perseverative Thinking, Stimulus Valence, and Joystick Value Recorded 5 or 10 s Earlier

Step and fixed effect	5-s lag						10-s lag
Step and fixed effect	β	SE	t	p	LR χ² (df)	BIC	β	SE	t	p	LR χ² (df)	BIC
1					27,096.95 (6)***	997,137					22,040.32 (6)***	845,048
PT	−0.06	< 0.01	−36.65	< .001			−0.06	< 0.01	−28.30	< .001
CS	−0.83	0.02	−47.17	< .001			−0.75	0.02	−32.84	< .001
StimNeg	−0.37	0.01	−47.77	< .001			−0.64	0.01	−65.51	< .001
StimPos	0.26	0.01	35.01	< .001			0.47	0.01	48.83	< .001
StimAmb	−0.23	0.01	−16.87	< .001			−0.42	0.02	−23.71	< .001
LagVal	0.82	0.02	45.14^a	< .001			0.72	0.02	29.25^b	< .001
2					764.10 (4)***	996,422					636.65 (4)***	844,460
StimNeg × LagVal	−0.04	< 0.01	−21.18	< .001			−0.05	< 0.01	−20.79	< .001
StimPos × LagVal	< .01	< 0.01	−1.31	.192			−0.01	< 0.01	−3.71	< .001
StimAmb × LagVal	−0.06	< 0.01	−14.63	< .001			−0.08	0.01	−14.64	< .001
Trait PT × LagVal	0.01	< 0.01	1.68^c	.093			0.01	0.01	1.69^d	.090
3					414.37 (6)***	996,082					608.95 (6)***	843,924
StimNeg × PT × LagVal	0.00	< 0.01	6.12	< .001			0.00	< 0.01	7.41	< .001
StimPos × PT × LagVal	0.01	< 0.01	11.24	< .001			0.01	< 0.01	10.13	< .001
StimAmb × PT × LagVal	0.02	< 0.01	16.91	< .001			0.02	< 0.01	17.30	< .001

Note: A 5-s lag represents the autocorrelation between the joystick value at time t and the joystick value at t + 5 s. A 10-s lag represents the autocorrelation between the joystick value at time t and the joystick value at t + 10 s. All models controlled for clinical status. The degrees of freedom values for the t tests were 232,800 (for the 5-s lag) and 185,300 (for the 10-s lag) unless otherwise specified. PT = perseverative thinking; CS = clinical status; StimNeg = stimulus negativity; StimPos = stimulus positivity; StimAmb = stimulus ambiguity; LagVal = joystick value recorded 5 or 10 s earlier; LR = likelihood ratio; BIC = Bayesian information criterion.

df = 49. ^bdf = 55. ^cdf = 48. ^ddf = 54.

***

p < .001.

Effects of Trait PT × Stimulus Valence interactions on cognitive inertia

Next, we tested whether the effect of trait PT on cognitive inertia depends on the type of stimulus that was just presented. We did this by introducing interactions between trait PT and stimulus valence into the equation shown above. The effects of interest were the three-way interactions between trait PT, stimulus valence, and lagged joystick value in predicting the current joystick value.

The autoregression models at 5- and 10-s lags produced identical results: Each three-way interaction between stimulus valence, trait PT, and the lagged value significantly predicted the current joystick value (Table 3, Step 3). Including these three-way interaction terms significantly improved model fit for both the 5-s-lag and 10-s-lag models. Given our particular interest in explaining inertia between earlier and later thoughts, we focused on analyses that tested the marginal effect of lagged joystick value on current joystick value as a function of trait PT and stimulus valence. Those analyses revealed significant results only for ambiguous stimuli. In both the 5-s-lag and 10-s-lag models, ambiguous stimuli led to more inert responding (i.e., higher autocorrelation) at higher levels of trait PT (Fig. 1d). Whereas low trait perseverators were sensitive to changes in ambiguity, exhibiting lower inertia after ambiguous stimuli than unambiguous stimuli, high trait perseverators responded with similarly high inertia to ambiguous and unambiguous stimuli. For the two-way interactions between trait PT and lagged joystick value, plotted separately for ambiguous stimuli and unambiguous stimuli, see Figure S2 in the Supplemental Material.

Discussion

Available measures of PT are ill-equipped to capture dynamic features of this rapidly unfolding process, which raises a pressing need for a new approach to measurement. In the present study, we tested a novel joystick approach for measuring PT in real time. Participants varying widely in trait PT used a joystick to provide 30 s of continuous ratings of the valence and intensity of their thoughts. Ratings were made following exposure to 36 scenarios describing negative (threatening), positive (rewarding), ambiguous, and neutral life events. Joystick ratings were reliably predicted by trait PT, clinical status, and stimulus valence. Higher trait perseverators exhibited more extreme joystick values overall, greater stability in values following threatening and ambiguous stimuli, weaker stability in values following positive stimuli, and greater inertia in values following ambiguous stimuli. These findings highlight the promise of the joystick method for illuminating the experience of PT while offering new insights into the nature and triggers of this clinically important process.

Several lines of evidence supported the validity of the joystick measure. Mean joystick levels increased monotonically across the six stimulus categories, from major negative stimuli through neutral stimuli to major positive stimuli; ambiguous stimuli fell between the minor negative and neutral categories. In a hierarchical linear model, composite variables of stimulus negativity, stimulus positivity, and stimulus ambiguity each significantly predicted joystick value in the expected direction. Furthermore, joystick value was related to participants’ ratings of emotional valence and tension for the same stimuli, which indicated that scenarios that were more emotionally evocative for the participant triggered more intense thoughts. These results provide promising early evidence for the joystick as a valid, sensitive measure of thoughts experienced across a range of eliciting contexts.

Our primary interest in this study was to predict the patterns of cognitive activity reported by individuals with different levels of trait PT. We found that as trait PT increased, participants’ cognitive responses to stimuli became more extreme: Responses to negative and ambiguous stimuli became more negative, and responses to positive stimuli became more positive. This implies that high trait PT is associated with amplified responding to emotional stimuli in general, rather than to negative or ambiguous stimuli specifically. One possible explanation for these findings is that PT sensitizes individuals to the stimuli they experience, perhaps by strengthening affective and cognitive states and making them more salient (Watkins, 2008). A different possibility is that it is not the PT process itself, but some other characteristic of high perseverators, that magnifies reactions to valenced stimuli. For example, neuroticism, a personality dimension strongly associated with both PT (du Pont et al., 2019) and emotional disorders (Barlow et al., 2014), has high emotional reactivity as a central feature.

Our finding that clinical status also interacted with stimulus valence to predict more extreme joystick values and that clinical status and trait PT each independently predicted heightened responding when controlling for the other hints that emotional and cognitive processes may contribute to the amplified responses observed here. For example, we found that clinical participants responded to ambiguous scenarios with negative thoughts on average, whereas nonclinical participants responded with neutral to positive thoughts. This downward bias by clinical participants is consistent with the negative cognitive styles observed in these disorders (Alloy et al., 1999; Beck, 1967, 1987). It may also reflect selective processing of emotionally negative information and the imposition of negative meanings on ambiguous information, biases that are common in anxiety and depression (Hirsch et al., 2016; Mathews & MacLeod, 2005).

The continuous data stream yielded by the joystick, sampled at a rate of 250 ms, also enabled us to examine the dynamics of cognitive responding and its association with trait PT. We began by examining the SSD, which reflects stability (or degree of change) from moment to moment in a time series. Following exposure to negative and ambiguous stimuli, participants with higher trait PT experienced a more stable pattern of cognitive responding, exhibiting smaller changes in thought valence over time. By contrast, following exposure to positive stimuli, participants with higher trait PT showed larger shifts in the moment-to-moment valence of their thoughts. Stable joystick values in the wake of negative and ambiguous stimuli may reflect the subjective experience of being stuck in negative thinking, whereas more variable joystick values in response to positive stimuli may reflect difficulty maintaining a positive train of thoughts. We explored joystick dynamics further by examining autocorrelation, a measure of inertia. Stronger inertia reflects response patterns that are more self-predictive over time and show relatively less return to baseline. Although there was a trend for individuals higher in trait PT to show more inert responding, the effect was significant only in interaction with stimulus ambiguity: Individuals with higher trait PT showed more inert thoughts across the board, regardless of ambiguity, whereas individuals with lower trait PT showed less inert thoughts after ambiguous stimuli than unambiguous stimuli. These results imply that a low degree of self-predictiveness is adaptive in response to ambiguous stimuli. Low inertia may reflect negative thoughts that are transient and tend not to linger over time, a tendency to see both the negative and nonnegative (benign or positive) interpretation of the ambiguous stimulus, or greater homeostatic recovery between thoughts. Whatever the reason for low inertia, the pattern of differential responding suggests that low trait perseverators are more sensitive to ambiguity as an informative stimulus feature and better able to flexibly adjust their thoughts in response to that feature rather than reacting to all stimuli with similar perseveration.

Although preliminary, these joystick results challenge some common assumptions about the nature of PT. First, trait PT is not associated with a blanket tendency to experience negative thoughts, at least during the time period directly proximal to stimulus exposure. Instead, thought content in PT appears to involve extremes of thinking rather than negative valence per se. This surprising finding runs counter to prevailing definitions of PT, which identify negative content as a core feature of the construct. Positively valenced forms of repetitive thinking do exist, such as positive rumination, typically studied in the context of mania and hypomania (Watkins, 2008). However, those forms of thinking involve prolonged processing of positive material, which has not traditionally been associated with depression or anxiety and was not evident in our data.

Second, high trait perseverators are not globally prone to persistent, stuck thinking. The PT style appears to involve relatively stable responses to negative stimuli, relatively unstable responses to positive stimuli, and relatively stable and self-predictive responses to ambiguous stimuli. In showing that both the content and process of thoughts depend on the eliciting context, our findings highlight the importance of studying cognitive responses to a broad range of stimuli because including only negative or ambiguous stimuli—which predominate in research on PT—would have resulted in an incomplete understanding of the phenomenon. In showing that trait PT is robustly associated with heightened, abbreviated cognitive responses to positive stimuli, our findings also raise more fundamental questions about what sorts of thoughts should be part of the definition of perseverative thinking, a term that is used interchangeably with repetitive negative thinking in the literature (Ehring et al., 2011).

Future directions

The joystick method is particularly well suited to identifying specific components of the temporal dynamics of PT. Theoretical accounts have emphasized the repetitive or stuck nature of PT, and we examined two temporal characteristics, SSD and autocorrelation, that might operationalize this component. Future research could identify other components of negative thought experienced by high perseverators, such as peak amplitude of negative response, rise time to peak, recovery time, degree of completeness in return to baseline, likelihood of recurrence, or the form and frequency of recurrence (Davidson, 1998). These components could be studied at rest, in the context of evocative stimuli, or during tasks explicitly requiring control over thoughts to isolate the features that best discriminate PT from occasional negative thoughts.

Another valuable direction would be to move beyond modeling group-level differences to inspecting within-persons variation in the joystick data. Dynamical systems modeling analyzes individual fluctuation within trajectories of data, which allows examination of how symptoms change within individuals over time (Frank et al., 2017). For example, dynamic modeling could be used to examine the rate of return to baseline following negative thought and the speed of this movement within the individual (Chow et al., 2005). Investigating individual-level temporal characteristics of intrusive thought not only could provide a more fine-grained understanding of PT but also could aid clinicians in designing personalized treatments to target the precise pattern of thinking in which a particular patient is getting stuck (e.g., Fisher & Boswell, 2016).

Going forward, the joystick method could be used to characterize episodes of PT as they occur in vivo. Future work might modify the joystick device to become handheld and portable, engineering it to wirelessly transmit data to a personal device. Such a method could be used to fully and richly characterize the experience of a bout of PT in the natural environment. This method could be combined with personal sensing data (Mohr et al., 2017) to shed light on the factors that maintain or disrupt bouts of PT in daily life.

Strengths and limitations

Our study had several limitations. We recruited participants who met established diagnostic criteria to ensure that our study would be clinically informative while at the same time analyzing PT as a dimensional construct outside the bounds of diagnostic categories. This design was a strength of our study given that it addressed calls to study dimensional processes that cut across disorders (Harvey et al., 2004). However, although our sample spanned a wide range of trait PT scores, nonclinical participants made up a small portion of the sample. Consequently, our results likely say more about pathological PT than about normal-range experiences of worry or rumination.

We measured PT rigorously using a composite of five questionnaires. This enabled us to capture many different facets of the PT construct in a robust, reliable measure rather than relying on a single stand-alone questionnaire. Despite these strengths, our PT composite was subject to the limitations of all global questionnaires, such as negative biases and errors in recall. Absent a gold-standard measure of PT, these questionnaires offer an imperfect yet necessary first step toward establishing the construct validity of our novel joystick method. Converging evidence provided by other measures—including interviewer-assessed clinical status, experimenter-classified stimulus valence, and state ratings of emotional valence and tension—increase confidence in our validity results. Nevertheless, there is a need for further validation using other types of measures. Because the joystick is fundamentally a state measure, it would be valuable for future research to establish convergent validity with other state measures of PT, such as EMA ratings, and to demonstrate incremental validity in predicting other real-time measures of experience, such as momentary symptom levels.

Relatedly, when a new measure is introduced, it is important to consider the psychometric model that is being invoked. Here, we posit a reflective measurement model (Borsboom et al., 2003) in which the latent construct of PT gives rise to observed patterns of joystick responding. We expect that the observed joystick ratings align more closely with the unobserved latent PT construct than trait questionnaire scores, which are influenced by the participant’s ability to accurately search and aggregate information across memory. That said, we made a number of assumptions in developing the joystick measure. One assumption was that cognitive experience is best represented by a single valence dimension rather than, for example, a two-dimensional valence-by-arousal space, which is more conventional in the affect literature (Posner et al., 2005). Another assumption concerned the influence of shared factors on the joystick ratings. Factors such as willingness to disclose thoughts, yay- or nay-saying biases, or even physical dexterity may well exert systematic effects on joystick values across stimuli. The implicit measurement model posited in this article specifies no such shared factors. Whether the data actually conform to this model is an important question for future research to address.

We recruited participants with GAD and MDD because PT is a prominent clinical feature of and most often studied in connection with these disorders (Ehring & Watkins, 2008). Nevertheless, PT also occurs in other disorders, and research is needed to determine the generalizability of our results to other disorders in which PT plays a role (Brozovich & Heimberg, 2008; Ehring et al., 2008; Julien et al., 2007, Lancee et al., 2017). Given the transdiagnostic nature of PT, future research could use the joystick to study common features of PT across disorders. Alternatively, researchers could focus more narrowly on cognitive processes within disorders, such as using the joystick to measure postevent processing following a social-interaction task in social anxiety disorder (e.g., Dannahy & Stopa, 2007) or to measure intrusive memories following an analogue stressor in posttraumatic stress disorder (e.g., Holmes & Bourne, 2008). Relatedly, future research could use the joystick as a new tool for exploring process features that may distinguish worry from rumination to help inform ongoing debates over the merits of separating these closely related constructs in the literature.

We used brief, imaginal stimuli delivered in a controlled laboratory setting. A strength of this approach was that it allowed us to expose participants to a range of stimuli that varied systematically in their valence and intensity. As a result, we were able to draw more precise conclusions about the conditions under which PT is likely to occur. However, in presenting discrete stimuli and then measuring thoughts, we were unable to study what happens to PT when events evolve over time or require the individual to act. Although our use of real-world scenarios as stimuli bolsters the external validity of these results, research involving other types of stimuli, particularly dynamic stimuli more closely resembling those encountered in daily life, will be important for gaining a full understanding of the natural course of PT.

Measuring cognitions, which are internal by nature, poses a challenge in any research, especially when the goal is to measure thought in an on-line fashion. There is a possibility that for participants, the process of reporting on thoughts changed the very nature of the thoughts. To minimize interference with the natural flow of thoughts, we created a device that allowed participants to report continuously on their thoughts using a physical action (i.e., pushing or pulling the joystick), which may be less disruptive to the PT process than interrupting thinking with a signal and asking participants to read questions and complete ratings about their thoughts. We also included a practice period to increase participants’ familiarity and comfort with the device, and thereby enhance automaticity, before the experiment began. In our pilot study, more than three quarters of participants indicated they felt able to use the joystick to accurately reflect the intensity of their thoughts. Whether using the joystick interferes with thoughts requires further study; however, given the covert nature of the phenomenon being measured, an objective test of the impact of joystick assessment on thoughts may be difficult to achieve.

We recorded the acute response to each stimulus over an interval of 30 s. Given the short time frame, our paradigm should be interpreted as measuring the initial response to a stimulus rather than as capturing complete episodes of PT. This idea is in line with Berenbaum’s (2010) two-phase model of worrying, which considers the initiation and termination phases of worry as two separate stages. In keeping with this model, thoughts occurring in the immediate aftermath of the stimuli likely reflected the initiation phase of PT. It is unclear whether our paradigm captured termination in some cases, and even if it did, it is possible that the standardized 30-s trial structure led some individuals to anticipate the end of the trial and work purposefully to disengage from negative thoughts. If this is the case, our findings may not represent the natural process of disengagement from PT. Future research could use the joystick to measure a longer period of thinking, both to reveal the shape of a complete PT episode and to see whether it is possible to detect two distinct phases of PT.

Conclusions

The joystick method represents a promising new measure of PT. The ability to measure PT in a near-continuous fashion opens up new avenues for studying this dynamic cognitive process. It allows theorized features of PT—such as its repetitive nature—to be formalized and tested empirically. It also allows temporal components of the thought experience to be linked in real time to antecedents, maintaining factors, and functional outcomes of PT. These advances have the potential to yield novel insights into the nature of PT and provide much needed guidance for interventions aimed at breaking the cycle of unconstructive repetitive thinking.

Supplemental Material

sj-pdf-1-cpx-10.1177_1049732320931430 – Supplemental material for A Novel Measure of Real-Time Perseverative Thought

Supplemental material, sj-pdf-1-cpx-10.1177_1049732320931430 for A Novel Measure of Real-Time Perseverative Thought by Elizabeth C. Stade, Rivka T. Cohen, Paddy Loftus and Ayelet Meron Ruscio in Clinical Psychological Science

Footnotes

Acknowledgements

Sections of this article were included in a poster presented at the Anxiety and Depression Association of America Annual Conference in March 2019.

Transparency

Action Editor: Andrew Littlefield

Editor: Kenneth J. Sher

Author Contributions

A. M. Ruscio developed the study concept, design, and materials and supervised data collection. E. C. Stade developed the hypotheses and drafted the manuscript, and E. C. Stade and R. T. Cohen performed the data analysis and interpretation under the supervision of A. M. Ruscio. All of the authors provided critical revisions of the manuscript and approved the final manuscript for submission.

ORCID iDs

Elizabeth C. Stade

Paddy Loftus

Supplemental Material

Additional supporting information can be found at

References

Alloy

L. B.

Abramson

L. Y.

Whitehouse

W. G.

Hogan

M. E.

Tashman

N. A. L.

Steinberg

Rose

D. T.

Donovan

(1999). Depressogenic cognitive styles: Predictive validity, information processing and personality characteristics, and developmental origins. Behaviour Research and Therapy, 37(6), 503–531. https://doi.org/10.1016/S0005-7967(98)00157-0

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed). https://doi.org/10.1176/appi.books.9780890425596

Barlow

D. H.

Sauer-Zavala

Carl

J. R.

Bullis

J. R.

Ellard

K. K.

(2014). The nature, diagnosis, and treatment of neuroticism: Back to the future. Clinical Psychological Science, 2(3), 344–365. https://doi.org/10.1177/2167702613505532

Bates

Mächler

Bolker

Walker

(2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01

Beck

A. T.

(1967). Depression: Clinical, experimental, and theoretical aspects. Harper & Row.

Beck

A. T.

(1987). Cognitive models of depression. Journal of Cognitive Psychotherapy: An International Quarterly, 1, 5–37.

Berenbaum

(2010). An initiation–termination two-phase model of worrying. Clinical Psychology Review, 30(8), 962–975. https://doi.org/10.1016/j.cpr.2010.06.011

Borkovec

T. D.

Newman

M. G.

Pincus

A. L.

Lytle

(2002). A component analysis of cognitive-behavioral therapy for generalized anxiety disorder and the role of interpersonal problems. Journal of Consulting and Clinical Psychology, 70(2), 288–298. https://doi.org/10.1037//0022-006X.70.2.288

Borkovec

T. D.

Robinson

Pruzinsky

DePree

J. A.

(1983). Preliminary exploration of worry: Some characteristics and processes. Behaviour Research and Therapy, 21(1), 9–16. https://doi.org/10.1016/0005-7967(83)90121-3

10.

Borsboom

Mellenbergh

G. J.

van Heerden

(2003). The theoretical status of latent variables. Psychological Review, 110(2), 203–219. https://doi.org/10.1037/0033-295X.110.2.203

11.

Brosschot

J. F.

Gerin

Thayer

J. F.

(2006). The perseverative cognition hypothesis: A review of worry, prolonged stress-related physiological activation, and health. Journal of Psychosomatic Research, 60(2), 113–124. https://doi.org/10.1016/j.jpsychores.2005.06.074

12.

Brown

T. A.

Barlow

D. H.

(2014). Anxiety and related disorders interview schedule for DSM-5 (ADIS-5L): Client interview schedule. Oxford University Press.

13.

Brozovich

Heimberg

R. G.

(2008). An analysis of post-event processing in social anxiety disorder. Clinical Psychology Review, 28(6), 891–903. https://doi.org/10.1016/j.cpr.2008.01.002

14.

Bylsma

L. M.

Morris

B. H.

Rottenberg

(2008). A meta-analysis of emotional reactivity in major depressive disorder. Clinical Psychology Review, 28(4), 676–691. https://doi.org/10.1016/j.cpr.2007.10.001

15.

Bylsma

L. M.

Taylor-Clift

Rottenberg

(2011). Emotional reactivity to daily events in major and minor depression. Journal of Abnormal Psychology, 120(1), 155–167. https://doi.org/10.1037/a0021662

16.

Chow

S.-M.

Ram

Boker

S. M.

Fujita

Clore

(2005). Emotion as a thermostat: Representing emotion regulation using a damped oscillator model. Emotion, 5(2), 208–225. https://doi.org/10.1037/1528-3542.5.2.208

17.

Craske

M. G.

Rauch

S. L.

Ursano

Prenoveau

Pine

D. S.

Zinbarg

R. E.

(2009). What is an anxiety disorder? Depression and Anxiety, 26(12), 1066–1085. https://doi.org/10.1002/da.20633

18.

Dannahy

Stopa

(2007). Post-event processing in social anxiety. Behaviour Research and Therapy, 45(6), 1207–1219. https://doi.org/10.1016/j.brat.2006.08.017

19.

Davidson

R. J.

(1998). Affective style and affective disorders: Perspectives from affective neuroscience. Cognition & Emotion, 12(3), 307–330. https://doi.org/10.1080/026999398379628

20.

Dichter

G. S.

(2010). Anhedonia in unipolar major depressive disorder: A review. The Open Psychiatry Journal, 4(1), 1–9. https://doi.org/10.2174/1874354401004010001

21.

du Pont

Rhee

S. H.

Corley

R. P.

Hewitt

J. K.

Friedman

N. P.

(2019). Are rumination and neuroticism genetically or environmentally distinct risk factors for psychopathology? Journal of Abnormal Psychology, 128(5), 385–396. https://doi.org/10.1037/abn0000430

22.

Dupuy

J.-B.

Beaudoin

Rhéaume

Ladouceur

Dugas

M. J.

(2001). Worry: Daily self-report in clinical and non-clinical populations. Behaviour Research and Therapy, 39(10), 1249–1255. https://doi.org/10.1016/S0005-7967(01)00011-0

23.

Ehring

Frank

Ehlers

(2008). The role of rumination and reduced concreteness in the maintenance of posttraumatic stress disorder and depression following trauma. Cognitive Therapy and Research, 32(4), 488–506. https://doi.org/10.1007/s10608-006-9089-7

24.

Ehring

Watkins

E. R.

(2008). Repetitive negative thinking as a transdiagnostic process. International Journal of Cognitive Therapy, 1(3), 192–205.

25.

Ehring

Zetsche

Weidacker

Wahl

Schönfeld

Ehlers

(2011). The Perseverative Thinking Questionnaire (PTQ): Validation of a content-independent measure of repetitive negative thinking. Journal of Behavior Therapy and Experimental Psychiatry, 42(2), 225–232. https://doi.org/10.1016/j.jbtep.2010.12.003

26.

Feldman

G. C.

Joormann

Johnson

S. L.

(2008). Responses to positive affect: A self-report measure of rumination and dampening. Cognitive Therapy and Research, 32(4), 507–525. https://doi.org/10.1007/s10608-006-9083-0

27.

Fisher

A. J.

Boswell

J. F.

(2016). Enhancing the personalization of psychotherapy with dynamic assessment and modeling. Assessment, 23(4), 496–506. https://doi.org/10.1177/1073191116638735

28.

Frank

Jacobson

N. C.

Hurley

McKay

(2017). A theoretical and empirical modeling of anxiety integrated with RDoC and temporal dynamics. Journal of Anxiety Disorders, 51, 39–46. https://doi.org/10.1016/j.janxdis.2017.09.002

29.

Gilbert

K. E.

Nolen-Hoeksema

Gruber

(2013). Positive emotion dysregulation across mood disorders: How amplifying versus dampening predicts emotional reactivity and illness course. Behaviour Research and Therapy, 51(11), 736–741. https://doi.org/10.1016/j.brat.2013.08.004

30.

Gruber

Eidelman

Johnson

S. L.

Smith

Harvey

A. G.

(2011). Hooked on a feeling: Rumination about positive and negative emotion in inter-episode bipolar disorder. Journal of Abnormal Psychology, 120(4), 956–961. https://doi.org/10.1037/a0023667

31.

Hamilton

(1959). The assessment of anxiety states by rating. British Journal of Medical Psychology, 32(1), 50–55. https://doi.org/10.1111/j.2044-8341.1959.tb00467.x

32.

Hamilton

(1960). A rating scale for depression. Journal of Neurology, Neurosurgery, and Psychiatry, 23, 56–62.

33.

Harvey

A. G.

Watkins

Mansell

Shafran

(2004). Cognitive behavioural processes across psychological disorders: A transdiagnostic approach to research and treatment. Oxford University Press.

34.

Hilbe

J. M.

(2011). Negative binomial regression (2nd ed.). Cambridge University Press.

35.

Hirsch

C. R.

Hayes

Mathews

(2009). Looking on the bright side: Accessing benign meanings reduces worry. Journal of Abnormal Psychology, 118(1), 44–54. https://doi.org/10.1037/a0013473

36.

Hirsch

C. R.

Meeten

Krahé

Reeder

(2016). Resolving ambiguity in emotional disorders: The nature and role of interpretation biases. Annual Review of Clinical Psychology, 12, 281–305. https://doi.org/10.1146/annurev-clinpsy-021815-093436

37.

Holmes

E. A.

Bourne

(2008). Inducing and modulating intrusive emotional memories: A review of the trauma film paradigm. Acta Psychologica, 127(3), 553–566. https://doi.org/10.1016/j.actpsy.2007.11.002

38.

Houben

Van Den Noortgate

Kuppens

(2015). The relation between short-term emotion dynamics and psychological well-being: A meta-analysis. Psychological Bulletin, 141(4), 901–930. https://doi.org/10.1037/a0038822

39.

Jennings

J. R.

Kamarck

Stewart

Eddy

Johnson

(1992). Alternate cardiovascular baseline assessment techniques: Vanilla or resting baseline. Psychophysiology, 29(6), 742–750. https://doi.org/10.1111/j.1469-8986.1992.tb02052.x

40.

Johnson

S. L.

McKenzie

McMurrich

(2008). Ruminative responses to negative and positive affect among students diagnosed with bipolar disorder and major depressive disorder. Cognitive Therapy and Research, 32(5), 702–713. https://doi.org/10.1007/s10608-007-9158-6

41.

Julien

O’Connor

K. P.

Aardema

(2007). Intrusive thoughts, obsessions, and appraisals in obsessive–compulsive disorder: A critical review. Clinical Psychology Review, 27(3), 366–383. https://doi.org/10.1016/j.cpr.2006.12.004

42.

Khazanov

G. K.

Ruscio

A. M.

Swendsen

(2019). The “brightening” effect: Reactions to positive events in the daily lives of individuals with major depressive disorder and generalized anxiety disorder. Behavior Therapy, 50(2), 270–284. https://doi.org/10.1016/j.beth.2018.05.008

43.

Kircanski

Thompson

R. J.

Sorenson

J. E.

Sherdell

Gotlib

I. H.

(2015). Rumination and worry in daily life: Examining the naturalistic validity of theoretical constructs. Clinical Psychological Science, 3(6), 926–939. https://doi.org/10.1177/2167702614566603

44.

Koval

Kuppens

Allen

N. B.

Sheeber

(2012). Getting stuck in depression: The roles of rumination and emotional inertia. Cognition & Emotion, 26(8), 1412–1427. https://doi.org/10.1080/02699931.2012.667392

45.

Kuppens

Verduyn

(2017). Emotion dynamics. Current Opinion in Psychology, 17, 22–26. https://doi.org/10.1016/j.copsyc.2017.06.004

46.

Lancee

Eisma

M. C.

van Zanten

K. B.

Topper

(2017). When thinking impairs sleep: Trait, daytime and nighttime repetitive thinking in insomnia. Behavioral Sleep Medicine, 15(1), 53–69. https://doi.org/10.1080/15402002.2015.1083022

47.

Lang

P. J.

McTeague

L. M.

Cuthbert

B. N.

(2007). Fear, anxiety, depression, and the anxiety disorder spectrum: A psychophysiological analysis. In Treat

T. A.

Bootzin

R. R.

(Eds.), Psychological clinical science: Papers in Honor of Richard M. McFall (pp. 167–195). Psychology Press. https://doi.org/10.4324/9780203810019

48.

Lüdecke

(2020). sjPlot: Data visualization for statistics in social science. Zenodo. https://doi.org/10.5281/zenodo.1308157

49.

Lyubomirsky

Nolen-Hoeksema

(1993). Self-perpetuating properties of dysphoric rumination. Journal of Personality and Social Psychology, 65(2), 339–349.

50.

Mathews

MacLeod

(2005). Cognitive vulnerability to emotional disorders. Annual Review of Clinical Psychology, 1, 167–195. https://doi.org/10.1146/annurev.clinpsy.1.102803.143916

51.

Mauss

I. B.

Levenson

R. W.

McCarter

Wilhelm

F. H.

Gross

J. J.

(2005). The tie that binds? Coherence among emotion experience, behavior, and physiology. Emotion, 5(2), 175–190. https://doi.org/10.1037/1528-3542.5.2.175

52.

McEvoy

P. M.

Brans

(2013). Common versus unique variance across measures of worry and rumination: Predictive utility and mediational models for anxiety and depression. Cognitive Therapy and Research, 37(1), 183–196. https://doi.org/10.1007/s10608-012-9448-5

53.

McEvoy

P. M.

Mahoney

A. E. J.

Moulds

M. L.

(2010). Are worry, rumination, and post-event processing one and the same? Development of the Repetitive Thinking Questionnaire. Journal of Anxiety Disorders, 24(5), 509–519. https://doi.org/10.1016/j.janxdis.2010.03.008

54.

McLaughlin

K. A.

Borkovec

T. D.

Sibrava

N. J.

(2007). The effects of worry and rumination on affect states and cognitive activity. Behavior Therapy, 38(1), 23–38. https://doi.org/10.1016/j.beth.2006.03.003

55.

Meyer

T. J.

Miller

M. L.

Metzger

R. L.

Borkovec

T. D.

(1990). Development and validation of the Penn State Worry Questionnaire. Behaviour Research and Therapy, 28(6), 487–495. https://doi.org/10.1016/0005-7967(90)90135-6

56.

Mineka

Rafaeli

Yovel

(2003). Cognitive biases in emotional disorders: Information processing and social-cognitive perspectives. In Davidson

Scherer

Goldsmith

(Eds.), Handbook of affective sciences (pp. 976–1009). Oxford University Press.

57.

Moberly

N. J.

Watkins

E. R.

(2008). Ruminative self-focus and negative affect: An experience sampling study. Journal of Abnormal Psychology, 117(2), 314–323. https://doi.org/10.1037/0021-843X.117.2.314

58.

Mohr

D. C.

Zhang

Schueller

S. M.

(2017). Personal sensing: Understanding mental health using ubiquitous sensors and machine learning. Annual Review of Clinical Psychology, 13, 23–47. https://doi.org/10.1146/annurev-clinpsy-032816-044949

59.

Nolen-Hoeksema

(1991). Responses to depression and their effects on the duration of depressive episodes. Journal of Abnormal Psychology, 100(4), 569–582.

60.

Nolen-Hoeksema

(2000). The role of rumination in depressive disorders and mixed anxiety/depressive symptoms. Journal of Abnormal Psychology, 109(3), 504–511.

61.

Nolen-Hoeksema

Wisco

B. E.

Lyubomirsky

(2008). Rethinking rumination. Perspectives on Psychological Science, 3(5), 400–424. https://doi.org/10.1111/j.1745-6924.2008.00088.x

62.

Olatunji

B. O.

Broman-Fulks

J. J.

Bergman

S. M.

Green

B. A.

Zlomke

K. R.

(2010). A taxometric investigation of the latent structure of worry: Dimensionality and associations with depression, anxiety, and stress. Behavior Therapy, 41(2), 212–228. https://doi.org/10.1016/j.beth.2009.03.001

63.

Posner

Russell

J. A.

Peterson

B. S.

(2005). The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Development and Psychopathology, 17(3), 715–734. https://doi.org/10.1017/S0954579405050340

64.

R Core Team. (2019). R: A language and environment for statistical computing (Version 3.6.1) [Computer software]. R Foundation for Statistical Computing. https://www.R-project.org/

65.

Ruscio

A. M.

Borkovec

T. D.

(2004). Experience and appraisal of worry among high worriers with and without generalized anxiety disorder. Behaviour Research and Therapy, 42(12), 1469–1482. https://doi.org/10.1016/j.brat.2003.10.007

66.

Ruscio

A. M.

Borkovec

T. D.

Ruscio

(2001). A taxometric investigation of the latent structure of worry. Journal of Abnormal Psychology, 110(3), 413–422.

67.

Ruscio

A. M.

Gentes

E. L.

Jones

J. D.

Hallion

L. S.

Coleman

E. S.

Swendsen

(2015). Rumination predicts heightened responding to stressful life events in major depressive disorder and generalized anxiety disorder. Journal of Abnormal Psychology, 124(1), 17–26. https://doi.org/10.1037/abn0000025

68.

Ruscio

A. M.

Seitchik

A. E.

Gentes

E. L.

Jones

J. D.

Hallion

L. S.

(2011). Perseverative thought: A robust predictor of response to emotional challenge in generalized anxiety disorder and major depressive disorder. Behaviour Research and Therapy, 49(12), 867–874. https://doi.org/10.1016/j.brat.2011.10.001

69.

Samtani

Moulds

M. L.

(2017). Assessing maladaptive repetitive thought in clinical disorders: A critical review of existing measures. Clinical Psychology Review, 53, 14–28. https://doi.org/10.1016/j.cpr.2017.01.007

70.

Solt

(2019). interplot: Plot the effects of variables in interaction terms (Version 0.2.2) [Computer software]. Comprehensive R Archive Network. https://cran.r-project.org/package=interplot

71.

Tallis

Eysenck

Mathews

(1992). A questionnaire for the measurement of nonpathological worry. Personality and Individual Differences, 13(2), 161–168. https://doi.org/10.1016/0191-8869(92)90038-Q

72.

Thielsch

Ehring

Nestler

Wolters

Kopei

Rist

Gerlach

A. L.

Andor

(2015). Metacognitions, worry and sleep in everyday life: Studying bidirectional pathways using ecological momentary assessment in GAD patients. Journal of Anxiety Disorders, 33, 53–61. https://doi.org/10.1016/j.janxdis.2015.04.007

73.

Trapnell

P. D.

Campbell

J. D.

(1999). Private self-consciousness and the five-factor model of personality: Distinguishing rumination from reflection. Journal of Personality and Social Psychology, 76(2), 284–304.

74.

Treynor

Gonzalez

Nolen-Hoeksema

(2003). Rumination reconsidered: A psychometric analysis. Cognitive Therapy and Research, 27(3), 247–259.

75.

Venables

W. N.

Ripley

B. D.

(2002). Modern applied statistics with S (4th ed.). Springer. http://www.stats.ox.ac.uk/pub/MASS4

76.

Watkins

E. R.

(2004). Adaptive and maladaptive ruminative self-focus during emotional processing. Behaviour Research and Therapy, 42(9), 1037–1052. https://doi.org/10.1016/j.brat.2004.01.009

77.

Watkins

E. R.

(2008). Constructive and unconstructive repetitive thought. Psychological Bulletin, 134(2), 163–206. https://doi.org/10.1037/0033-2909.134.2.163

78.

Zeileis

Hothorn

(2002). Diagnostic checking in regression relationships. R News, 2(3), 7–10.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB