Measures of Motivation for Psychiatric Treatment Based on Self-Determination Theory

Abstract

Self-determination theory is potentially useful for understanding reasons why individuals with mental illness do or do not engage in psychiatric treatment. The current study examined the psychometric properties of three questionnaires based on self-determination theory—The Treatment Entry Questionnaire (TEQ), Health Care Climate Questionnaire (HCCQ), and the Short Motivation Feedback List (SMFL)—in a sample of 348 Dutch adult outpatients with primary diagnoses of mood, anxiety, psychotic, and personality disorders. Structural equation modeling showed that the empirical factor structures of the TEQ and SMFL were adequately represented by a model with three intercorrelated factors. These were interpreted as identified, introjected, and external motivation. The reliabilities of the Dutch TEQ, HCCQ, and SMFL were found to be acceptable but can be improved on; congeneric estimates ranged from 0.66 to 0.94 depending on the measure and patient subsample. Preliminary support for the construct validities of the questionnaires was found in the form of theoretically expected associations with other scales, including therapist-rated motivation and treatment engagement and with legally mandated treatment. Additionally, the study provides insights into the relations between measures of motivation based on self-determination theory, the transtheoretical model and the integral model of treatment motivation in psychiatric outpatients with severe mental illness.

Keywords

motivation factor analysis reliability self-determination theory mental disorders psychiatric treatment

Introduction

Self-determination theory (SDT) is a theory on human motivation, which has been applied in various life domains including health-related behavior changes, such as tobacco dependence, diet, and physical activity (Deci, Eghrari, Patrick, & Leone, 1994; Ryan & Deci, 2008). Several studies suggest that SDT could also be useful for understanding why individuals with mental illness do or do not engage in psychiatric treatment (Pelletier, Tuson, & Haddad, 1997; Ryan & Deci, 2008; Zeldman, Ryan, & Fiscella, 2004; Zuroff et al., 2007). The primary aim of this study was to translate three measures based on SDT into Dutch and to investigate their psychometric properties in a population of patients with various primary psychiatric disorders in outpatient treatment. Additionally, we aimed to provide insights into the relations between measures of motivation based on SDT, the transtheoretical model, and the integral model of treatment motivation in psychiatric outpatients with severe mental illness. In the following, we will first describe SDT and its potential utility for application in Dutch outpatient psychiatric treatment, and second, we will describe some of the measurement issues regarding SDT measures. Finally, our hypotheses regarding the relationships between SDT measures and criterion measures will be described.

Applying Self-Determination Theory to Psychiatric Treatment

SDT distinguishes between different types of motivation that fall along a continuum of self-determination in the following order from most to least self-determined: intrinsic, integrated, identified, introjected, external, and amotivation (Ryan & Deci, 2008; Ryan, Patrick, Deci, & Williams, 2008). The most self-determined form of motivation is intrinsic motivation, where people feel that a certain activity or behavior is pleasant in itself. For example, this would be the case for a patient who enters therapy purely for the pleasure of gaining a deeper personal understanding of himself (Pelletier et al., 1997). Psychiatric professionals would like to see that their patients present with such intrinsic willingness to change their problems and participate in treatment processes, as this is thought to lead to positive and lasting results (Ryan & Deci, 2008; Ryan, Lynch, Vansteenkiste, & Deci, 2010). However, not many patients will present with such motivations for treatment, as treatment is usually followed with the intent to find relief of psychiatric symptoms as opposed to it being pleasant in itself or highly valued (Vansteenkiste, Williams, & Resnicow, 2012). Thus, more controlled forms of motivation seem to be applicable. According to SDT, the most self-determined form of controlled motivation is integrated motivation, where a patient identifies with the importance of behaviors but also integrates this into aspects of the self (Deci & Ryan, 2000). An example is a patient who has completed treatment but now wants to reenter a treatment program to help him maintain changes, as he has internalized the value of sustaining mental health. That is, sustaining mental health is consistent (integrated) with his current identity (Pelletier et al., 1997). Less self-integrated is identified motivation, in which a patient recognizes and accepts that treatment is useful for achieving personally relevant goals (Ryan & Deci, 2008). An example would be a patient who engages in treatment because he thinks it is the best way to help him live a healthy life, or a patient who finds it important to take medications as a way of preventing relapse into psychoses. This type of motivation is thus more instrumental to achieving a goal, as opposed to integrated motivation, which is more aligned with the person’s perception of the self. An even less self-integrated form of motivation is introjected motivation, where a patient is driven by feelings of guilt, shame, or anxiety and might feel ashamed or disappointed if he did not remain in treatment. An example would be a man who seeks treatment because he is overwhelmed by feelings of shame, as he feels that he is a bad husband for having repeatedly battered his wife. Engaging in treatment is driven by his introjected motive to improve his relationship and family situation. Finally, the most externally determined form of motivation is when a patient remains in treatment because he feels pressured to do so (Ryan & Deci, 2008). This could be the case for a patient who is court-ordered into treatment or a patient who enters treatment because his wife has pressured him to change his drinking problem or else seek out a divorce attorney (Pelletier et al., 1997). A separate category of motivation is called “amotivation,” where people experience no regulation at all over their behavior and are very likely to drop out or reject treatment (Deci & Ryan, 2008). A patient who is amotived to engage in treatment is characterized by not having a clear understanding of why he does so and has a sense of hopelessness, believing that treatment will undoubtedly lead to failure and disappointment (Pelletier et al., 1997).

According to SDT, engaging in treatment for a long period of time and maintenance of changed behaviors over time requires that patients internalize values and skills for change (Ryan et al., 2008). That is, patients with a more internalized form of motivation will experience greater ownership of the behavior and be more self-determined. SDT predicts that people with more internalized motives for engaging in treatment and engaging in behavior changes will have better mental and psychical health outcomes, compared with those with more externalized motives (Ryan et al., 2008). The rationale is that behaviors that are more self-determined and intrinsically rewarding are most likely to be performed again, whereas behaviors that are primarily driven by external motives will only be performed again in the presence of such external pressures. As Ryan and Deci (2008) put it:

This is so because, to the extent that people experience treatment or change as a function of external factors, they will experience conflict and division in the process of change, rendering it unstable. Unless the client internalizes responsibility for the process of change, there can be little hope for long-term success. (p. 187)

According to SDT, the different types of motivation become manifest due to the (lack of) support for three basic psychological needs: autonomy, competence and relatedness. Autonomy refers to the need to be the originator of one’s actions and the desire for volition and choice. Competence refers to the need to feel capable of achieving desired outcomes, whereas relatedness is the need to feel close to and understood by others (Deci & Ryan, 2000). SDT predicts that patients who feel that their therapeutic environment is supportive of their basic psychological needs will more easily go through the process of internalization and identify with adhering to treatment processes and engaging in behavior changes, leading to better treatment outcomes compared with those patients who feel thwarted in their needs (Ryan & Deci, 2008).

Measurement Issues

Several studies have found that autonomy support is positively related to treatment outcomes in psychotherapy (McBride et al., 2010; Wild, Cunningham, & Ryan, 2006; Zuroff et al., 2007), but the evidence is still rather scarce and no previous studies have reported on the utility of SDT in patients with severe mental illness (Jochems, Mulder, van Dam, & Duivenvoorden, 2011). One of the reasons for this may be that there is a lack of psychometrically adequate instruments that enable the measurement of these types of motivation in mental health care services for patients with severe mental illness. Although a scale for intrinsic motivation was developed for patients with schizophrenia by Choi, Mogami, and Medalia (2010), this scale has several limitations. It measures subjective experiences of interest/enjoyment, effort, value/usefulness, pressure/tension, relatedness, and perceived choice, and it thus seems conceptually ambiguous since it encompasses concepts that are in fact psychological needs (such as relatedness) and behavior resulting from motivation (such as effort). Furthermore, although psychiatric patients may experience intrinsic motivation for many activities in life, it is questionable whether this applies to engaging in psychiatric treatment (Jochems et al., 2011; Vansteenkiste et al., 2012).

The Treatment Entry Questionnaire (TEQ) developed by Wild et al. (2006) seems to represent SDT’s types of motivation more adequately and was developed specifically for treatment settings. It formed internally consistent dimensions for identified motivation, introjected motivation, and external motivation (Wild et al., 2006) and consists of 27 items. Although the TEQ does not measure all six different types of motivation postulated by SDT (which are intrinsic, integrated, identified, introjected, external, and amotivation), its extension to more external forms of motivation might prove valuable and insightful for the treatment of patients with severe mental illness, such as those with psychotic disorders. Thus, the TEQ is in need of further theoretical and empirical investigation. In line with the development of a shortened 12-item version of the English TEQ (Martino, Menchetti, Pozzi, & Berardi, 2012), we developed a short version of the Dutch TEQ that could be easily applied in clinical practice, for example, during treatment sessions as a way of discussing the patient’s current motivation for treatment. This Short Motivation Feedback List (SMFL) was also in need of psychometric testing. To measure the extent to which health care providers are autonomy supportive, the Health Care Climate Questionnaire (HCCQ) was developed (Williams, Grow, Freedman, Ryan, & Deci, 1996), but application of this scale to a Dutch psychiatric outpatient population is not known to us.

Therefore, we were interested in determining the empirical factor structures of the patient-rated SMFL and TEQ and examine the reliabilities and construct validities of the TEQ, HCCQ, and SMFL in a Dutch outpatient sample. Convergent and discriminant validities for the TEQ and SMFL were determined by using clinician ratings of the TEQ and SMFL. The use of observer-rated motivation questionnaires is rare in SDT literature, which is regretful as self-report questionnaires may be influenced by socially desirable response bias and insight into illness, especially for patients with psychiatric problems. According to Campbell and Fiske (1959), convergent validity is determined by correlations between different measures for similar traits, the so-called monotrait–heteromethod correlations, whereas discriminant validity is supported when monotrait–heteromethod correlations exceed the heterotrait–monomethod correlations and the heterotrait–heteromethod correlations. The validities of the TEQ, SMFL, and HCCQ were investigated further by using several criterion measures. For example, the TEQ was investigated in relation to clinician-rated treatment engagement and legally mandated treatment assessed at the same time, as well as to two other measures of motivation to engage in treatment. The first was a measure for motivational concepts within the integral model of treatment motivation (IM; Drieschner, Lammers, & van der Staak, 2004). The IM postulates six proximal predictors of the patient’s motivation to engage in treatment (MET) and is conceptually clear about the distinction between predictors of motivation, treatment motivation itself (having an intention), and treatment engagement (actual behavior; Jochems et al., 2011). The second was based on the transtheoretical model (TTM; McConnaughy, Prochaska, & Velices, 1983; Prochaska & DiClemente, 1983), which asserts that patients move through certain stages in the process of behavior change, and these stages are generally seen as different levels of motivation (Drieschner et al., 2004; Jochems et al., 2011). In a previous theoretical exploration, we have argued that the IM and TTM are mainly concerned with a quantification of the motivation concept (low or high motivation) whereas SDT is concerned with a more differential motivation concept (types or quality of motivation; Jochems et al., 2011).

The HCCQ and SMFL were subjected to similar tests as the TEQ. That is, the SMFL was expected to show the theoretically expected associations with the patient-rated and clinician-rated TEQ scales and substantial associations with the IM and TTM motivation scales, clinician-rated treatment engagement, and with legally mandated treatment. For the HCCQ, the Helping Alliance Questionnaire (HAQ) was used as a criterion measure. The helping alliance refers to the patient’s experience of the relationship with the therapist as helpful (Alexander & Luborsky, 1986) and, more specifically, to perceptions of the patient regarding cooperation (such as working together with the clinician to achieve goals and perception of the influence of the clinician on the healing process) and helpfulness (such as the patient’s confidence in his clinician, the treatment in general and his own capacities).

Hypotheses

We hypothesized that more internalized forms of motivation measured within the SDT framework would be associated with higher levels of motivation measured within the TTM and IM frameworks (Jochems et al., 2011) and more positively associated with clinician-rated treatment engagement (Wild et al., 2006) compared with more externalized forms of motivation. We expected that patients with a legal mandate for psychiatric treatment would show higher levels of external motivation than patients without a legal mandate for treatment. It should not necessarily be so that patients with a legal mandate show lower levels of identified motivation, as these different motivational types do not exclude each other and different types may co-occur at high levels at the same time (Silverstein, 2010). For the HCCQ, we hypothesized that the cooperation subscale of the HAQ would be more strongly associated with autonomy support than the helpfulness subscale, because both cooperation and autonomy support specifically refer to the patient’s perception of the clinician, whereas helpfulness refers to a more broad perception of the usefulness of the treatment in general and the patient’s competence.

Method

Participants and Procedures

We invited 11 function assertive community treatment teams (FACT teams), one outpatient forensic psychiatric team, and one treatment program applying cognitive behavioral therapy to patients with mood and anxiety disorders from the Western North Brabant Mental Health Center and the Breburg Mental Health Center to participate in this study. FACT is a team treatment model where individuals with severe mental illness are offered community-based, assertive, outreaching, and supportive psychiatric services (Drake et al., 2001; van Veldhuizen, 2007). Besides assertive outreach, the emphasis is on out-of-office interventions and home visits (van Veldhuizen, 2007). First, clinicians who provided care to patients with a primary diagnosis of mood or anxiety, psychotic, or personality disorder were approached for participation. Second, participating clinicians were asked to provide a list of their caseload to the primary researcher, to select eligible patients. Patients were eligible for participation if they were aged at least 18 years or older, were in outpatient treatment, and had a primary diagnosis of anxiety, mood, psychotic, or personality disorder. Comorbid psychiatric disorders were allowed as long as the anxiety, mood, psychotic, or personality disorder was the primary diagnosis. These diagnoses were obtained from the patient’s medical records. Exclusion criteria were a poor understanding of the Dutch language, mental retardation, and a documented history of dementia or chronic toxic encephalopathy. Eligible patients were informed and asked for their consent. Patients were asked to fill in questionnaires and provide some sociodemographic data (e.g., age, gender, education level) and information regarding their treatment (e.g., age of first contact with mental health, legal mandate, previous admissions). Clinicians were asked to fill in questionnaires regarding this patient. To study the test–retest reliabilities of the TEQ and HCCQ, 70 patients with psychotic disorders and/or personality disorders were readministered the TEQ and HCCQ 1 year after the first measurement. These 70 patients were in the treatment as usual arm (control condition) of the randomized controlled trial in which the data of the current study were gathered (Jochems et al., 2012). The SMFL was administered at a later time point than the TEQ and HCCQ, as the TEQ and HCCQ were administered during the baseline assessment of the previously mentioned trial (Jochems et al., 2012) whereas the SMFL was administered sometime after this (median = 65 days). Additionally, four repeated measures with the SMFL were available for 52 patients with psychotic disorders and/or personality disorders, with a median of 43 days apart. The current research was approved by an official medical ethical committee and by the committees for scientific research within the two mental health institutions where the data were collected.

Measures

SDT: Treatment Entry Questionnaire

The types of motivation that are distinguished by SDT were measured with the Treatment Entry Questionnaire (TEQ; Ryan, Plant, & O’Malley, 1995; Wild et al., 2006). It was shown that the original English TEQ was reliable for external (Cronbach’s α = .89), introjected (Cronbach’s α = .89), and identified motivation (Cronbach’s α = 0.85) in a study of patients seeking substance abuse treatment (Wild et al., 2006). Construct validity for the English TEQ exists in the form of theoretically expected correlations between TEQ subscales and referral source (i.e., legal mandate or self-referral), social network pressures to seek treatment, and problem severity (Urbanoski & Wild, 2012; Wild et al., 2006). The original TEQ consists of 27 items rated on a scale from 1 (strongly disagree) to 7 (strongly agree), and subscale scores are computed by averaging the item scores. Higher scale scores denote higher levels of that type of motivation.

For the translation of the 27 items of the original TEQ, we first adapted the wording to fit a population of patients with mental illness in psychiatric treatment. For example, where in the original TEQ the focus was on substance abuse treatment and included items such as “I plan to go through with a treatment program because I’ll hate myself if I don’t get my habit under control,” we focused on a more general psychiatric treatment and adapted the item to “I plan to go through with a treatment program because I’ll hate myself if I don’t get my problems under control.” Two translators performed forward translations of the original TEQ independently into Dutch and adapted the wording to fit its application to outpatient psychiatric treatment. A consensus version based on these two translations was established. This consensus version was back-translated by a native English speaker (second language Dutch, also native level) and compared with the original English version. Only minor adaptations were necessary to establish the final Dutch items.

SDT: Short Motivation Feedback List

The Dutch TEQ items served as the basis for item creation and selection in the SMFL. The SMFL contains only eight items (three items intended to measure identified motivation, two items for introjected motivation, and three items for external motivation). The eight items were selected based on the highest factor loadings in the study by Wild et al. (2006) on the original TEQ. The items were shortened and simplified where possible, to aid patients and clinicians in understanding them. All items begin with the phrase: “Currently, I remain in treatment because . . .” followed by a specific ending (e.g., “I can solve my problems this way” for identified motivation, “I may not disappoint myself” for introjected motivation, and “other people think that I should” for external motivation). After these adjustments, none of the SMFL items were identical to TEQ items anymore, although they resembled them. The items are rated on a scale from 0 (totally disagree) to 10 (totally agree). Higher scale scores denote higher levels of that type of motivation. Preliminary impressions of the use of the SMFL among 13 patients with primarily anxiety and depressive symptoms receiving outpatient treatment showed that the list was comprehensible and easy to use in clinical practice. Clinicians appreciated the brevity and clarity of the items, which could function as a starting point for the discussion with the patient regarding his/her current motivation to engage in treatment.

SDT: Health Care Climate Questionnaire

The HCCQ (Williams et al., 1996) was administered to patients to assess the degree in which clinicians were perceived to be autonomy supportive. Autonomy support consists of providing choices to the patient, being open to the perspective of the patient, and minimizing pressure and control (Ryan & Deci, 2008). The HCCQ has 15 items that are scored on a Likert-type scale, ranging from 1 (strongly disagree) to 7 (strongly agree). The items are summed up to obtain a total scale score, with higher scores reflecting higher perceived autonomy support. The HCCQ was found to be reliable in a study on psychotherapy for depressed outpatients (Cronbach’s α = .88; Zuroff et al., 2007). The original HCCQ was translated into Dutch by two independent translators who subsequently established a consensus version. This consensus version was back-translated into English by two independent expert translators to check for discrepancies between the original version and the back-translation. On the basis of consensus between all translators, the final Dutch HCCQ was constructed.

IM: Motivation to Engage in Treatment Scale

The scale for MET that is part of the Treatment Motivation Scale for Forensic Patients (TMS-f; Drieschner & Boomsma, 2008a) is a self-report questionnaire that was administered to patients to measure their motivation for outpatient treatment. Although the TMS-f was developed within Dutch forensic psychiatric outpatient treatment, we felt that the items on the MET subscale were sufficiently general for application outside a forensic setting. The MET subscale measures commitment for treatment engagement, session attendance, treatment completion, and readiness to make sacrifices necessary for treatment (e.g., money, emotional burden, and lifestyle changes; Drieschner & Boomsma, 2008b). The MET subscale has 16 items that are rated on a scale from 1 (totally agree) to 5 (totally disagree). The TMS-f has been found to be reliable and valid measure in a population of Dutch (forensic) outpatients with a variety of psychiatric disorders (Drieschner & Boomsma, 2008a, 2008b).

TTM: University of Rhode Island Change Assessment

The Dutch version of the University of Rhode Island Change Assessment (Jonge, Schaap, & Schippers, 2002) is a self-report scale that asks a patient to rate the agreement with a particular statement reflecting a specific stage of change. It has 24 items rated on a scale from 1 (totally disagree) to 5 (totally agree) representing four stages of change: precontemplation, contemplation, action, and maintenance. A total readiness for change score was calculated by subtracting the precontemplation scale score from the sum of the other three scale scores, as in a previous study in patients with severe mental illness where it was found to be sufficiently reliable and valid (Nidecker, DiClemente, Bennett, & Bellack, 2008). Higher scores reflect more readiness to change.

Helping Alliance Questionnaire

The HAQ contains two subscales that refer to cooperation (5 items) and helpfulness (5 items) that are rated on a 5-point scale (completely disagree to completely agree; De Weert-Van Oene, De Jong, Jorg, & Schrijvers, 1999). Both a patient and a clinician version have been developed. The patient version (HAQ-P) includes items such as “I feel the clinician understands me.” The clinician version (HAQ-C) has similar items but worded differently, such as “I understand the patient.” The HAQ was found to be reliable, and construct validity was supported by positive correlations between the HAQ and other relationship scales and outcomes such as length of stay in detox treatment programs and noncompliance with treatment (De Weert-Van Oene et al., 1999).

Service Engagement Scale

The Service Engagement Scale (SES) was used to assess the patient’s engagement with outpatient psychiatric treatment (Tait, Birchwood, & Trower, 2002). This scale was developed specifically for assessing engagement with services in community mental health settings. It has four subscales that refer to availability, collaboration, help seeking, and treatment engagement (including medication adherence). The 14 items of the SES are rated by clinicians on a 4-point scale ranging from 0 (not at all) to 3 (most of the time). For the current study, the total scale score was used. In case patients were not prescribed medications, these items were replaced with the total mean of the other items to compute the total scale score. A higher score on the SES reflects higher levels of treatment engagement. The SES has shown good psychometric properties in patients with psychotic disorders (Staring et al., 2010; Tait et al., 2002).

Statistical Analyses

The analysis strategy consisted of four steps: (a) exploratory factor analysis (EFA), (b) confirmatory factor analysis (CFA), (c) estimating reliabilities, and (d) determining construct and criterion validity. Although Wild et al. (2006) have reported on the structure of the English TEQ, several items referring to substance or drug abuse problems were rewritten in the Dutch translation to fit a population of outpatients with primarily psychiatric problems and these alterations might influence the structural model. Thus, EFA was first conducted to empirically determine the number of factors that would be appropriate for the Dutch TEQ. The fit of structural models with varying numbers of latent factors (i.e., from one factor up to four factors) was compared using robust maximum-likelihood approaches (MLR) and the chi-square statistic (χ²). Also, the scree plot was inspected and the Kaiser criterion (eigenvalues > 1) was considered. Additionally, theoretical psychological interpretation guided the decision making, informed by the study with the original English TEQ by Wild et al. (2006) in which three latent factors were retained.

Subsequently, CFA was performed on the latent factors that were identified by EFA, to establish whether the factor structure of the Dutch TEQ was similar to the original English TEQ. Although the theoretical constructs defined by SDT are distinct, it is likely that these constructs are related when translated in empirical statistical terms. Therefore, we decided to analyze all constructs simultaneously by determining the relative fits of four different models to the TEQ data. First, an uncorrelated factors model (27 items loading on three uncorrelated factors) and an intercorrelated factors model (27 items loading on three intercorrelated factors) were tested to see if three factors underlie the measured items and if these should be correlated. Third, a bifactor model (27 items simultaneously loading on three uncorrelated factors and one general factor) was specified to test if a model with an independent general motivation factor and three additional more specific motivation factors was appropriate. Although it is unlikely with EFA to obtain a bifactor model (since the aim of EFA is to obtain simple structure and the number of factors), bifactor models can be appropriate for psychological scales (Babyak & Green, 2010). Finally, a second-order factor model (27 items loading on three factors that loaded on a single higher order factor) was specified to test if the three motivation factors had a common underlying (higher order) construct (i.e., motivation in general). It is of great importance to identify which of these different factor models is most plausible, both clinically/theoretically and empirically/statistically. It is an important preliminary step before conducting larger structural models that also include relationships between predictors/determinants and outcome variables. For example, CFA can help determine if there are specific motivation factors that could be used as unique predictors of treatment outcomes (in case we find most support for the uncorrelated or intercorrelated factor model), or if there are both specific factors and an independent general motivation factor, which may be hierarchically ordered (in case we find most support for the second-order factor model) or not (in case of most support for the bifactor model).

The quality of the models was evaluated in two steps. First, the analyses were rerun if the models were improperly specified. Individual fit was assessed by evaluating individual estimated parameters (e.g., no negative variances or correlations of ≥|1.00| and no parameter estimates with values that contradicted psychological theoretical interpretation were allowed). Second, overall fit was assessed by using the chi-square statistic (χ²), χ²/df, Bentler’s comparative fit index (CFI), Tucker–Lewis index (TLI), and the root mean square error of approximation (RMSEA). The following rules of thumb were used to specify cutoff points for fit indices (Hu & Bentler, 1999): CFI/TLI > 0.95 (good fit), 0.90 to 0.95 (borderline fit), and <0.90 (poor fit); RMSEA <0.06 (good fit), 0.06 to 0.08 (fair fit), 0.08 to 0.10 (borderline fit), and >0.10 (poor fit). Also, the fit indices of Akaike information criterion (AIC), Bayesian information criterion (BIC), and sample-size adjusted Bayesian information criterion (S-BIC) were inspected. Furthermore, in search for the most parsimonious assessment with the TEQ, the TEQ items with a substantial loading on a certain factor (i.e., ≥|0.40|) and low loadings on the other factors (i.e., ≤|0.20|) were considered as “factor-pure” items. We performed secondary (and similar) CFA on only these factor-pure items to determine the final structural model of the Dutch TEQ. For all models, robust MLR as implemented in Mplus (Muthén & Muthén, 1998-2012) was used because of nonnormality of the frequency distribution of the items. The factor structure of the SMFL was established with the same procedure, by first conducting EFA to determine the number of factors. The relative fits of models with one, two, or three latent factors were evaluated. Subsequently, we fitted the following models in CFA: a general factor model, an uncorrelated factors model, an intercorrelated factors model, a bifactor model, and a second order factor model.

The reliabilities of the TEQ, SMFL and HCCQ were determined by the computation of Cronbach’s α and either tau-equivalence or congeneric estimates of reliability. The reliability of the composite of items is a function of the square of the sum of the factor loadings on one factor, as this decomposes the observed variance into a systematic component and an error component. Although Cronbach’s α is the most commonly used measure of internal consistency, it only yields an unbiased estimate of the reliability if the factor loadings and error variances of the items are equal (Reuterberg & Gustafsson, 1992). If however, only factor loadings are equal but error variances are not, a tau-equivalence estimate for reliability is more appropriate, and if both factor loadings and error variances are heterogeneous, the congeneric reliability estimate is most appropriate (Reuterberg & Gustafsson, 1992). Therefore, additional to reporting traditional Cronbach’s αs, either tau-equivalent or congeneric estimates of reliability will be reported where appropriate. Also, test–retest correlations were computed for the TEQ. For the interpretation of the reliability coefficients, we used the guidelines of Nunnally and Bernstein (1994), who stated that a value of 0.70 is sufficient for early stages of research but a reliability coefficient of 0.80 or higher should be attempted, whereas reliability coefficients of 0.90 or higher are desirable when important decisions are to be made with the test scores. Test–retest reliabilities for the scales were explored by computing Pearson correlations (in case of normally distributed data) or Spearman rank correlations (for nonnormally distributed data) between the first and second assessments. For the SMFL, intraclass correlation coefficients (ICCs) were computed to estimate the test–retest stability over four consecutive SMFL assessments, using a two-way random model of consistency.

Validity was investigated by computing Pearson correlations (in case of linear normally distributed data) or Spearman rank correlations (for nonnormally distributed data) between all patient-rated and clinician-rated scales. The conventional guidelines by Cohen (1988) were used for the interpretation of the correlation coefficients: r ≤ |0.29| as weak, |0.30 to 0.49| as moderate, and r ≥ |0.49| as large.

Furthermore, the mean scores on the TEQ subscales were compared between patients who were rated by their clinicians as having high levels of treatment engagement (measured by the SES) to those with low levels, and between patients with and without legally mandated treatment. Since the SES scores showed a skewed distribution, the SES was dichotomized at the median score. SES ≤30 was interpreted as low treatment engagement and SES >30 as high treatment engagement. Logistic regression analyses were performed using the TEQ scales as predictors for the dichotomized SES and legally mandated treatment (yes/no). Statistical tests were performed using SPSS version 21 for Windows (SPSS Inc, Chicago, IL) and Mplus version 7.0 (Muthén & Muthén, 1998-2012).

Results

A total of 72 clinicians agreed to participate of which 49 were female (68%). Their mean age was 43 years (SD = 10.81), and they had a mean of 15.4 years of clinical working experience in mental health services (SD = 9.32). From their caseloads, a total of 349 patients agreed to participate. Table 1 shows an overview of the patient characteristics. Within the subsample of patients with psychotic disorders, the majority of patients were diagnosed with schizophrenia (48%), schizoaffective disorder (16%), or psychotic disorder not otherwise specified (24%). Within the subsample of personality disorders, 40% had a borderline personality disorder, 13% had antisocial personality disorder, and 26% had a personality disorder not otherwise specified. Within the subsample of patients with mood and anxiety disorders, 46% had an anxiety disorder.

Table 1.

Patient Characteristics for the Total Sample and Diagnostic Groups.

	Total patient sample (N = 349)	Psychotic disorders (n = 199)	Personality disorders (n = 95)	Mood and anxiety disorders (n = 55)
Age, mean (SD)	43.3 (10.6)	43.0 (10.2)	46.0 (10.2)*	39.8 (11.5)*
Male gender, n (%)	200 (57.5)	133 (66.2)*	46 (50.0)*	21 (38.5)*,^a
Education level, n (%)
No education/elementary	129 (37.1)	76 (37.8)	31 (33.7)	22 (40.0)
Secondary school	144 (41.4)	77 (38.3)	47 (51.1)	20 (36.4)
Upper high school and over	67 (19.3)	47 (23.4)	12 (13.0)	8 (14.5)
Unknown	8 (2.3)	1 (0.5)	2 (2.2)	5 (9.1)
Country of birth, n (%)
Netherlands	286 (82.2)	158 (78.6)	81 (88.0)	47 (85.5)
Morocco	15 (4.3)	10 (5.0)	1 (1.1)	4 (7.3)
Netherlands Antilles	6 (1.7)	5 (2.5)	1 (1.1)	0 (0)
Other	39 (11.2)	28 (13.9)	9 (9.8)	2 (3.6)
Unknown	2 (0.6)	0 (0)	0 (0)	2 (3.6)
Treatment duration (months)
Mean (SD)	34.3 (25.5)	37.3 (24.2)*	42.1 (25.9)*	8.3 (8.47)*,^b
Median (IQR)	29 (17-48)	30 (20.5-52)	37.5 (26.3-53)	5.5 (3-12)
GAF score, mean (SD)	57.6 (8.3)	56.7 (9.0)	58.3 (7.5)	59.7 (6.2)
Legal mandate, n (% yes)	24 (6.9)	13 (6.5)	11 (12.0)*	0 (0)*
One or more previous admissions, n (% yes)	234 (67.2)	162 (80.6)*	65 (70.7)*	7 (12.7)*,^b

Note. Percentages reflect column percentages.

All three diagnostic groups significantly differed from each other.

Patients with mood and anxiety disorders significantly differed from the other two diagnostic groups.

Statistically significant values are shown in boldface: *p < .017 (two-tailed, the conventional p value of α = .05 was divided by the number of groups [3] in the comparison).

Exploratory Factor Analyses

EFA analyses on the 27 items of the TEQ revealed superior fit for a structural model with three latent factors (χ² = 534.64, df = 273, p = .0000, χ²/df = 1.96, CFI = 0.90; TLI = 0.87; RMSEA = 0.05), as opposed to a model with one latent factor (χ² = 1762.74, df = 324, p = .0000, χ²/df = 5.44, CFI = 0.46; TLI = 0.42; RMSEA = 0.11) or two latent factors (χ² = 693.50, df = 298, p = .0000, χ²/df = 2.33, CFI = 0.85; TLI = 0.83; RMSEA = 0.06). A three-factor model was most consistent with the original English version of the TEQ (Wild et al., 2006), and these three factors could theoretically be interpreted as representing identified, introjected, and external motivation.

The SMFL was subjected to EFA, in which a superior fit for a model with three latent factors was also found (χ² = 7.16, df = 7, p = .4127, χ²/df = 1.02, CFI = 0.99; TLI = 0.99; RMSEA = 0.01), as opposed to two latent factors (χ² = 35.91, df = 13, p = .0006, χ²/df = 2.76, CFI = 0.92; TLI = 0.83; RMSEA = 0.11) or one latent factor (χ² = 113.14, df = 20, p = .0000, χ²/df = 5.66, CFI = 0.68; TLI = 0.55; RMSEA = 0.19). Theoretically, SMFL Items 1 and 2 were interpreted as identified motivation; Items 3, 5, and 7 were interpreted as introjected motivation; and Items 6 and 8 as external motivation. Item 4 was discarded as it did not load substantially on any of the retained factors and could not be interpreted theoretically as belonging to any of these three factors. In the following analyses, these three SMFL subscales were used.

Confirmatory Factor Analyses

CFA analysis of the 27 items of the TEQ (see Table 2) revealed superior fit for the bifactor model (χ² = 613.30, df = 299, p = .0000, χ²/df = 2.05; CFI = 0.85; TLI = 0.82; RMSEA = 0.06) compared with the other models. CFA did not converge for the second-order factor model, even after a loosening of constraints on the model and enlarging the number of iterations. Second, only the TEQ items with a substantial loading on a relevant factor (i.e., ≥|.40|) and low loadings on the other factors (i.e., ≤|.20|) were brought into a secondary (and similar) CFA (see Table 2). A total of 18 items were selected, 6 items per factor. Again, the bifactor model showed superior fit (χ² = 249.65, df = 118, p = .0000, χ²/df = 2.12; CFI = 0.90; TLI = 0.87; RMSEA = 0.06) compared with the other models. The information criteria (AIC, BIC, S-BIC) were also lowest for the bifactor model, which supports that this model was most plausible. However, subsequent reliability analysis on the bifactor model revealed that several subscales had inadequate reliability estimates for both the total sample and subsamples (e.g., congeneric estimates of 0.10 and 0.18). Since our goal was to establish reliable sum scores for the subscales of the TEQ, we chose to continue with the next best fitting model, which was the intercorrelated factors model. This model had borderline acceptable fit (χ² = 315.18, df = 135, p = .0000, χ²/df = 2.33; CFI = 0.86; TLI = 0.84; RMSEA = 0.06) but was theoretically more plausible than the bifactor model solution and reliability estimates for all patient samples were acceptable (see Table 3). The item cross-loadings were inspected for all confirmative analyses and no indication was found for any cross-loading. Standardized coefficients (factors loadings) between the observed measures (18 TEQ items) and latent variables (motivation factors) for the intercorrelated factors model are shown in Figure 1. The three factors in the final model were interpreted as identified motivation, introjected motivation, and external motivation. The average variance explained (R²) by the final model was 35% (R² for the items ranging from 11% to 56%). In all subsequent analyses, the TEQ 18-item-version was used.

Table 2.

Confirmatory Factor Analyses for the TEQ and SMFL.

Questionnaire	Model	χ²	df	p Value	χ²/df	CFI	TLI	RMSEA	90% CI for RMSEA	AIC	BIC	S-BIC
TEQ (27 items)	Uncorrelated factors	984.96	325	.0000	3.03	0.68	0.66	0.08	0.071 to 0.082	37810.51	38118.46	37864.67
	Intercorrelated factors	830.65	322	.0000	2.58	0.76	0.74	0.07	0.062 to 0.073	37638.04	37957.54	37694.23
	Bifactor	613.30	299	.0000	2.05	0.85	0.82	0.06	0.049 to 0.061	37409.43	37817.46	37481.20
	Second-order factor	d.n.c.
TEQ (18 items)	Uncorrelated factors	423.86	138	.0000	3.07	0.78	0.75	0.08	0.069 to 0.086	24898.85	25095.16	24933.37
	Intercorrelated factors	315.18	135	.0000	2.33	0.86	0.84	0.06	0.053 to 0.071	24779.37	24987.23	24851.93
	Bifactor	249.65	118	.0000	2.12	0.90	0.87	0.06	0.047 to 0.066	24724.88	24998.18	24772.95
	Second-order factor	360.59	135	.0000	2.67	0.83	0.80	0.07	0.061 to 0.078	24828.60	25036.46	24865.16
SMFL (7 items)	Uncorrelated factors	d.n.c.
	Intercorrelated factors	24.55	9	.0035	2.72	0.92	0.82	0.11	0.060 to 0.167	4464.64	4540.37	4458.12
	Bifactor	23.42	7	.0014	3.35	0.92	0.75	0.13	0.075 to 0.192	4468.43	4549.98	4461.40
	Second-order factor	d.n.c.

Note. TEQ = Treatment Entry Questionnaire; SMFL = Short Motivation Feedback List; χ² = chi-square statistic; df = degrees of freedom; CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; AIC = Akaike information criterion; BIC = Bayesian information criterion; S-BIC = sample-size adjusted bayesian information criterion; d.n.c. = did not converge; CI = confidence interval.

Table 3.

Descriptive Statistics and Reliabilities for the SDT Measures in the Total Client Sample and Diagnostic Groups.

Questionnaires	Total patient sample (N= 348)	Psychotic disorders (n = 201)	Personality disorders (n = 92)	Mood and anxiety disorders (n = 55)
TEQ identified motivation subscale
Mean (SD)	34.5 (7.13)	33.53 (7.56)	34.6 (6.77)	37.9 (4.71)
Internal consistency
Cronbach’s α	0.78	0.78	0.76	0.74
Congeneric estimate	0.78	0.75	0.82	0.86
Test–retest reliability (1 year)	0.58** (n = 70)	0.60** (n = 48)	0.53* (n = 22)
TEQ introjected motivation subscale
Mean (SD)	22.2 (9.24)	22.3 (9.37)	23.1 (9.33)	20.3 (8.48)
Internal consistency
Cronbach’s α	0.77	0.76	0.78	0.78
Congeneric estimate	0.72	0.71	0.73	0.76
Test–retest reliability (1 year)	0.60** (n = 70)	0.58** (n = 48)	0.62** (n = 22)
TEQ external motivation subscale
Mean (SD)	15.85 (8.41)	18.4 (8.39)	14.5 (7.73)	8.7 (3.57)
Internal consistency
Cronbach’s α	0.74	0.71	0.69	0.58
Congeneric estimate	0.75	0.74	0.69	0.74
Test–retest reliability (1 year)	0.45** (n = 70)	0.48** (n = 48)	0.30 (n = 22)
SMFL identified motivation subscale
Mean (SD)	13.54 (4.02)	13.24 (3.86)	13.19 (4.24)	14.00 (4.07)
Internal consistency
Cronbach’s α	0.76	0.85	0.72	0.71
Congeneric estimate	0.81	0.89	0.85	0.85
SMFL introjected motivation subscale
Mean (SD)	14.01 (7.14)	14.20 (7.75)	11.38 (6.65)	15.33 (6.53)
Internal consistency
Cronbach’s α	0.63	0.67	0.63	0.54
Congeneric estimate	0.93	0.90	0.92	0.91
SMFL external motivation subscale
Mean (SD)	6.89 (5.18)	8.28 (5.93)	6.16 (4.46)	6.04 (4.63)
Internal consistency
Cronbach’s α	0.70	0.84	0.56	0.54
Congeneric estimate	0.84	0.88	0.74	0.66
HCCQ total scale score
Mean (SD)	88.56 (15.06)	86.02 (16.17)	89.73 (14.01)	95.85 (8.82)
Internal consistency
Cronbach’s α	0.93	0.93	0.92	0.91
Congeneric estimate	0.93	0.94	0.93	0.91
Test–retest reliability (1 year)	0.58** (n = 131)	0.58** (n = 84)	0.56** (n = 7)

Note. SDT = self-determination theory; TEQ = Treatment Entry Questionnaire; SMFL = Short Motivation Feedback List; HCCQ = Health Care Climate Questionnaire.

p < .05. **p < .01 (two-tailed).

Figure 1.

Standardized coefficients (factor loadings) obtained with the intercorrelated factors model for the Treatment Entry Questionnaire.

Regarding the structure for the SMFL, CFA on the seven items revealed that the intercorrelated factors model had relatively better fit compared with the other models.

Reliability

Internal consistencies of the three questionnaires for the total patient sample are summarized in Table 3. As the assumptions for tau-equivalence estimates of reliabilities were never met, these were not reported. We found acceptable reliabilities for the TEQ subscales of identified motivation (congeneric estimate = 0.78), introjected motivation (congeneric estimate = 0.72), and external motivation (congeneric estimate = 0.75) in the total sample. Reliabilities did not vary substantially between subgroups, except for the external motivation subscale that was lower for the patients with personality disorders (congeneric estimate = 0.69). Test–retest reliability of the TEQ was determined by correlating the first measurement with the measurement after 12 months of community mental health treatment in a sample of patients with psychotic and personality disorders (N = 70). The test–retest correlations for the subscales were acceptable: r = 0.58 (p < .01) for internal motivation, r = .60 (p < .01) for introjected motivation, and r = .45 (p < .01) for external motivation.

The SMFL had adequate internal consistency in the total sample for identified motivation (congeneric estimate = 0.81), introjected motivation (congeneric estimate = 0.93), and external motivation (congeneric estimate = 0.84) at the first administration. The reliability for external motivation varied between subgroups; it was questionable for patients with mood and anxiety disorders. For 52 patients with personality and/or psychotic disorders, the SMFL was administered four times with a median of 43 days apart. The ICCs for identified, introjected, and external motivation were 0.69, 0.89, and 0.91, respectively. Table 4 shows that the Spearman correlations increased with the number of times the SMFL was administered. Furthermore, the introjected and external motivation scales consistently showed strong intercorrelations over time, whereas identified motivation only showed moderate correlations to introjected motivation when it was assessed at the same time.

Table 4.

Test–Retest Spearman Correlations (Below Diagonal) and Corresponding 95% Confidence Intervals (Above Diagonal) of Four Assessments With the Patient-Rated SMFL Scales.

	IDEN 1	IDEN 2	IDEN 3	IDEN 4	INTRO 1	INTRO 2	INTRO 3	INTRO 4	EXT 1	EXT 2	EXT 3	EXT 4
IDEN 1	—	0.06 to 0.44	0.05 to 0.52	−0.03 to 0.56	0.20 to 0.53	−0.17 to 0.29	−0.22 to 0.25	−0.26 to 0.31	−0.03 to 0.35	−0.17 to 0.27	−0.26 to 0.20	−0.25 to 0.33
IDEN 2	0.26*	—	0.14 to 0.62	0.07 to 0.64	0.07 to 0.48	0.18 to 0.57	−0.19 to 0.34	−0.32 to 0.35	−0.22 to 0.24	0.01 to 0.43	−0.09 to 0.43	−0.44 to 0.16
IDEN 3	0.31*	0.40*	—	0.22 to 0.68	0.28 to 0.64	−0.03 to 0.45	0.05 to 0.52	−0.03 to 0.41	−0.09 to 0.41	−0.22 to 0.28	−0.07 to 0.40	−0.24 to 0.28
IDEN 4	0.30*	0.37**	0.48**	—	0.03 to 0.60	−0.21 to 0.43	−0.20 to 0.15	0.03 to 0.62	−0.29 to 0.23	−0.46 to 0.15	−0.31 to 0.28	−0.17 to 0.40
INTRO 1	0.37*	0.28**	0.48**	0.32*	—	0.33 to 0.69	0.31 to 0.74	0.36 to 0.81	0.49 to 0.74	0.23 to 0.63	0.36 to 0.74	0.35 to 0.76
INTRO 2	0.06	0.39**	0.21	0.09	0.53**	—	0.46 to 0.79	0.49 to 0.87	0.26 to 0.63	0.51 to 0.79	0.63 to 0.86	0.37 to 0.81
INTRO 3	0.02	0.08	0.30*	0.08	0.54**	0.65**	—	0.45 to 0.87	0.26 to 0.66	0.34 to 0.73	0.54 to 0.84	0.43 to 0.82
INTRO 4	0.03	0.03	0.21	0.33*	0.63**	0.72**	0.69**	—	0.09 to 0.61	0.22 to 0.74	0.48 to 0.84	0.64 to 0.90
EXT 1	0.17	−0.00	0.18	−0.03	0.63**	0.45**	0.48**	0.37**	—	0.38 to 0.71	0.42 to 0.79	0.38 to 0.78
EXT 2	0.04	0.22*	0.03	−0.17	0.45**	0.67**	0.56**	0.49**	0.56**	—	0.54 to 0.83	0.44 to 0.82
EXT 3	−0.04	0.16	0.17	−0.02	0.57**	0.77**	0.71**	0.70**	0.63**	0.71**	—	0.69 to 0.93
EXT 4	0.04	−0.13	0.02	0.12	0.60**	0.62**	0.66**	0.80**	0.61**	0.67**	0.84**	—

Note. SMFL = Short Motivation Feedback List; IDEN = identified motivation; INTRO = introjected motivation; EXT = external motivation.

Statistically significant values are shown in boldface: *p < .05; **p < .01 (two-tailed).

The HCCQ showed excellent internal consistency for the total sample (congeneric estimate = 0.93), and also for all subsamples. The test–retest correlation for the HCCQ after 1 year was r = 0.58 (p < .01).

Validity

Table 5 shows Spearman correlations that were computed between the subscales of the TEQ, HCCQ, SMFL, and the MET, RTC, HAQ, and SES for the total sample. To emphasize that the MET and RTC scales were based on theoretical perspectives different from SDT, we will use their theory abbreviations in front of the scale names from this point onward. That is, IM-MET for the motivation scale based on the integral model of treatment motivation and TTM-RTC for the motivation scale based on the transtheoretical model.

Table 5.

Spearman Correlations Between SDT Measures and Criterion Measures Rated by Patients and Clinicians.

	Patient-rated scales											Clinician-rated scales

	SDT							IM	TTM				SDT

		TEQ IDEN	TEQ INTRO	TEQ EXT	HCCQ	SMFL IDEN	SMFL INTRO	SMFL EXT	MET	RTC	HAQ-C	HAQ-H	TEQ IDEN	TEQ INTRO	TEQ EXT	SMFL IDEN	SMFL INTRO	SMFL EXT	HAQ-C	HAQ-H	SES
Patient-rated scales	TEQ IDEN	—
	TEQ INTRO	0.35**	—
	TEQ EXT	−0.34**	0.16**	—
	HCCQ	0.49**	0.13*	−0.25**	—
	SMFL IDEN	0.24**	0.05	−0.15	0.22**	—
	SMFL INTRO	0.22**	0.46**	0.05	0.04	0.34**	—
	SMFL EXT	−0.05	0.36**	0.33**	−0.11	0.09	0.58**	—
	MET	0.18**	0.04	−0.21**	0.34**	0.16	−0.04	−0.09	—
	RTC	0.50**	0.25**	−0.30**	0.33**	0.31**	0.18	−0.01	0.13*	—
	HAQ-C	0.47**	0.21**	−0.12*	0.67**	0.25**	0.18	0.15	0.26**	0.34**	—
	HAQ-H	0.25**	0.03	−0.03	0.43**	0.25**	0.24*	0.13	0.19**	0.13*	0.59**	—
Clinician-rated scales	TEQ IDEN	0.29**	0.13*	−0.10	0.30**	0.07	−0.10	−0.08	0.17**	0.21**	0.29**	0.18**	—
	TEQ INTRO	0.16**	0.15*	−0.04	0.17**	0.22*	0.01	0.03	0.08	0.14*	0.15*	0.05	0.43**	—
	TEQ EXT	−0.20**	0.03	0.30**	−0.05	0.03	0.16	0.09	−0.09	−0.24**	−0.00	0.14*	−0.30**	0.12	—
	SMFL IDEN	0.15	−0.06	−0.21**	0.22**	0.28**	0.18*	−0.06	0.16	0.16	0.10	0.02	0.27**	0.33**	0.05	—
	SMFL INTRO	0.30**	0.07	−0.11	0.20*	0.24**	0.26**	0.11	−0.01	0.23*	0.15	0.13	0.30**	0.29**	0.09	0.56**	—
	SMFL EXT	0.15	0.17*	0.07	0.07	0.17*	0.33**	0.28**	−0.10	0.09	0.12	0.22*	0.12	0.03	0.25**	0.20**	0.55**	—
	HAQ-C	0.14*	0.01	0.02	0.24**	0.13	−0.10	−0.10	0.16**	0.05	0.24**	0.16**	0.47**	0.16**	−0.11	0.27**	0.26**	0.02	—
	HAQ-H	0.19**	0.04	0.03	0.17**	0.04	−0.10	−0.21*	0.18**	0.06	0.22**	0.25**	0.50**	0.23**	−0.08	0.21*	0.19*	0.01	0.61**	—
	SES	0.25**	0.04	−0.05	0.21**	0.13	−0.11	−0.16	0.28**	0.09	0.29**	0.30**	0.58**	0.18**	−0.15*	0.25*	0.16	−0.04	0.55**	0.54**	—

Note. Gray cells represent monomethod correlations. Black cells represent heteromethod correlations. SDT = self-determination theory; IM = integral model of treatment motivation; TTM = transtheoretical model; TEQ = Treatment Entry Questionnaire; HCCQ = Health Care Climate Questionnaire; SMFL = Short Motivation Feedback List; MET = Motivation to Engage in Treatment Scale; RTC = Readiness to Change; HAQ-C = Helping Alliance Questionnaire Cooperation; HAQ-H = Helping Alliance Questionnaire Helpfulness; SES = Service Engagement Scale.

Statistically significant values are shown in boldface: *p < .05; **p < .01 (two-tailed).

Concurrent and Discriminant Validity

Table 5 shows the patterns of monomethod correlations (gray cells) and the heteromethod correlations (black cells) for the TEQ and SMFL. The intercorrelations of the TEQ subscales showed similar patterns for both methods. That is, identified and external motivation were moderately negatively correlated and introjected motivation showed stronger positive correlation with identified motivation than with external motivation. For the SMFL, the general patterns for both methods were also comparable, although the correlations were stronger for clinician ratings than for patient ratings. Contrary to theoretical expectations, however, identified and external motivations were not negatively correlated but marginally positive. The convergent validities for the TEQ and SMFL are represented by the correlations on the diagonal of the black boxes within Table 5. These monotrait–heteromethod correlations generally showed moderate correlations (between r = 0.26 and r = 0.30), although the TEQ introjected scale showed weak correlation (r = 0.15). Discriminant validities for the TEQ and SMFL were supported by the finding that generally, higher correlations were found between similar motivation concepts than between different motivation concepts. That is, all monotrait–heteromethod correlations exceeded the heterotrait–heteromethod correlations except for the external motivation scale of the SMFL rated by the clinician, which showed higher correlation with patient-rated introjected than external motivation.

Criterion Validity for the TEQ

Regarding associations between the TEQ and measures for the quality of the therapeutic relationship, most theoretically expected associations were confirmed. It was found that identified motivation showed substantial positive correlation with autonomy support measured by the HCCQ and with both subscales of the patient-rated HAQ. Correlations were stronger for patient-rated TEQ than clinician-rated TEQ. Introjected motivation rated by patients showed weak positive correlation with HCCQ (p = .13, p < .05) and the external motivation subscale showed moderate negative association with the HCCQ (r_s = −0.24, p < .01). The correlations of clinician-rated TEQ with the therapeutic relationship scales showed a similar pattern but were less pronounced.

The TEQ scores were also correlated with motivation measures based on the two other theories. Regarding associations with the IM-MET-scale, TEQ identified motivation showed weak positive correlation and introjected motivation showed no association for both methods. External motivation showed weak negative association with IM-MET (r_s = −0.21, p < .01) when rated by patients, but no association between clinician-rated external motivation and IM-MET was found. Furthermore, for both TEQ methods it was found that identified motivation showed strongest positive correlation with TTM-RTC, introjected motivation showed less strong positive association and external motivation showed moderate negative association.

In line with theoretical expectations, treatment engagement as measured with the SES was positively correlated with clinician-rated TEQ identified motivation (r_s = 0.58, p < .01), less strong with introjected motivation (r_s = 0.18, p < .01), and negatively with external motivation (r_s = −0.15, p < .05). For patient-rated TEQ scales, however, neither introjected motivation nor external motivation was associated with SES. The means of the TEQ scales were compared between those who were rated by their clinicians as having high levels of treatment engagement (i.e., SES score > 30) to those who were rated as having low levels (i.e., SES score ≤ 30). Table 6 shows that patients who rated themselves higher on identified motivation were rated by their clinicians as having high levels of treatment engagement at the same time (mean difference −2.61, p < .01).

Table 6.

Relationship Between TEQ Subscales and Clinician-Rated Treatment Engagement.

	Treatment engagement (SES)
TEQ subscale, mean (SD)	Low (SES ≤ 30), n = 130	High (SES > 30), n = 113	Mean difference (95% CI)	t	df	p Value
Patient-rated TEQ
Identified motivation	32.89 (7.65)	35.50 (6.30)	−2.61 (−4.40 to −0.82)	−2.92	240	.004
Introjected motivation	23.25 (9.13)	23.31 (9.70)	1.20 (−2.45 to 2.30)	−0.06	232	.952
External motivation	18.15 (8.76)	17.48 (8.21)	0.67 (−1.49 to 2.82)	0.61	240	.540
Clinician-rated TEQ
Identified motivation	26.67 (7.01)	32.97 (4.76)	−6.29 (−7.70 to −4.87)	−8.77	259	.000
Introjected motivation	20.36 (7.28)	22.09 (6.58)	−1.72 (−3.39 to −0.05)	−2.03	270	.043
External motivation	21.34 (7.93)	19.09 (7.47)	2.24 (0.39 to 4.09)	2.39	270	.018

Note. TEQ = Treatment Entry Questionnaire; CI = confidence interval; SES = Service Engagement Scale. Statistically significant values are shown in boldface (p < 0.05).

Table 7 shows results from independent samples t tests that were performed to investigate the relationship between means on the TEQ subscales and legal mandate for treatment. Patients who had a legal mandate for psychiatric treatment scored significantly lower on identified motivation and significantly higher on external motivation, both for clinician-rated and patient-rated TEQ scales.

Table 7.

Relationship Between TEQ Subscales and Legal Mandate for Treatment.

	Legal mandate
TEQ subscale, mean (SD)	Yes (n = 24)	No (n = 267)	Mean difference (95% CI)	t	df	p Value
Patient-rated TEQ
Identified motivation	29.83 (8.83)	34.24 (7.08)	4.41 (1.37 to 7.44)	2.86	289	.005
Introjected motivation	19.83 (8.85)	22.83 (9.37)	2.99 (−0.91 to 6.91)	1.51	289	.133
External motivation	20.45 (7.22)	16.89 (8.42)	−3.57 (−7.06 to −0.07)	−2.01	289	.045
Clinician-rated TEQ
Identified motivation	23.68 (9.71)	30.09 (6.30)	6.41 (2.04 to 10.77)	3.04	23	.006
Introjected motivation	19.54 (9.98)	21.31 (6.83)	1.76 (−2.71 to 6.26)	0.81	23	.425
External motivation	26.95 (5.87)	19.70 (7.64)	−7.25 (−10.00 to −4.51)	−5.42	27	.000

Note. TEQ = Treatment Entry Questionnaire; CI = confidence interval. Statistically significant values are shown in boldface (p < 0.05).

Criterion Validity for the SMFL

Not all of the theoretically expected associations between the SMFL and the TEQ subscales were confirmed. For example, looking at patient-rated scales it can be seen that although SMFL identified motivation correlated most positive with TEQ identified motivation (r_s = 0.24, p < .01) and the SMFL introjected scale showed strongest positive association with TEQ introjected motivation (r_s = 0.46, p < .01), these correlational patterns were not found for clinician-rated scales. Other mixed findings include that patient-rated SMFL scales showed almost no significant associations with clinician-rated TEQ scales, although they were positively associated with most clinician-rated SMFL scales. For both methods it was found that the SMFL external scale was most strongly positively associated with TEQ introjected motivation instead of with TEQ external motivation. Clinician-rated SMFL scales showed an unexpected pattern of correlations with patient-rated TEQ scales as clinician-rated SMFL introjected motivation was positively correlated with patient-rated TEQ identified motivation (r_s = 0.30, p < .01) and clinician-rated SMFL external motivation was weakly positively correlated with patient-rated introjected motivation (r_s = 0.17, p < .01).

Looking at associations between the SMFL and measures for the therapeutic relationship, the findings were generally consistent with expectations. For the HCCQ, both patient-rated and clinician-rated SMFL identified motivation showed significantly positive correlations and, additionally, so did clinician-rated introjected motivation. Three positive correlations were found of patient-rated SMFL with patient-rated HAQ, but no positive correlations were found with clinician-rated HAQ. Vice versa, clinician-rated SMFL scales showed several significant positive correlations with clinician-rated HAQ but only one with patient-rated HAQ.

Finally, regarding associations between the SMFL and motivation scales from the other two theories, it was found that patient-rated identified motivation was positively associated with TTM-RTC (r_s = 0.31, p < .01) and so was clinician-rated introjected motivation (r_s = 0.23, p < .05). However, neither patient-rated nor clinician-rated SMFL scales were associated with IM-MET.

Criterion Validity for the HCCQ

The HCCQ showed exceptionally strong positive association with the patient-rated HAQ cooperation (r_s = 0.71, p < .01) and strong correlation with patient-rated HAQ helpfulness (r_s = 0.47, p < .01). The HCCQ showed weaker associations with the clinician-rated HAQ: r_s = 0.24 (p < .01) for cooperation and r_s = 0.18 (p < .01) for helpfulness, which suggests that the correlations between patient-rated scales are likely inflated due to common method variance. Consistent with theoretical expectations, the HCCQ showed somewhat stronger association with patient-rated TEQ identified motivation (r_s = 0.49, p < .01) than with the IM-MET scale (r_s = 0.34, p < .01) and the TTM-RTC score (r_s = 0.33, p < .01). Finally, the HCCQ showed a positive association with the SES (r_s = 0.23, p < .01), which was less strong than the associations between the different HAQ scales and the SES.

Discussion

The current study investigated the psychometric properties of the TEQ, HCCQ, and SMFL in a Dutch sample of psychiatric outpatients and generally found support for the use of these instruments for further research applications.

Factor Structures

The factor structure for the TEQ was best represented by an intercorrelated factors model with three factors, which were interpreted as identified, introjected, and external motivation. This structure was consistent with the original English TEQ (Wild et al., 2006). In search for the most parsimonious assessment with the TEQ, it was found that a scale with 18 items (as opposed to the original 27 items), represented by 6 items per subscale, showed best fit. All 18 items loaded onto the same factors as the original English TEQ, except the item “I remain in treatment because I want others to see that I am really trying to deal with my problems.” In our solution, this item was incorporated into the introjected motivation subscale, whereas for the original English version it loaded highest onto the external motivation subscale (Wild et al., 2006).

The analysis for the SMFL revealed that this questionnaire had a similar factor structure as the TEQ. The SMFL item ending on “I find it interesting” did not fit adequately within the internal/external approach/avoidance matrix, which was consistent with SDT since this form of motivation represents a truly internal motive that does not seem to apply to psychiatric treatment engagement (Jochems et al., 2011; Ryan & Deci, 2008). That is, generally patients do not feel that engaging in psychiatric treatment is “interesting” or “pleasurable” in itself but rather see this as instrumental to the goal of achieving relief from symptoms and/or resolving problems that result from the psychiatric illness and disabilities.

Reliability

The reliabilities of the Dutch TEQ subscales were generally found to be acceptable, although there is still room for improvement as they ranged between 0.66 (external motivation scale for patients with mood and anxiety disorders) and 0.86 (identified motivation scale for patients with mood and anxiety disorders). Possibly, this could be due to heterogeneity of diagnoses and other patient characteristics within these subsamples. Also, it is likely that a selection bias occurred in the study sample as participation in this study was voluntary, and it is plausible to assume that the less motivated or more extrinsically motivated patients were less likely to participate. The found reliabilities are not sufficient for the scale to be used for assessment at the individual level but may currently be used to interpret the sum scores at the level of groups. The test–retest reliabilities after 1 year were generally adequate. The SMFL subscales showed adequate congeneric estimates of reliability, although the external motivation scale for patients with mood and anxiety disorders was too low. The test–retest correlations were generally good. Future studies using larger samples of more homogenous patient groups are needed to determine if the reliabilities of the TEQ and SMFL can be improved on for use of assessments at the individual level.

The HCCQ revealed excellent internal consistency for all subsamples and the total sample, and acceptable test–retest reliability. It should be noted however that the HCCQ showed a ceiling effect, such that there was a restriction of range to high scores as most patients were very satisfied regarding their relationship with their clinician. This was a limitation to the correlational analyses.

Validity

The construct validities of the questionnaires were generally supported by correlations between the different scales and methods, although this was less so for the SMFL. The associations between the three TEQ scales and the HCCQ were consistent with theoretical expectations, as the different types of motivation were differentially related to perceived autonomy support in the manner that SDT predicted. Also, the association between the HCCQ and TEQ identified motivation was relatively stronger than the associations with the IM-MET scale and the TTM-RTC score. Correlations between the TEQ scales and the IM-MET scale and TTM-RTC score suggest that the more broad external–internal distinction of SDT is linked with the quantitative motivation concepts of the TTM and with the IM (although less so) in the manner hypothesized by Jochems et al. (2011), namely, that higher levels of identified motivation are associated with higher levels of IM-MET and TTM-RTC whereas higher levels of external motivation are associated with lower levels of IM-MET and TTM-RTC.

Theoretically, it was expected that when clinicians scored their patients as high on the SES, thereby implicating that patients are available for appointments, seek help when needed, and actively engage in treatment (including taking prescribed medications), this would be positively associated with identified motivation and negatively with external motivation. This hypothesis was only partially supported, since it was found that the patient-rated identified motivation scale was significantly associated with clinician-rated treatment engagement, but neither patient-rated introjected nor external motivation was. It did not matter if we treated the SES scores as a continuous variable (in correlational analyses) or as a dichotomous variable (in t tests and logistic regression analyses) or if we looked at subscales of the SES (results not presented). Looking at clinician-rated TEQ, the associations with SES were stronger and more consistent with theory, which was probably due to their common method. The TEQ subscales of identified and external motivation were associated with legally mandated treatment in the theoretically expected manner, regardless of who rated the TEQ.

Little support was found for the validity of the patient-rated SMFL scales when looking at the associations with criterion measures. Most support was found for SMFL identified motivation, which showed positive association with TTM-RTC, the HCCQ, and with HAQ cooperation and helpfulness as rated by the patient, but not with clinician-rated HAQ subscales or the SES. The implications of the findings on the SMFL are less clear and deserve further study as we face multiple possible explanations for these findings. A plausible explanation is that the correlations were low due to the fact that the SMFL was not administered at the same time as the TEQ and HCCQ. Other explanations are that perhaps the SMFL is an inadequate measure, the criterion measures might have been inadequate, the design of this study could be inappropriate, or a combination of the above. At the moment, it seems that the SMFL might be useful as a short and simple way to communicate with patients about different types of motivation for treatment, as it slightly resembles the TEQ, but it is far from assessment applications at the individual level.

The HCCQ showed moderate to strong association with the HAQ, depending on the rater (patient or clinician) and the subscale (cooperation or helpfulness). As expected, the strongest correlation was found between the HCCQ and the cooperation subscale of the HAQ when rated by the patient (r_s = 0.71, p < .01). This is an exceptionally high correlation when considering that Hemphill (2003) found that correlations of 0.50 or higher correspond to the 89th percentile of most psychological assessment studies. Also, a similar correlation pattern with the TEQ was found for the HCCQ and HAQ cooperation scale when all were rated by the patient. Both the HCCQ and the HAQ were moderately strongly related to clinician-rated treatment engagement. It seems that the scales (and underlying concepts) are not completely interchangeable but they seem to be perceived highly similar, implying a need for a better differentiation of the concept of “clinician autonomy support” from other (perhaps more broadly defined) positive clinician behaviors.

Strengths, Limitations, and Implications

Both a strength and a limitation of this study is the heterogeneity of our study sample that included outpatients with varying primary psychiatric diagnoses and comorbid psychiatric disabilities. Within the three diagnostic groups that we have specified, it is likely that heterogeneity exists between patients with different disorders, such as differences between types of personality disorders, psychotic disorders, or mood disorders. These differences might influence the motivational profiles on the questionnaires. This heterogeneity was ignored to obtain adequate sample sizes for each subgroup, but future studies are needed to replicate the findings. On the other hand, the heterogeneity strengthens the generalizability of the findings to a broad psychiatric outpatient population.

Currently, we feel that the SMFL should not be used for assessment purposes, but might be used for structured communication purposes, such as discussing the patient’s current motivation for engaging in psychiatric treatment. This is relevant considering that our findings seem to suggest that patients and clinicians have different views on the patient’s motivation, which was in line with another study that we recently performed (Jochems et al., unpublished data). In an exploratory analysis on the patient-rated SMFL, where we looked at the proximities of the items, we saw that the items formed an ellipse shape rather than clearly distinct clusters of items (results not reported). We felt that the interpretation of this solution could be understood by consideration of both SDT and approach-avoidance theories of motivation (Elliot, 2006). One axis within the SMFL represents the internal–external continuum proposed by SDT and appeared to be the most important one, whereas the other axis represents approach–avoidance motives, which operates orthogonally to the internal–external axis. Several items can be interpreted as representing approach (“I will feel proud of myself if I do so”) or avoidance motives (“I may not disappoint myself” and “I may not disappoint others”). Using the SMFL as a starting point for a discussion between the patient and clinician regarding the patient’s current motivation for treatment, these external/internal and approach/avoidance motives can be explored. It should be noted that theoretically, patients can endorse all these motives simultaneously as they are not mutually exclusive. A high score for one type of motivation does not preclude a high score for another type. As an example, Silverstein (2010) mentions supported employment for patients with schizophrenia, where the intrinsically motivating psychological benefits of work (e.g., sense of competence and having a valued social role, sense of autonomy) are combined with the benefits of earning money (Silverstein, 2010).

Investigating the construct validity of the scales is an ongoing iterative evaluation process (Smith, 2005) that should be continued after this study, preferably with other criterion measures and in other (larger and more homogenous) patient populations. The predictive utility of the scales should be investigated with longitudinal data, with other measures of treatment engagement, and other outcomes such as dropout, psychosocial functioning, and the patient’s quality of life. These criterion measures should include more objective measures of treatment engagement, such as percentage of no-shows in clinical practice and/or blood levels of (antipsychotic) medication use. Nevertheless, the preliminary results of this study are promising and suggest that the TEQ, SMFL, and HCCQ could be valuable instruments for research on SDT in psychiatric outpatients or for clinical purposes such as discussing the patient’s motivation to engage in treatment.

Footnotes

Acknowledgements

The authors thank all the clinicians and students who helped to obtain the data for this study.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Western North Brabant Mental Health Center in collaboration with the Epidemiological and Social Psychiatric Research Institute of the Erasmus MC University Medical Center Rotterdam. The collection, analyses, and interpretation of data were performed independently of the funder under the responsibility of the first author. Breburg Mental Health Center contributed time, energy, and resources to the study for the inclusion of patients but was otherwise not a funder of the study.

References

Alexander

L. B.

Luborsky

(1986). The Penn Helping Alliance Scales. In Greenberg

L. S.

Pinsof

W. M.

(Eds.), The psychotherapeutic process: A research handbook (pp. 325-366). New York, NY: Guilford Press.

Babyak

M. A.

Green

S. B.

(2010). Confirmatory factor analysis: An introduction for psychosomatic medicine researchers. Psychosomatic Medicine, 72, 587-597. doi:10.1097/PSY.0b013e3181de3f8a

Campbell

D. T.

Fiske

D. W.

(1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.

Choi

Mogami

Medalia

(2010). Intrinsic Motivation Inventory: An adapted measure for schizophrenia research. Schizophrenia Bulletin, 36, 966-976.

Cohen

(1988). Statistical power analysis for the behavioral sciences (Vol. 2). Hillsdale, NJ: Erlbaum.

De Weert-Van Oene

G. H.

De Jong

C. A.

Jorg

Schrijvers

G. J.

(1999). The Helping Alliance Questionnaire: Psychometric properties in patients with substance dependence. Substance Use & Misuse, 34, 1549-1569. doi:10.3109/10826089909039414

Deci

E. L.

Eghrari

Patrick

B. C.

Leone

D. R.

(1994). Facilitating internalization: The self-determination theory perspective. Journal of Personality, 62, 119-142. doi:10.1111/j.1467-6494.1994.tb00797.x

Deci

E. L.

Ryan

R. M.

(2000). The “what” and “why” of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry: An International Journal for the Advancement of Psychological Theory, 11, 227-268.

Deci

E. L.

Ryan

R. M.

(2008). Facilitating optimal motivation and psychological well-being across life’s domains. Canadian Psychology, 49(1), 14-23.

10.

Drake

R. E.

Goldman

H. E.

Leff

Lehman

A. F.

Dixon

Mueser

K. T.

Torrey

W. C.

(2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52, 179-182.

11.

Drieschner

K. H.

Boomsma

(2008a). The Treatment Motivation Scales for Forensic Outpatient Treatment (TMS-F): Construction and psychometric evaluation. Assessment, 15, 224-241. doi:10.1177/1073191107311650

12.

Drieschner

K. H.

Boomsma

(2008b). Validation of the Treatment Motivation Scales for Forensic Outpatient Treatment (TMS-F). Assessment, 15, 242-255. doi:10.1177/1073191107311651

13.

Drieschner

K. H.

Lammers

S. M. M.

van der Staak

C. P. F.

(2004). Treatment motivation: An attempt for clarification of an ambiguous concept. Clinical Psychology Review, 23, 1115-1137. doi:10.1016/j.cpr.2003.09.003

14.

Elliot

A. J.

(2006). The hierarchical model of approach-avoidance motivation. Motivation & Emotion, 30, 111-116.

15.

Hemphill

J. F.

(2003). Interpreting the magnitudes of correlation coefficients. American Psychologist, 58, 78-79.

16.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

17.

Jochems

E. C.

Mulder

C. L.

van Dam

Duivenvoorden

H. J.

(2011). A critical analysis of the utility and compatibility of motivation theories in psychiatric treatment. Current Psychiatry Reviews, 7, 298-312. doi:10.2174/157340011797928204

18.

Jochems

E. C.

Mulder

C. L.

van Dam

Duivenvoorden

H. J.

Scheffer

S. C.

van der Spek

van der Feltz-Cornelis

C. M.

(2012). Motivation and treatment engagement intervention trial (MotivaTe-IT): The effects of motivation feedback to clinicians on treatment engagement in patients with severe mental illness. BMC Psychiatry, 12, 209. doi:10.1186/1471-244X-12-209

19.

Jonge

J. M. D.

Schaap

C. P. D. R.

Schippers

G. M.

(2002). Motivatie voor verandering: Een Nederlandse versie van de University of Rhode Island Change Assessment (URICA-NL) [Motivation for change: A Dutch version of the University of Rhode Island Change Assessment (URICA-NL)]. Diagnostiek-wijzer, 5(3), 114-122.

20.

Martino

Menchetti

Pozzi

Berardi

(2012). Predictors of dropout among personality disorders in a specialist outpatients psychosocial treatment: A preliminary study. Psychiatry and Clinical Neurosciences, 66, 180-186. doi:10.1111/j.1440-1819.2012.02329.x

21.

McBride

Zuroff

D. C.

Ravitz

Koestner

Moskowitz

D. S.

Quilty

Bagby

R. M.

(2010). Autonomous and controlled motivation and interpersonal therapy for depression: Moderating role of recurrent depression. British Journal of Clinical Psychology, 49, 529-545. doi:10.1348/014466509X479186

22.

McConnaughy

E. A.

Prochaska

J. O.

Velices

W. F.

(1983). Stages of change in psychotherapy: Measurement and sample profiles. Psychotherapy: Theory, Research and Practice, 20, 368-375.

23.

Muthén

L. K.

Muthén

B. O.

(1998-2012). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

24.

Nidecker

DiClemente

C. C.

Bennett

M. E.

Bellack

A. S.

(2008). Application of the transtheoretical model of change: Psychometric properties of leading measures in patients with co-occurring drug abuse and severe mental illness. Addictive Behaviors, 33, 1021-1030. doi:10.1016/j.addbeh.2008.03.012

25.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (Vol. 3). New York, NY: McGraw-Hill.

26.

Pelletier

L. G.

Tuson

K. M.

Haddad

N. K.

(1997). Client Motivation for Therapy Scale: A measure of intrinsic motivation, extrinsic motivation, and amotivation for therapy. Journal of Personality Assessment, 68, 414-435.

27.

Prochaska

J. O.

DiClemente

C. C.

(1983). Stages and processes of self-change in smoking: Toward an integrative model of change. Journal of Consulting and Clinical Psychology, 5, 390-395.

28.

Reuterberg

S.-E.

Gustafsson

J.-E.

(1992). Confirmatory factor analysis and reliability: Testing measurement model assumptions. Educational and Psychological Measurement, 52, 795-811.

29.

Ryan

R. M.

Deci

E. L.

(2008). A self-determination theory approach to psychotherapy: The motivational basis for effective change. Canadian Psychology, 49, 186-193. doi:10.1037/a0012753

30.

Ryan

R. M.

Lynch

M. F.

Vansteenkiste

Deci

E. L.

(2010). Motivation and autonomy in counseling, psychotherapy, and behavior change: A look at theory and practice. Counseling Psychologist, 39, 193-260. doi:10.1177/0011000009359313

31.

Ryan

R. M.

Patrick

Deci

E. L.

Williams

G. C.

(2008). Facilitating health behaviour change and its maintenance: Interventions based on self-determination theory. European Health Psychologist, 10, 2-5.

32.

Ryan

R. M.

Plant

R. W.

O’Malley

(1995). Initial motivations for alcohol treatment: Relations with patient characteristics, treatment involvement, and dropout. Addictive Behaviors, 20, 279-297. doi:10.1016/0306-4603(94)00072-7

33.

Silverstein

S. M.

(2010). Bridging the gap between extrinsic and intrinsic motivation in the cognitive remediation of schizophrenia. Schizophrenia Bulletin, 36, 949-956.

34.

Smith

G. T.

(2005). On construct validity: Issues of method and measurement. Psychological Assessment, 17, 396-408.

35.

Staring

A. B.

Van der Gaag

Koopmans

G. T.

Selten

J. P.

Van Beveren

J. M.

Hengeveld

M. W.

. . . Mulder

C. L.

(2010). Treatment adherence therapy in people with psychotic disorders: Randomised controlled trial. British Journal of Psychiatry, 197, 448-455. doi:10.1192/bjp.bp.110.077289

36.

Tait

Birchwood

Trower

(2002). A new scale (SES) to measure engagement with community mental health services. Journal of Mental Health, 11, 191-198. doi:10.1080/09638230020023570-2

37.

Urbanoski

K. A.

Wild

T. C.

(2012). Assessing self-determined motivation for addiction treatment: Validity of the Treatment Entry Questionnaire. Journal of Substance Abuse Treatment, 43(1), 70-79. doi:10.1016/j.jsat.2011.10.025

38.

van Veldhuizen

J. R.

(2007). FACT: A Dutch version of ACT. Community Mental Health Journal, 43, 421-433. doi:10.1007/s10597-007-9089-4

39.

Vansteenkiste

Williams

G. C.

Resnicow

(2012). Toward systematic integration between self-determination theory and motivational interviewing as examples of top-down and bottom-up intervention development: Autonomy or volition as a fundamental theoretical principle. International Journal of Behavioral Nutrition & Physical Activity, 9, 23.

40.

Wild

Cunningham

J. A.

Ryan

R. M.

(2006). Social pressure, coercion, and client engagement at treatment entry: A self-determination theory perspective. Addictive Behaviors, 31, 1858-1872. doi:10.1016/j.addbeh.2006.01.002

41.

Williams

G. C.

Grow

V. M.

Freedman

Z. R.

Ryan

R. M.

Deci

E. L.

(1996). Motivational predictors of weight-loss and weight-loss maintenance. Journal of Personality and Social Psychology, 70, 115-126.

42.

Zeldman

Ryan

R. M.

Fiscella

(2004). Motivation, autonomy support, and entity beliefs: Their role in methadone maintenance treatment. Journal of Social and Clinical Psychology, 23, 675-696.

43.

Zuroff

D. C.

Koestner

Moskowitz

McBride

Marshall

Bagby

(2007). Autonomous motivation for therapy: A new common factor in brief treatments for depression. Psychotherapy Research, 17, 137-147.