Development of a collective efficacy measure for use in social service organizations

Abstract

• Summary: The present study summarizes the development of a collective efficacy measure for use in social service organizations: The Collective Efficacy Measure for Social Services (CEMSS). The measure was adapted from Goddard's Collective Efficacy Scale (2002). Factorial validity was assessed for a sample of 278 respondents employed at child welfare and child psychiatric agencies across Norway. Exploratory factor analysis (EFA) was conducted with a random split-half sample of 139 respondents; the second random split-half sample (n = 139) was tested using confirmatory factor analysis (CFA). Tests for construct validity were conducted using Teamwork, Leadership, and Organizational Culture subscales from the Readiness for Organizational Learning and Evaluation Questionnaire (ROLE; Preskill & Torres, 1999).

• Findings: A six-item scale was identified in the first sample using EFA. The CFA conducted with the second sample, indicated good model fit, χ²(10) = 14.34, p = .16; CFI = .97; RMSEA = .06. Scale reliability for the entire sample was adequate (α = .74). As expected, the six-item scale was significantly correlated with Teamwork, r(250) = .34, p < .01; Leadership, r(273) = .36, p < .01; and Organizational Culture, r(276) = .41, p < .01.

• Applications: The results of the analyses support the use of the CEMSS as a valid measure of collective efficacy. This measure has a variety of possible applications for organizational assessment and research in social service agencies.

Keywords

social work collective efficacy implementation measures methodology organizational structure social work research

The purpose of the present study is to document the development of a measure of collective efficacy for use in therapeutic organizations. The construct of collective efficacy is explained by Bandura's (1997) social cognitive theory; it is an expansion of self-efficacy intended to capture individual group members’ perceptions of the capability of the group to achieve specific goals. In therapeutic organizations, for example, collective efficacy is the aggregate of an individual practitioner's and leader's perceptions that their agency, as a whole, is capable of creating positive outcomes for their clients. Individuals who report higher levels of collective efficacy for their workplace are more certain that the total efforts of their agency can affect positive change for clients.

There is a large body of research that suggests perceived collective efficacy is an important predictor of group behavior and outcomes for organizational tasks; for example, organizations (schools, military, and businesses) with high levels of collective efficacy have shown better group performance on a range of work-related tasks, as well as better decision making and job performance outcomes (e.g. Chen & Bliese, 2002; Ross & Gray, 2006; Tasa & Whyte, 2005). In the field of school research, Goddard, Hoy, and Hoy (2000) have shown a significant contribution of perceived collective teacher efficacy in predicting student academic achievement.

Given the scope and consistency of previous research findings, it follows that therapeutic organizations such as child welfare agencies and psychiatric treatment centers could benefit from a collective efficacy measure. In order for collective efficacy scores in social service agencies to be meaningful, the measure used to assess collective efficacy must be relevant to the specific tasks of those working in therapeutic organizations. Bandura (2006) described the importance of specificity when measuring efficacy, ‘The ‘‘one measure fits all’’ approach usually has limited explanatory and predictive value because most of the items in an all-purpose test may have little or no relevance to the domain of functioning’ (p. 307). To the best of our knowledge, no measure of collective efficacy has been developed specifically for use in therapeutic organizations.

A valid measure of collective efficacy in social service agencies may be particularly useful to implementation researchers. The readiness of an organization for successful implementation of evidence-based practices may be predicted, in part, by an organization's level of collective efficacy (Ledgerwood, 2007). In order to assess whether this is the case for social service agencies implementing new programs, a valid measure of collective efficacy in these organizations is needed. Because implementation science in social services is relatively young, there is a need to develop measures of the organizational factors that may facilitate or hamper the implementation process (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). The purpose of the present study is to develop a new measure of collective efficacy that is applicable to the work conducted in social service organizations.

Our measure of collective efficacy was adapted for use in a study of the nationwide implementation of Parent Management Training Oregon (PMTO) model in Norway. PMTO is an evidence-based intervention, practiced by certified therapists, aimed at assisting parents to develop skills that will help correct and prevent behavior problems in children (Forgatch, Bullock, & Patterson, 2004). During the implementation process, questionnaires assessing organizational characteristics were completed by therapists and leaders from 110 child welfare and child psychiatric agencies throughout Norway; these included agencies participating in the implementation of PMTO and control agencies conducting treatment as usual. Our measure, the Collective Efficacy Measure for Social Services (CEMSS), is an adaptation of the Collective Efficacy Scale developed by Goddard (2002). Goddard's 12-item Collective Efficacy Scale was created for use in schools and was adapted from a longer 21-item questionnaire (Collective Teacher Efficacy) developed by researchers at the University of Michigan and Ohio State University (Goddard et al., 2000).

We hypothesize that: 1) Our collective efficacy scale, the CEMSS, will comprise a single, six-item factor. 2) There will be a positive relationship between the CEMSS and scales of leadership, teamwork, and organizational culture measured by the Readiness for Organizational Learning and Evaluation instrument (Preskill & Torres, 1999).

Method

Participants and procedure

Leaders from every child welfare and child psychiatric organization in Norway were invited to enroll their agencies in the present study. Records of the number of organizations contacted were not kept by the project staff; the number of social service organizations in Norway at the time of the study is also unknown. Of those contacted, 110 agencies agreed to participate. Within the 110 participating agencies, 278 respondents completed packets of eight questionnaires. Respondents were employed as agency leaders (n = 88), PMTO certified therapists (n = 83), and control therapists (n = 107). Of the total sample, 72 percent of respondents were female and 28 percent were male; males comprised 27 percent of the therapist sample and 36 percent of agency leaders.

Participating organizations ranged in size from fewer than six employees (n = 4) to more than 70 employees (n = 5) with a mean of 24 employees; 63 percent were child welfare agencies and 37 percent were child psychiatric centers. Organizations implementing PMTO comprised 75 percent of the sample and 25 percent conducted treatment as usual. Agencies were distributed fairly evenly throughout Norway, representing large, urban cities, rural communities and moderately sized towns.

Measures

Participants completed a packet of eight questionnaires relating to the work environment in their agencies (e.g., leadership quality, agency size, etc.). Surveys and instructions were printed in Norwegian. Two measures from the packet were used in the present study: a collective efficacy measure and the Readiness for Organizational Learning and Evaluation instrument (ROLE), developed by Preskill and Torres (1999).

Our collective efficacy measure is a survey adapted for this study from Goddard's Collective Efficacy Scale (2002). Items were rated on a five-point Likert scale from 1 (completely disagree) to 5 (completely agree). Our measure was modified for use in a study of PMTO program implementation, translated to Norwegian, and back-translated to English, at the Norwegian Center for Child Behavioral Development. The items on Goddard's Collective Efficacy Scale (2002) for schools were changed to reflect the work conducted in therapeutic organizations. For example, Goddard's question, ‘Teachers in this school are able to get through to difficult students’, was reworded to read, ‘Employees in this organization are able to reach difficult clients’. The number of items and the order items were presented was not changed from Goddard's measure, however, due to the differences in school and therapeutic environments, it was necessary to change the meaning of some questions.

We divided our measure into two components with six items categorized as ‘internally focused’ (IF) and six items categorized as ‘externally focused’ (EF) based on the face validity of the items. Internally focused questions were intended to elicit respondents’ beliefs about their organization's ability to help clients. Externally focused items were intended to elicit respondents’ beliefs about clients’ ability and willingness to change during treatment, as well as conditions in the service area. The measure with externally and internally focused items can be seen in Table 1. The IF items make up the construct of interest in the present study. Goddard (2002) also identified two possible factors for his measure. The items in our IF scale closely correspond to those Goddard identified as group competence (GC). The EF items in our measure did not translate well from an educational setting to a therapeutic setting. The decision to focus our analysis on the six IF items was based on the face validity of the items and a desire to define a parsimonious scale of collective efficacy that directly measured employee perceptions of their organization's ability, as a whole, to effectively treat their clients.

Table 1.

Internally focused and externally focused items

1. Employees in this organization are able to reach difficult clients.	IF
2. Employees are sure they can motivate their clients.	IF
3. Employees sincerely believe that change is possible for all their clients.	IF
4. If a client won't change she/he is given up on by the professionals.	EF
5. Employees do not have the skills needed to promote change.	IF
6. When our clients come here they are motivated for change.	EF
7. Employees here can help families change parent–child interaction.	IF
8. Our clients are generally not motivated to change.	EF
9. The conditions in our service area ensure users good opportunities for seeking professional help to change behavior and interaction.	EF
10. Insecurity in the work situations make it difficult to conduct change oriented work.	EF
11. Alcohol and drug abuse in our community make it difficult for clients to change behavior and interaction patterns.	EF
12. Employees do not have the skills needed to work with behavioral problems in children and adolescents.	IF

Note: IF = internally focused, EF = externally focused.

The ROLE was used to assess organizational readiness to learn from evaluation. Three subscales from the questionnaire were used in the present study to demonstrate construct validity with the CEMSS: 1) Organizational Culture, 27 items; 2) Leadership, 14 items; and 3) Teamwork, eight items. Questions were rated on a five-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).

Analysis

Evaluating the factorial structure of the CEMSS required splitting the sample into two subsamples and conducting exploratory factor analysis with subsample 1 and confirmatory factor analysis with subsample 2 (Fabrigar, Wegener, McCallum, & Strahan, 1999). Principal axis factoring was used to identify the number of determined factors in the first random subsample. Confirmatory factor analysis using structural equation modeling (SEM) was applied to the second subsample to confirm the factor structure identified in the first subsample. To provide additional validation of the CEMSS scores in our sample, we conducted tests for construct validity with other organizational items from the ROLE instrument (Preskill & Torres, 1999).

In the present study there were multiple respondents within the same organization which created a nested data structure; the average number of participants per agency was 2.53. Multilevel analysis is often employed with nested data to account for the shared variance of the group. However, as cases within groups approach 1, portioning variance accurately across levels becomes problematic. In a series of Monte Carlo simulations, Clark (2008) suggested that type I error may increase with as few as two cases per group when nested data were analyzed using single-level analyses; however, the results of the study also suggested a minimum of five cases per group were needed to calculate accurate parameter estimates. Because we were more concerned with accurate model parameter estimates and factor loadings than with type I error, we determined that single-level analysis was the better option. A calculation of the design effect (Muthén & Satorra, 1995) provided further support for our decision. The design effect is an estimate of how much bias is introduced if the assumption of independence is violated. In the present study, the design effect (deff = 1.81) was low enough to suggest that the results of a single-level analysis would not be overly biased (Maas & Hox, 2004; Muthén & Satorra, 1995).

Frequency distributions

The frequency distribution of the items was examined for normality. The univariate skewness ranged from −0.27 to −1.15 and the kurtosis was 0.15 to 1.29, well within the range of plus or minus 2.00 suggested by Curran, West, and Finch (1996). However, an examination of skewness and kurtosis in relation to the standard error of skewness (ss = 0.02) and standard error of kurtosis (sk = 0.09) on a z distribution suggested that the values for skew and kurtosis were likely not a result of chance. This is discussed later in relation to confirmatory factor analysis.

Exploratory factor analysis

Subsample 1 was analyzed using principal axis factoring with the Statistical Package for the Social Sciences (SPSS) version 15.0. The number of factors extracted was freely allowed. The number of factors and assignation of items to factors was assessed using multiple criteria; the items were evaluated based on eigenvalues, structure coefficients and theoretical relevance.

Confirmatory factor analysis

Confirmatory factor analysis (CFA) was conducted with subsample 2 using the program Amos version 7 (Arbuckle, 2006) with the items identified in the exploratory factor analysis. Model fit was evaluated using chi-square goodness of fit, the root mean squared error (RMSEA; Brown & Cudeck, 1993), and comparative fit index (CFI; Bentler, 1990). As previously mentioned, univariate normality could not be assumed due to skewness and some kurtosis in the observed variables. An important assumption of maximum likelihood estimation, that of multivariate normality, was also likely violated as a result of the univariate non-normality. We, therefore, used boot-strapped standard errors in order to minimize over-inflation of chi-square and under-estimation of the standard errors.

Construct validity

Following the confirmatory factor analysis, a mean score of collective efficacy was computed for all respondents. Indication that our collective efficacy scale positively identified the appropriate dimension was assessed using correlations with other scales designed to measure similar, but not identical, concepts. Chen and Bliese (2002) identified leadership as an important antecedent to collective efficacy. Measures of teamwork and workplace culture tap into domains related to collective efficacy and were also expected to correlate well with collective efficacy (Marshall, 2004; Mattingly, 2008). Convergence was assessed using bi-variate correlations of the CEMSS and three subscales taken from the ROLE instrument: 1) Leadership; 2) Organizational Culture; 3) Teamwork.

Results and discussion

Factor analysis

The initial results of the factor analysis using six items in the scale yielded a single factor solution. The factor had moderate to high structure coefficients for the six items in the scale (Table 2) and accounted for 47.7 percent of the total variance; the eigenvalue for the first factor was 2.86. The reliability for the six-item scale in the subsample supported good measurement consistency (α = .78).

Table 2.

EFA structure coefficients for the six CEMSS items

1. Employees in this organization are able to reach difficult clients.	.70
2. Employees are sure they can motivate their clients.	.65
3. Employees sincerely believe that change is possible for all their clients.	.60
4. Employees do not have the skills needed to promote change.^a	.77
5. Employees here can help families change parent-child interaction.	.69
6. Employees do not have the skills needed to work with behavioral problems in children and adolescents.^a	.73

Items reverse-coded.

Confirmatory factor analysis

Structural equation modeling was used to conduct a factor analysis with the six-item scale for subsample 2. The results of the CFA indicated good fit to the data for the subsample, χ²(10) = 14.34, p = .16; CFI = .97; RMSEA = .06. Achieving good model fit necessitated the opening of covariances on error terms between items 1, 2 and 3 and between items 4 and 6 (Figure 1).

Figure 1.

Confirmatory factor analysis of the six CEMSS items. CE 01: Employees in this organization are able to reach difficult clients. CE 02: Employees are sure they can motivate their clients. CE 03: Employees sincerely believe that change is possible for all their clients. CE 04a: Employees do not have the skills needed to promote change. CE 05: Employees here can help families change parent-child interaction. CE 06a: Employees do not have the skills needed to work with behavioral problems in children and adolescents. ^aItems reverse-coded.

Freeing covariances in CFA is acceptable given a reasonable justification (Byrne, 1991; Thompson & Daniel, 1996). In this case, items 1: ‘Employees in this organization are able to reach difficult clients’, 2: ‘Employees are sure they can motivate their clients’ and 3: ‘The employees sincerely believe that change is possible for all their clients’, are broad, positively worded statements with similar tone that reference general client change. Items 4: ‘Employees do not have the skills needed to promote change’, and 6: ‘Employees do not have the skills needed to work with behavioral problems in children and adolescents’, are similar; the questions are both negatively worded and focus on employees’ skill to help their clients. Therefore, we believe it is reasonable that the measurement error for these two sets of questions covaried. Reliability of the six-item scale for the full sample supported good measurement consistency (α = .74).

Convergent validity

The three subscales from the ROLE indicated high convergence with the six-item collective efficacy scale identified in the factorial validation. The Cronbach's alpha reliabilities for the leadership, teamwork, and workplace culture subscale scores in our sample were α = .95, α = .91, and α = .87 respectively. All three subscales from the ROLE showed moderate positive correlations with collective efficacy (Table 3).

Table 3.

Correlations between the CEMSS, Leadership, Workplace Culture and Teamwork

	CEMSS	Leadership	Culture	Teamwork
CEMSS	1.00
Leadership	.36**	1.00
Culture	.41**	.77**	1.00
Teamwork	.34**	.61**	.70**	1.00

Correlation is significant at the .01 level, 2-tailed.

Conclusion

Our first hypothesis that the six internally focused items from the original questionnaire would comprise a single factor was supported. The final scale was constructed of the six items, which we believe has good face validity for the construct of collective efficacy (i.e. group members’ perceptions of the capability of the group as a whole to achieve specific goals).

Our second hypothesis was also supported. The correlations between the CEMSS and the three discrete scales from the ROLE, with constructs similar to collective efficacy, are a good indication that our measure is capturing the construct of interest. The Cronbach's alpha for the final six-item measure supported moderate consistency within in this sample (α = .74). The reliability coefficient of the measure may have been higher if more items had been included in the final scale. Researchers who wish to develop this measure could experiment with adding additional items to the scale, however, more items could artificially inflate the alpha without necessarily improving the internal consistency of the measure (Streiner, 2003).

Previous researchers (e.g. Goddard et al., 2000) have provided evidence for the predictive validity of collective efficacy in organizations and schools; our study was an effort to develop a measure that can be used to extend this work into social service organizations. It is our hope that researchers will use this measure to investigate the function of collective efficacy in social service organizations (e.g. the relationship between collective efficacy and program implementation outcomes), as well as to provide further support for the validity of the measure.

Ethics

Human subjects’ protocols from The Norwegian National Committee for Research Ethics and the Norwegian Social Science Data Registry were followed. In addition, the Internal Review Board for research with human subjects at Oregon Social Learning Center in the United States approved the study prior to data collection.

Footnotes

Acknowledgements

The authors would like to thank Eyvind Elgesem for his work developing the Collective Efficacy Measure for Social Services (CEMSS), Carl Ivar Holmen for translating the measure to Norwegian and Asgeir Røryhus Olseth for back-translating the CEMSS to English. We would also like to acknowledge Marion Forgatch and Terje Ogden as principal investigators of the PMTO program in Norway.

Funding

This work was supported by the Prevention Research Branch, NIDA, US PHS (grant no. RO1 DA 16097). Data used in this study are available from the authors.

References

Arbuckle, J. L. (2006). Amos (Version 7.0) [Computer software]. Spring House, PA: Amos Development Corporation.

Bandura

. (1997). Self-efficacy: The exercise of control. New York: W. H. Freeman and Company.

Bandura

. (2006). Guide for constructing self-efficacy scales. In Pajares

Udan

(Eds.), Self-efficacy beliefs of adolescents (pp. 307–337). Greenwich, CT: Information Age Publishing.

Bentler

P. M.

. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246.

Brown

M. W.

Cudeck

. (1993). Alternative ways of assessing model fit. In Bollen

K. A.

Long

J. S.

(Eds.), Testing structural equation models (pp. 136–162). Newbury Park, CA: SAGE.

Byrne

B. M.

. (1991). The Maslach burnout inventory: Validating factorial structure and invariance across intermediate, secondary, and university educators. Multivariate Behavioral Research, 26(4), 583–605.

Chen

Bliese

P. D.

. (2002). The role of different levels of leadership in predicting self- and collective efficacy: Evidence for discontinuity. Journal of Applied Psychology, 87(3), 549–556.

Clark

. (2008). When can group level clustering be ignored? Multilevel models versus single-level models with sparse data. Journal of Epidemiology and Community Health, 62(8), 752–758.

Curran

P. J.

West

S. G.

Finch

J. F.

. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16–29.

10.

Fabrigar

L. R.

Wegener

D. T.

McCallum

R. C.

Strahan

E. J.

. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272–299.

11.

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

. (2005). Implementation research: A synthesis of the literature. National Implementation Research Network. University of South Florida.

12.

Forgatch

M. S.

Bullock

B. M.

Patterson

G. R.

. (2004). From theory to practice: Increasing effective parenting through role-play: The Oregon model of parent management training (PMTO). In Steiner

(Ed.), Handbook of mental health interventions in children and adolescents: An integrated developmental approach. San Francisco, CA: Jossey-Bass.

13.

Goddard

R. D.

. (2002). A theoretical and empirical analysis of the measurement of collective efficacy: The development of a short form. Educational and Psychological Measurement, 62(1), 97–110.

14.

Goddard

R. D.

Hoy

W. K.

Hoy

A. W.

. (2000). Collective teacher efficacy: Its meaning, measure, and impact on student achievement. American Educational Research Journal, 37(2), 479–507.

15.

Ledgerwood, A. D. (2007). Supporting successful implementation of evidence-based programs: Assessing readiness and collective efficacy. Master's thesis, Miami University, Department of Psychology.

16.

Maas

C. J. M.

Hox

J. J.

. (2004). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58(2), 127–137.

17.

Marshall

L. C.

. (2004). The relationship between efficacy, teamwork, effort and patient satisfaction. Dissertation Abstracts International: Section B. Sciences and Engineering, 64(9).

18.

Mattingly

J. W.

. (2008). A study of relationships of school climate, school culture, teacher efficacy, collective efficacy, teacher job satisfaction and intent to turnover in the context of year-round education calendars. Dissertation Abstracts International: Section A. Humanities and Social Sciences, 68(10).

19.

Muthén

B. O.

Satorra

. (1995). Complex sample data in structural equation modeling. Sociological Methodology, 25, 267–316.

20.

Preskill

Torres

. (1999, November). Assessing an organization's readiness for learning from evaluative inquiry, Presented at the American Evaluation Association Conference, Orlando, FL.

21.

Ross

J. A.

Gray

. (2006). Transformational leadership and teacher commitment to organizational values: The mediating effects of collective teacher efficacy. School Effectiveness and School Improvement, 17(2), 179–199.

22.

Streiner

D. L.

. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99–103.

23.

Tasa

Whyte

. (2005). Collective efficacy and vigilant problem solving in group decision making: A non-linear model. Organizational Behavior and Human Decision Processes, 96(2), 119–129.

24.

Thompson

Daniel

L. G.

. (1996). Factor analytic evidence for the construct validity of scores: A historical overview and some guidelines. Educational and Psychological Measurement, 56(2), 197–208.