Organizational Capabilities for Integrating Care

Abstract

The success of integrated care interventions is highly dependent on the internal and collective capabilities of the organizations in which they are implemented. Yet, organizational capabilities are rarely described, understood, or measured with sufficient depth and breadth in empirical studies or in practice. Assessing these capabilities can contribute to understanding why some integrated care interventions are more effective than others. We identified, organized, and assessed survey instruments that measure the internal and collective organizational capabilities required for integrated care delivery. We conducted an expert consultation and searched Medline and Google Scholar databases for survey instruments measuring factors outlined in the Context and Capabilities for Integrating Care Framework. A total of 58 instruments were included in the review and assessed based on their psychometric properties, practical considerations, and applicability to integrated care efforts. This study provides a bank of psychometrically sound instruments for describing and comparing organizational capabilities. Greater use of these instruments across integrated care interventions and studies can enhance standardized comparative analyses and inform change management. Further research is needed to build an evidence base for these instruments and to explore the associations between organizational capabilities and integrated care processes and outcomes.

Keywords

integrated care integrated delivery systems organizational context organizational behavior performance measurement

As populations age, health system reform efforts in many countries emphasize integrated care as a means to better manage the complex health and social care needs of the elderly and other vulnerable patient groups (Johri, Beland, & Bergman, 2003; Pham, 2010; Wodchis, Dixon, Anderson, & Goodwin, 2015). Integrated care interventions bring together multiple health and/or social service professionals and organizations to replace fragmented care with care that is better coordinated, patient centered, and cost effective (Singer et al., 2011). Despite the growth of integrated care interventions, evidence of their effectiveness is mixed and relatively little is known about what factors are associated with successful implementation over time and across diverse contexts (Bardsley, Steventon, Smith, & Dixon, 2013; Low, Yap, & Brodaty, 2011). As a result, experts often challenge the transferability of successful integrated care interventions and emphasize adaptation to the local context as a prerequisite for success (Goodwin, Dixon, Anderson, & Wodchis, 2014; Stewart, Georgiou, & Westbrook, 2013). The success of integrated care interventions may thus be strongly influenced by the context and associated capabilities of the organizations in which they are implemented (Bardsley et al., 2013; Curry & Ham, 2010; Woolf & Johnson, 2005). Assessing these capabilities is challenging, as there are many relevant concepts and measures linked to improved care integration.

We define “organizational capabilities” as the ability and capacity of an organization, or group of organizations, to perform coordinated sets of tasks that support integrated care delivery. This definition encompasses both the internal capabilities of individual organizations and the collective interorganizational capabilities needed to leverage and combine knowledge and resources from multiple organizations. Organizational capabilities derive from the knowledge and skills of the people in the organization as well as organizational structures, processes, and norms supporting integration (Teece, Pisano, & Shuen, 1990). Organizational capabilities can include leadership, governance, and information technology as well as key processes such as improving quality of care. Such factors are widely recognized as influencing the success or failure of integrated care interventions (Curry & Ham, 2010; Friedman & Goes, 2001; Gillies, Shortell, Anderson, Mitchell, & Morgan, 1993; Ling, Brereton, Conklin, Newbould, & Roland, 2012; Suter, Oelke, Adair, & Armitage, 2009), but the current knowledge base lacks specificity about when and how these factors matter. This gap in knowledge is attributable, in large part, to the limited description and measurement of organizational capabilities in empirical studies on integrated care.

Many studies evaluating integrated care interventions describe the organizations involved in implementing the intervention in a few sentences, if at all (e.g., Baillie et al., 2014; Kautz, Gittell, Weinberg, Lusenhop, & Wright, 2007; Solberg et al., 2009), or incorporate a few, easily measureable variables such as the organizations’ age, size, and staff mix (Burns et al., 2001; Lin & Wan, 1999; Wan, Ma, & Lin, 2001). Some integrated care studies provide more detailed assessments of organizational capabilities but use qualitative case study methods (e.g., Williams & Sullivan, 2009; Kreindler et al., 2012), limiting standardized comparative analyses. The growth of efforts in many settings to integrate care speaks to the broad need for more systematic and standardized approaches, utilizing both qualitative and quantitative methods, to measure and evaluate the organizational capabilities required to implement and sustain integrated care interventions (Wodchis et al., 2015). To date few measurement systems have been developed with the aim of rigorously characterizing and comparing a wide range of organizational capabilities, spanning structural, process, and sociocultural elements (e.g., Estabrooks, Squires, Cummings, Birdsell, & Norton, 2009).

The aim of this article was to identify, organize, and assess survey instruments and scales for describing and comparing the organizational capabilities required for effective integrated care delivery and to examine the extent to which identified instruments have been used to study integrated care interventions. Using a systematic approach, we developed a bank of psychometrically sound instruments to describe and compare a range of organizational capabilities; to aid the standardization of data collection on organizational capabilities across integrated care delivery models, settings, and studies; and to inform change management efforts.

Method

We used the Context and Capabilities for Integrating Care (CCIC) Framework to conceptualize organizational capabilities and to guide the identification of instruments. The CCIC Framework was developed through a literature review of integrated care and revised and validated through semistructured interviews with leaders and care providers engaged in the implementation and operation of integrated care networks in Ontario, Canada (Evans, Grudniewicz, Baker, & Wodchis, 2016). Participants represented over 35 distinct integrated care networks from across the province of Ontario and hailed from diverse organizations including primary care practices, hospitals, and community-based health and social service organizations. The CCIC Framework consists of 18 organizational and interorganizational capabilities in three categories: (1) basic structures, (2) people and values, and (3) key processes (Table 1). The framework shows how these organizational capabilities interact with characteristics of the integrated care intervention and characteristics of the patient population to shape the implementation and outcomes of integrated care interventions.

Table 1.

Context and Capabilities for Integrating Care (CCIC) Framework: Definitions and Examples.

Concept	Definition	Examples
Basic structures
Physical features	Structural and geographic characteristics of the organization/practice and network	Organization/practice size and age, urban or rural location, facilities, and geographic proximity of network members
Resources	Availability of tangible and intangible assets for ongoing operations at the organization/practice and for network activities	Staffing, funding, knowledge, time, project management support, administrative support, and brand/reputation
Governance	How the board or steering committee is organized and its activities to direct, manage, and monitor the affairs of the organization/practice and network	Board/committee composition, types of subcommittees, frequency of meetings, types of decisions made (e.g., extent of centralized planning and standardization)
Accountability	The mechanisms in place to ensure that people and organizations meet formal expectations in the organization/practice and network	Regulations enforced by an authority (e.g., government), formal agreements between organizations (e.g., data sharing), organizational mandates, and professional scope of practice
Information technology	The availability and ease of use of technology-based communication and information storage mechanisms in the organization/practice and across the network	Shared electronic medical records, e-mail communication, videoconferencing, data access and mining, and telehealth care
Organizational/network design	The arrangement of units and roles and how they interact to accomplish tasks in the organization/practice and network	Organizational chart (hierarchy), types of departments/programs, job descriptions, communication, and decision-making channels (e.g., extent of centralization and formalization)
People and values
Leadership approach	The methods and behaviors used by formal leaders in the organization/practice or network (i.e., individual leaders, leadership teams, or lead organizations)	Vision for the organization/practice or network, strategies used to empower staff, leadership style, and competencies
Clinician engagement and leadership	The formal and informal roles held by clinicians in the organization/practice and network, particularly physicians, that enable them to buy into and steer change, and influence others	Active involvement of clinicians in planning, leading or supporting new initiatives (e.g., clinical champions or directors, networks led by primary care practices)
Organizational/ network culture^a	Widely shared values and habits in the organization/practice or network	Perceptions regarding what is important and what is appropriate behavior
Focus on patient centeredness and engagement	Commitment to placing patients at the center of clinical, organizational and network decision-making	Collection and use of patient feedback, consideration for patient needs and preferences, patient input and representation on committees as a standard practice, patient involvement in codesigning services
Commitment to learning	The existence of a set of values and practices that support ongoing development of new knowledge and insights within the organization/practice and network	Experimentation encouraged and rewarded, forums for meeting with and learning from other organizations and external experts, time and resources to reflect on past performance
Work environment	How employees perceive and experience their job and their workplace in the organization/practice and network	Opportunity for input, job satisfaction, and burnout
Readiness for change	The extent to which organizations and individuals are willing and able to implement change in the organization/practice and network	Attitudes toward change and toward new or innovative ideas, extent of fit between current vision/strategy and the change
Key processes
Partnering	The development and management of formal and informal connections between different organizations/practices	Sharing information, sharing staff, engaging in collaborative problem-solving, building a common understanding and vision, exchanging knowledge, and implementing referral and discharge/transfer agreements
Delivering care	The methods used by providers in caring for patients in the organization/practice and network	Interprofessional teamwork and joint care planning, use of standardized decision support tools, medical model versus holistic model of care, and shared patient-provider decision-making
Measuring performance	The systematic collection of data about how well the organization/practice and network is meeting its goals	Shared performance measurement framework, regular measurement and reporting, and data access and mining
Improving quality	The use of practices and processes that continuously enhance patient care in the organization/practice and network	Providing quality improvement (QI) training to staff, systematic use of QI methods (e.g., process mapping, control charts), and application of best practices

^aCapabilities such as focus on patient centeredness and engagement, commitment to learning, and readiness for change may manifest in the culture of the organization or network.

Independent, iterative searches were conducted for each concept in the CCIC Framework using Medline and Google Scholar. Test searches were conducted to gauge the yield and foci of papers, and terms were modified accordingly to enhance relevance. The results of the database searches were displayed in order of relevance using each database’s sort function. We screened titles and abstracts for a minimum of the first 100 results and a maximum of the first 350 results; the average number of papers screened per factor was 284. We terminated screening when at least 30 consecutive papers were deemed irrelevant based on the inclusion criteria. To ensure that the remaining, unscreened references did not include potentially relevant studies, as part of our initial scoping searches we reviewed for relevancy five randomly selected titles/abstracts per search from among those that would not be screened based on our search strategy. None of the papers reviewed were relevant. Additional details on the search strategy are available upon request from the authors.

Abstracts and the full text of potentially relevant papers were screened for inclusion by two authors (J.M.E. and A.G.). Papers had to meet the following criteria for inclusion: (a) written in the English language, (b) description of the development and/or use of the quantitative instrument (or scale) in sufficient detail to enable assessment of its content, (c) the instrument measures one or more constructs in the conceptual framework, (d) the intended respondents of the instrument include health-care leaders and/or care providers (not exclusively patients or family members who may be unable to assess most dimensions), (e) a full copy of the instrument is accessible or can be obtained from the author(s) free of charge, and (f) the instrument has been used in a health-care setting. Papers published up to and including the year 2013 were included. We excluded proprietary instruments, as their cost may limit use by researchers and practitioners. We also excluded instruments that do not have details on their development and measurement properties in peer-reviewed publications, such as those described in books and theses. Systematic reviews of instruments that emerged during our search and the reference lists of included papers were also reviewed to identify potentially relevant papers for screening.

We supplemented our literature search with an expert consultation. A convenience sample of experts in health services management research across Canada were asked to identify instruments that measure the organizational capabilities outlined in the CCIC Framework. Expert consultation is commonly used in scoping and systematic reviews (Levac, Colquhoun, & O’Brien, 2010; Petticrew & Roberts, 2005), including reviews of measurement instruments (e.g., Kaplan et al., 2010; Scott, Mannion, Davies, & Marshall, 2003). Investigators with the Health System Performance Research Network (www.hsprn.ca) and other Canadian experts known to members of the research team were contacted. A total of 40 experts responded and collectively identified 30 instruments/papers.

The following data were extracted from each included paper: purpose of the study, study design, setting, respondents, name of the instrument, purpose of the instrument, instrument dimensions and definitions, theoretical basis of the instrument, number of survey items, type of scale, and psychometric properties. The data extraction table was organized by instrument and focused primarily on the original paper that introduced and tested the instrument.

Three key criteria were used to assess the instruments. The first criterion was psychometric properties. Using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) Checklist (http://www.cosmin.nl/the-cosmin-checklist_8_5.html) and the work of Brennan, Bosch, Buchan, and Green (2012, 2013) as guides, we examined each instrument’s psychometric properties (validity and reliability). The second criterion was practical considerations, including cognitive complexity, length, previous response rates, and complex administration requirements (e.g., group consensus method with facilitator). Instruments with very weak psychometric properties, insufficient detail on psychometric properties, or high respondent or researcher burden were excluded at this stage. The third criterion was applicability to integrated care interventions. Two questions guided assessment of instrument applicability: (1) Has the instrument been used with an integrated care intervention? If not, can it be readily applied or easily modified for use with an integrated care intervention? and (2) to what extent is the instrument applicable to the diverse types of care providers and organizations that may be involved in an integrated care intervention (e.g., primary care providers, acute care hospitals, home care providers, public health units, and social service organizations)? If not widely applicable, can the instrument be easily modified to enhance relevance to multiple health-care settings? To determine use in an integrated care intervention or setting, we searched Google Scholar using the instrument name paired with the term “integrated care.” Relevant academic as well as gray literature publications were included. Instruments deemed too specific to a particular context (e.g., intensive care units) or set of professionals (e.g., doctor–nurse relationships), and not easily modified for broader applicability, were excluded at this stage. Only instruments that passed all criteria described above are reported on here (n = 58).

Results

A total of 4,580 titles and abstracts were screened for inclusion based on the database searches and the expert consultation. After the removal of irrelevant papers, 175 underwent full-text screening and an additional 32 papers were excluded for irrelevancy. The remaining 143 papers/instruments were reviewed against the inclusion criteria, and 60 instruments were excluded. Several relevant instruments were excluded at this stage because the instruments were proprietary, not publicly available, authors were unreachable, or authors did not respond to our request for a copy of the instrument. Examples of very promising proprietary instruments that were excluded include the Multifactor Leadership Questionnaire (Bass & Avolio, 1990) and the Organizational Social Context Measure (Glisson et al., 2008). The 83 remaining instruments were assessed for psychometric properties, practical considerations, and applicability to integrated care interventions, and 25 survey instruments were subsequently excluded. Reasons for exclusion were related primarily to a lack of information on scale development and/or psychometric properties and to weak applicability to integrated care interventions (e.g., survey items that were specific to a particular health-care setting, interprofessional relationship, or clinical innovation). Figure 1 summarizes the screening process. Table 2 outlines the name of included instruments (n = 58) as well as which factor(s) in the CCIC Framework each instrument measures. Appendices outlining the characteristics of the included instruments and their psychometric properties are available online as supplementary materials.

Figure 1.

Flowchart of screening process.

Table 2.

Included Instruments and Organizational Factors Measured.

Instrument	Organizational Factor(s) Measured From the Context and Capabilities for Integrating Care Framework
Basic structures
Clinical Information System Implementation Evaluation Scale (Gugerty, Maranda, & Rook, 2006)	Information technology
Clinical Information Systems Success Model Questionnaire (Garcia-Smith & Effken, 2013)	Information technology
People and values
Survey of Transformational Leadership (Edwards, Knight, Broome, & Flynn, 2010)	Leadership approach
Clinical Governance Development Index (Gauld, Horsburgh, & Brown, 2011)	Clinician engagement and leadership
Learning Climate Questionnaire (Mikkelsen & Grønhaug, 1999)	Commitment to learning
Short-Form Learning Organization Survey (Singer, Moore, Meterko, & Williams, 2012)	Commitment to learning
Research and Development Culture Index (Watson, Clarke, Swallow, & Forster, 2005)	Commitment to learning
Change Readiness Survey (Holt, Armenakis, Feild, & Harris, 2007)	Readiness for change
Commitment to Change Measure (Herscovitch & Meyer, 2002)	Readiness for change
Group Innovation Inventory (Caldwell & O’Reilly, 2003)	Readiness for change
Organizational Change Questionnaire (Bouckenooghe, Devos, & Van den Broeck, 2009)	Readiness for change
Organizational Readiness to Change Assessment (Helfrich, Li, Sharp, & Sales, 2009)	Readiness for change
Organizational Readiness for Change Scales (Pare, Sicotte, Poba-Nzaou, & Balouzakis, 2011)	Readiness for change (and information technology)
Resistance to Change Scale (Oreg, 2003)	Readiness for change
Medical Group Practice Culture (Kralewski, Dowd, Kaissi, Curoe, & Rockwood, 2005)	Organizational culture
Organizational Culture Assessment Instrument (Cameron & Quinn, 2011; Zammuto & Krakower, 1991)	Organizational culture
Performance Values Questionnaire (Minvielle et al., 2008)	Organizational culture
Alberta Context Tool (Estabrooks, Squires, Cummings, Birdsell, & Norton, 2009)	Work climate (and resources, leadership, organizational culture, measuring performance, and delivering care)
Copenhagen Burnout Inventory (Kristensen, Borritz, Villadsen, & Christensen, 2005)	Work climate
Organizational Justice Evaluation Scale (Moorman, 1991)	Work climate
Job Content Questionnaire (Karasek et al., 1998)	Work climate
Organizational Citizenship Behavior (Moorman, 1991	Work climate
Organizational Climate Measure (Patterson et al., 2005)	Work climate (and leadership approach, organizational design, measuring performance, and readiness for change)
Organizational Climate Questionnaire (Dawson, González-Romá, Davis, & West, 2008)	Work climate
Organizational Commitment Questionnaire (Porter, Steers, Mowday, & Boullian, 1974 & Mowday, Steers, & Porter, 1979)	Work climate
Psychological Empowerment (Spreitzer, 1995)	Work climate
Survey of Organizational Attributes for Primary Care (SOAPC) (Ohman-Strickland et al., 2007)	Work climate (and clinician engagement and leadership)
Key processes
Interprofessional Collaboration Survey (Nuno-Solinis, Zabalegui, Arce, Rodríguez, & Polanco, 2013)	Teamwork/Delivering care^a
Group Climate and Group Process Questionnaire (Wilkens & London, 2006)	Teamwork/Delivering care
Relational Coordination Survey (Gittell, 2002)	Teamwork/Delivering care
Team Climate Inventory Short-Form (Kivimaki & Elovainio, 1999; original by Anderson & West, 1998)	Teamwork/Delivering care
Team Decision Making Questionnaire (Batorowicz & Shepherd, 2008)	Teamwork/Delivering Care
Team Integration Measure (Lichtenstein, Alexander, Jinnett, & Ullman, 1997)	Teamwork/Delivering care
Team Psychological Safety and Team Learning Behavior Scales (Edmondson, 1999)	Teamwork/Delivering care
Hospital Project Team Questionnaire (Lemieux-Charles et al., 2002)	Teamwork/Delivering care
Transactive Memory System Scale (Lewis, 2003)	Teamwork/Delivering care
Trust in Teams Measure (Costa & Anderson, 2011)	Teamwork/Delivering care
Work Group Characteristics Measure (Campion, Medsker, & Higgs, 1993)	Teamwork/Delivering care
Collaborative Practice Assessment Tool (Schroder et al., 2011)	Teamwork/Delivering Care
Care Process Self-Evaluation Tool (Vanhaecht et al., 2007; Seys et al., 2013)	Teamwork/Delivering care (and focus on patient centeredness and engagement)
Work Group/Organizational Climate Measure (Benzer et al., 2011)	Teamwork (and work climate)
Internal Participation Scale (Korner & Witz, 2013)	Teamwork/Delivering care
Index of Interdisciplinary Collaboration (Bronstein, 2002)	Teamwork/Delivering care
Multigroup Measurement Scale for Interprofessional Collaboration (Kenaszchuk, Reeves, Nicholas, & Zwarenstein, 2010)	Teamwork/Delivering care
The Team Survey (Millward & Ramsay, 1998)	Teamwork/Delivering care
Assessment of Interprofessional Team Collaboration Scale (Orchard, King, Khalili, & Bezzina, 2012)	Teamwork/Delivering care
Teamwork Scales (Shortell et al., 2004)	Teamwork/Delivering care
Evaluation of Quality Systems Survey (Wagner, De Bakker, & Groenewegen, 1999)	Improving quality
Quality Improvement Implementation Survey (Shortell et al., 1995)	Improving quality
QI Maturity Tool (Joly, Booth, Mittal, & Shaler, 2012)	Improving quality
Evidence-Based Practice Attitude Scale (Aarons, 2004; Aarons, McDonald, Sheehan, & Walrath-Greene, 2007)	Delivering care
Context Assessment Index (McCormack, McCarthy, Wright, & Coffey, 2009)	Delivering care
Implementation Leadership Scale (Aarons, Ehrhart, & Farahnak, 2014)	Delivering care (and leadership approach)
Organizational Readiness for Evidence- Based Interventions (Stamatakis et al., 2012)	Delivering care (and readiness for change)
Partnership Self-Assessment Tool (Weiss, Anderson, & Lasker, 2002; short-form by Cramm, Strating, & Nieboer, 2011)	Partnering
Synergy in Health Promotion Partnerships Scale (“Jones Synergy Scale”; Jones & Barry, 2011a)	Partnering
Trust in Health Promotion Partnerships Scale (Jones & Barry, 2011b)	Partnering

^aDue to the high number of instruments specifically on teamwork, we categorized these instruments as “teamwork/delivering care” to distinguish them from four other instruments categorized under “delivering care” which do not examine teamwork.

Although most of the instruments were designed for use in health-care settings, some were originally developed and tested in business settings and later adapted for use in health care (n = 14). All of the instruments were self-administered questionnaires with ordinal scales. Most of the instruments measured factors captured under “people and values” (n = 26) and “key processes” (n = 30) as opposed to “basic structures” (n = 2). Organizational factors with the greatest selection of instruments include teamwork (n = 19, under the heading “delivering care”), work environment (n = 11), and readiness for change (n = 7). Only one instrument was identified for each of senior leadership approach and clinician engagement and leadership factors. There were also several factors, primarily under basic structures, for which no dedicated instruments were identified, including physical features, resources, governance, accountability, organizational design, measuring performance, and focus on patient centeredness and engagement. However, as Table 2 depicts, several instruments have survey items that reflect these elements. For example, the Partnership Self-Assessment Tool (PSAT) includes subscales on leadership and resources (Weiss, Anderson, & Lasker, 2002); the Alberta Context Tool includes subscales on work environment, culture, leadership, resources, and measuring performance (Estabrooks et al., 2009); the Survey of Organizational Attributes in Primary Care (SOAPC) includes items on clinician engagement, measuring performance, and organizational design (Ohman-Strickland et al., 2007); and the Care Process Self-Evaluation Tool (CPSET) includes items on teamwork, focus on patient centeredness and engagement, measuring performance, and partnering (Vanhaecht et al., 2007).

Instrument Development and Psychometric Properties

The development of all included instruments involved, at minimum, a literature review with reference to relevant theories. However, the quality or depth of the reviews and descriptions of the underlying instrument domains varied widely. The instruments were developed and/or applied in various countries, including the United States, the United Kingdom, Canada, Australia, New Zealand, Italy, Spain, France, Germany, Norway, Belgium, Denmark, Finland, Sweden, and the Netherlands. Common methods used during the development of the instruments included expert assessment (n = 21), stakeholder interviews or focus groups (n = 18), and pretesting or pilot testing (n = 19); most studies used a combination of these methods. Only four studies used cognitive testing. Most of the instruments underwent exploratory and/or confirmatory factor analysis to determine dimensionality (only 5 did not) as well as hypothesis testing to examine validity (12 did not). Internal consistency of the instruments and their scales, measured using Cronbach’s α, was usually above the standard threshold of .70; however, 16 instruments contained one or more scales that fell within the range of .52–.69. Interrater reliability and test–retest reliability were rarely measured.

Practical Considerations

Most of the instruments were moderate in length (between 20 and 40 items; n = 20), but several were also very short (less than 10 items; n = 7), short (less than 20 items; n = 14), lengthy (between 41 and 60 items; n = 11), or very lengthy (more than 60 items; n = 6). Response rates varied, but the majority were well above 60%, which may indicate that, in general, the instruments are easy to understand and complete.

Applicability to Integrated Care Interventions

The instruments were most often tested in hospital settings, though a few studies examined instruments in mental health, rehabilitation, or home care settings. The most common respondents consisted of nurses, physicians, and leaders/managers as well as members of multidisciplinary clinical teams. Only nine instruments require minor modifications to the wording of some items to enhance applicability across multiple health-care settings (e.g., replacing the term “hospital” with “organization” and “physician” with “clinician” or the insertion of specific professional groups or information systems).

Although all of the identified instruments may be applicable in assessing integrated care, only eight of the identified instruments have been used or explicitly recommended for use with integrated care interventions. The Organizational Culture Assessment Instrument (Zammuto & Krakower, 1991) was used by Maharaj (2010) to examine how organizational culture influenced the implementation of a care model aimed at integrating mental illness and substance use services. The Learning Organization Survey (Garvin, Edmondson, & Gino, 2008; Singer, Moore, Meterko, & Williams, 2012) was recommended for use by Mechanic, Perloff, Zinner, Razavi, and Keane (2012) in a report describing an evaluation plan for assessing health system reforms in the United States aimed, in part, at coordinating care and improving integrated delivery systems. The Team Climate Inventory Short-Form (Kivimaki & Elovainio, 1999) was used by Gibbon et al. (2002) to examine the impact of integrated care pathways and team notes on multidisciplinary teamwork in stroke care and rehabilitation and by Ouwens et al. (2009) to examine team climate before and after the implementation of an integrated care program for patients with head and neck cancers. The Relational Coordination Survey (Gittell, 2002) and the PSAT (Weiss et al., 2002) were used by Cramm and Nieboer (2012) to evaluate 19 chronic disease management programs that linked multidisciplinary professionals as well as primary care practices and hospitals. The PSAT was also recommended by Bainbridge, Brazil, Krueger, Ploeg, and Taniguchi (2010) to evaluate integrated palliative care. The Quality Improvement Implementation Survey II (Shortell et al., 2000) was used by Strandberg-Larsen et al. (2010) to compare levels of clinical integration in Kaiser Permanente Northern California and the Danish health-care system. The SOAPC (Ohman-Strickland et al., 2007) was used to evaluate two health promotion interventions in Spain involving interprofessional and interorganizational collaboration (Sanchez et al., 2009). Finally, the CPSET (Vanhaecht et al., 2007) was used by Vanhaecht et al. (2010) to examine the impact of clinical pathways on the organization of care for patients undergoing total joint replacement.

Discussion

This study identified 58 instruments that are currently available to assess organizational capabilities for integrated care delivery. Only eight of these instruments have been used, or recommended for use, with integrated care interventions. The eight instruments are focused primarily on organizational processes, such as teamwork, improving quality, and partnering, and were used in interventions focused on integrating specific sets of services associated with a single condition (e.g., cancer) or care sector (e.g., mental health) and mostly within as opposed to across organizational boundaries. These findings support our assessment that limited attention has been given to measuring organizational capabilities in empirical research on integrated care. Instead, the focus tends to be on understanding and measuring progress toward integrated care delivery in particular interventions rather than underlying organizational capabilities and conditions. In order to explain the effectiveness of integrated care interventions, explicit measures of the outcomes of integrated care must be supplemented by measures of organizational capabilities for integrated care including senior leadership style, organizational learning, work climate, and readiness for change, among others. Despite widespread recognition of the influence of these factors on the success of integrated care initiatives (Friedman & Goes, 2001; Ham & Walsh, 2013; Hartgerink et al., 2013; Ling et al., 2012; Martin-Rodriguez, Beaulieu, D’Amour, & Ferrada-Videla, 2005; Shortell, Gillies, Anderson, Mitchell, & Morgan, 1993; Suter et al., 2009; Williams & Sullivan, 2009), evidence is lacking on when and how these factors matter. Research needs to move beyond general statements about variations in the performance of integrated care interventions being due, for example, to “culture” or “leadership,” to more specific assessments of these capabilities. For example, which aspects of leadership are important at each stage of implementation and by which leaders (e.g., administrative or clinical)?

Collecting data on organizational capabilities as part of the planning, implementation, and/or evaluation stages of integrated care interventions could yield new insights on their outcomes and performance. Differing internal and collective capabilities among the organizations involved in integrating care may help explain the mixed outcomes reported in the literature (Bardsley et al., 2013; Curry & Ham, 2010). In practice, issues and challenges related to organizational capabilities, in particular to the compatibility and alignment (or lack thereof) among partnering organizations, are identified informally and only after considerable time and resources have been invested in implementation and change management (Burns & Pauly, 2002; Williams & Sullivan, 2009). An understanding of organizational capabilities early on in the implementation process across diverse organizations and practices may inform partner selection, the development of shared understandings of strengths and weaknesses among partners, and the design of targeted change management strategies.

Building a more robust evidence base on integrated care to inform practice requires the use of common instruments and consistent data collection methods across integrated care settings, interventions, and studies. Future research should explore links between the organizational capabilities in the CCIC Framework and integrated care delivery using some of the identified instruments. In order to generate sample sizes large enough to detect significant differences, the instruments can be used to promote standardization and improve evaluation in system-wide transformation efforts involving integrated care, such as the Health Links in Ontario, Canada, or the Integrated Care Pioneers Program in England as well as in U.S. demonstration projects involving integrated care, such as Accountable Care Organizations, Patient-Centered Medical Homes, or the Program for All-Inclusive Care for the Elderly.

This review provides a bank of validated survey instruments for researchers and practitioners to select from. A contingency perspective may assist in comparing and selecting instruments for use when critical constructs are identified a priori. Some instruments focus on a single construct, while others measure multiple constructs from the CCIC Framework, as Table 2 demonstrates. Focused instruments are useful for in-depth analysis of a specific area previously identified as important. Broader instruments, on the other hand, may be used as diagnostic tools to identify strengths and weaknesses across multiple organizational capabilities. Furthermore, some instruments may be particularly well suited for use in practice given their length and low cognitive complexity, while others with more depth, breadth, or theoretical grounding may be preferred for use in research studies. Finally, the instruments may be used at one point in time or at multiple points over time. However, multiple administrations may erode the sensitivity of some instruments.

Previous papers have attempted to systematically identify, compare, and recommend instruments for use in research and practice. However, these assessments tend to focus on specific capabilities, such as organizational culture or teamwork, and comprehensive up-to-date assessments are not available for all relevant factors. A notable exception is the recent work by Brennan et al. (2012, 2013) in which the authors systematically collated, categorized, and reviewed quantitative self-report instruments measuring organizational, process, and team-level and individual-level factors thought to influence the success of continuous quality improvement interventions. The latter results do not adequately address issues related to the spanning of interventions across diverse organizations applicable to integrated care interventions. Furthermore, despite some overlap in the inclusion of constructs such as teamwork, quality improvement, and readiness for change, the CCIC Framework incorporates additional constructs such as governance, focus on patient centeredness and engagement, delivering care, and information technology, among others.

This review also complements and extends Project INTEGRATE, a European initiative which involved the development of a common methodological framework and set of tools for evaluating and comparing integrated care interventions (Cash-Gibson & Rosenmoller, 2014). Their data collection tool kit consists of a case study report template that incorporates guidelines for retrospective process and data analysis as well as semistructured interviews with stakeholders. Their methods focus primarily on describing integrated care initiatives and their implementation. A section on the context for integrated care is also included in which five broad, open-ended questions are provided to aid in the identification of key enablers and barriers. The present review identifies validated quantitative survey instruments that may be implemented alongside the qualitative and secondary analysis methods proposed by Project INTEGRATE.

This study has some limitations. The instrument search strategy was based on the CCIC Framework and thus omits literature on topics not included in the framework. Furthermore, conducting a comprehensive and rigorous review of instruments is challenged by our focus on numerous constructs, many of which represent rich bodies of academic literature. As a result, it is possible that relevant papers and instruments were missed. The potential for inadvertently excluding applicable instruments is exacerbated by our search methods which involved using two databases, screening only the first 100–350 papers ordered by relevance, limiting our search terms to “measurement tool” and “measurement instrument,” and including instruments published in the academic literature and used in a health-care setting. However, our scoping searches suggest good coverage using Medline and Google Scholar, and a focus on health care ensures that the instruments reviewed can be readily applied to integrated care initiatives. We also expect that the expert consultation, while limited to selected experts in Canada, minimized the risk of missing relevant instruments. Nevertheless, some relevant instruments were ultimately excluded because we were unable to obtain a copy of the instrument for assessment.

Conclusion

This study provides a bank of psychometrically sound survey instruments to describe and compare a range of organizational capabilities. The results can be used to standardize data collection on organizational capabilities across integrated care interventions, settings, and studies. The use of standardized survey instruments will encourage comparison across both studies and settings, and facilitate longitudinal data collection to help inform our understanding of the influence of internal and collective organizational capabilities on performance and how these capabilities evolve over time. The application of standardized survey instruments across organizations and practices involved in an integrated care intervention can also inform change management efforts and possibly improve integrated care delivery as well as sustainability. Additional research is needed to build an evidence base for many of the tools reported here, validate their use in integrated care settings, test their sensitivity to changes in organizational capabilities over time, and explore the associations between organizational capabilities and various measures of integrated care processes and outcomes.

As policy interest in integrated care continues to grow, and with it demands for evidence of improved outcomes, providers and organizations are under increasing pressure to identify strategies to enhance the probability that their integrated care interventions will succeed. In this article, we support explicit attention to standardized measurement of organizational capabilities for the implementation, management, performance, and sustainability of integrated care interventions.

Footnotes

Authors’ Note

The views expressed in this article are those of the authors and do not necessarily reflect those of the funders.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by grants from the Canadian Institutes of Health Research (CIHR, funding reference number TTF-128263) and from the Ontario Ministry of Health and Long-term Care Health System Performance Research Network (HSPRN, Grant # 06034).

Supplemental Material

The online appendices are available at .

References

Aarons

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6, 61–74.

Aarons

Ehrhart

Farahnak

(2014). The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science, 9, 45. doi:10.1186/1748-5908-9-45

Aarons

McDonald

Sheehan

Walrath-Greene

(2007). Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale (EBPAS) in a geographically diverse sample of community mental health providers. Administration and Policy in Mental Health and Mental Health Services Research, 34, 465–469.

Anderson

N. R.

West

M. A.

(1998). Measuring climate for work group innovation: Development and validation of the team climate inventory. Journal of Organizational Behavior, 19, 235–258.

Bainbridge

Brazil

Krueger

Ploeg

Taniguchi

(2010). A proposed systems approach to the evaluation of integrated palliative care. BMC Palliative Care, 9, 8. doi:10.1186/1472-684X-9-8

Bardsley

Steventon

Smith

Dixon

(2013). Evaluating integrated and community based care: How do we know what works? London, England: Nuffield Trust.

Bass

Avolio

(1990). The implications of transactional and transformational leadership for individual, team and organizational development. Research in Organizational Change and Development, 4, 231–272.

Batorowicz

Shepherd

T. A.

(2008). Measuring the quality of transdisciplinary teams. Journal of Interprofessional Care, 22, 612–620.

Benzer

J. K.

Young

Stolzmann

Osatuke

Meterko

Caso

… Mohr

D. C.

(2011). The relationship between organizational climate and quality of chronic disease management. Health Services Research, 46, 691–711.

10.

Bouckenooghe

Devos

Van den Broeck

(2009). Organizational change questionnaire–climate of change, processes, and readiness: Development of a new instrument. Journal of Psychology, 143, 559–599.

11.

Brennan

Bosch

Buchan

Green

(2012). Measuring organizational and individual factors thought to influence the success of quality improvement in primary care: A systematic review of instruments. Implementation Science, 7, 121. doi:10.1186/1748-5908-7-121

12.

Brennan

Bosch

Buchan

Green

(2013). Measuring team factors thought to influence the success of quality improvement in primary care: A systematic review of instruments. Implementation Science, 8, 20. doi:10.1186/1748-5908-8-20

13.

Bronstein

L. R.

(2002). Index of interdisciplinary collaboration [instrument development]. Social Work Research, 26, 113.

14.

Burns

Pauly

(2002). Integrated delivery networks: A detour on the road to integrated health care? Health Affairs, 21, 128–143.

15.

Burns

Walston

Alexander

Zuckerman

Anderson

Torrens

Hilberman

(2001). Just how integrated are integrated delivery systems? Results from a national survey. Health Care Management Review, 26, 20–39.

16.

Caldwell

D. F.

O’Reilly

C. A.

(2003). The determinants of team-based innovation in organizations: The role of social influence. Small Group Research, 34, 497–517.

17.

Cameron

Quinn

(2011). Diagnosing and changing organizational culture (3rd ed.). San Francisco, CA: Jossey-Bass.

18.

Campion

Medsker

Higgs

(1993). Relations between work group characteristics and effectiveness: Implications for designing effective work groups. Personnel Psychology, 46, 823–847.

19.

Cash-Gibson

Rosenmoller

(2014). Project INTEGRATE—A common methodological approach to understand integrated health care in Europe. International Journal of Integrated Care, 14, e035.

20.

Costa

Anderson

(2011). Measuring trust in teams: Development and validation of a multifaceted measure of formative and reflective indicators of team trust. European Journal of Work and Organizational Psychology, 20, 119–154.

21.

Cramm

J. M.

Nieboer

A. P.

(2012). THE CARE SPAN: In the Netherlands, rich interaction among professionals conducting disease management led to better chronic care. Health Affairs, 31, 2493–2500.

22.

Cramm

J. M.

Strating

Nieboer

(2011). Development and validation of a short version of the Partnership Self-Assessment Tool (PSAT) among professionals in Dutch disease-management partnerships. BMC Research Notes, 4, 224. doi:10.1186/1756-0500-4-224

23.

Curry

Ham

(2010). Clinical and service integration. The route to improved outcomes. London, England: The King’s Fund.

24.

Dawson

J. F.

González-Romá

Davis

West

M. A.

(2008). Organizational climate and climate strength in UK hospitals. European Journal of Work and Organizational Psychology, 17, 89–111.

25.

Edmondson

(1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44, 350–383.

26.

Edwards

J. R.

Knight

D. K.

Broome

K. M.

Flynn

P. M.

(2010). The development and validation of a transformational leadership survey for substance use treatment programs. Substance Use & Misuse, 45, 1279–1302.

27.

Estabrooks

C. A.

Squires

Cummings

Birdsell

Norton

(2009). Development and assessment of the Alberta Context Tool. BMC Health Services Research, 9, 234. doi:10.1186/1472-6963-9-234

28.

Evans

J. M.

Grudniewicz

Baker

G. R.

Wodchis

W. P.

(2016). Organizational context and capabilities for integrating care: A framework for improvement. International Journal of Integrated Care, 16, e2416.

29.

Friedman

Goes

(2001). Why integrated health networks have failed. Frontiers of Health Services Management, 17, 3–28.

30.

Garcia-Smith

Effken

J. A.

(2013). Development and initial evaluation of the clinical information systems success model (CISSM). International Journal of Medical Informatics, 82, 539–552.

31.

Garvin

Edmondson

Gino

(2008). Is yours a learning organization? Harvard Business Review, 86, 109–116.

32.

Gauld

Horsburgh

Brown

(2011). The clinical governance development index: Results from a New Zealand study. BMJ Quality & Safety, 10. doi:10.1136/bmjqs.2011.051482

33.

Gibbon

Watkins

Barer

Waters

Davies

Lightbody

Leathley

(2002). Can staff attitudes to team working in stroke care be improved? Journal of Advanced Nursing, 40, 105–111.

34.

Gillies

Shortell

Anderson

Mitchell

Morgan

(1993). Conceptualizing and measuring integration: Findings from the health system integration study. Hospital and Health Services Administration, 38, 467–489.

35.

Gittell

(2002). Coordinating mechanisms in care provider groups: Relational coordination as a mediator and input uncertainty as a moderator of performance effects. Management Science, 48, 1408–1426.

36.

Glisson

Landsverk

Schoenwald

Kelleher

Hoagwood

Mayberg

, … Research Network on Youth Mental Health. (2008). Assessing the Organizational Social Context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health, 35, 98–113.

37.

Goodwin

Dixon

Anderson

Wodchis

(2014). Providing integrated care for older adults with complex needs: Lessons from seven international case studies. London, England: The King’s Fund.

38.

Gugerty

Maranda

Rook

(2006). The clinical information system implementation evaluation scale. Studies in Health Technology and Informatics, 122, 621.

39.

Ham

Walsh

(2013). Making integrated care happen at scale and pace (lessons from experience). London, England: The King’s Fund.

40.

Hartgerink

Cramm

de Vos

Bakker

Steyerberg

Mackenbach

Nieboer

A. P.

(2013). Situational awareness, relational coordination and integrated care delivery to hospitalized elderly in the Netherlands: A comparison between hospitals. BMC Geriatrics, 14. doi:10.1186/1471-2318-14-3

41.

Helfrich

C. D.

Y. F.

Sharp

N. D.

Sales

A. E.

(2009). Organizational readiness to change assessment (ORCA): Development of an instrument based on the Promoting Action on Research in Health Services (PARIHS) framework. Implementation Science, 4, 38. doi:10.1186/1748-5908-4-38

42.

Herscovitch

Meyer

J. P.

(2002). Commitment to organizational change: Extension of a three-component model. Journal of Applied Psychology, 87, 474.

43.

Holt

D. T.

Armenakis

A. A.

Feild

H. S.

Harris

S. G.

(2007). Readiness for organizational change: The systematic development of a scale. Journal of Applied Behavioral Science, 43, 232–255.

44.

Johri

Beland

Bergman

(2003). International experiments in integrated care for the elderly: A synthesis of the evidence. International Journal of Geriatric Psychiatry, 18, 222–235.

45.

Joly

B. M.

Booth

Mittal

Shaler

(2012). Measuring quality improvement in public health: The development and psychometric testing of a QI maturity tool. Evaluation & the Health Professions, 35, 119–147.

46.

Jones

Barry

M. M.

(2011a). Developing a scale to measure synergy in health promotion partnerships. Global Health Promotion, 18, 36–44.

47.

Jones

Barry

M. M.

(2011b). Developing a scale to measure trust in health promotion partnerships. Health Promotion International, 26, 484–491.

48.

Kaplan

Brady

Britz

Hooper

Linam

Froehle

Margolis

(2010). The influence of context on quality improvement success in health care: A systematic review of the literature. Milbank Quarterly, 88, 500–559.

49.

Karasek

Brisson

Kawakami

Houtman

Bongers

Amick

(1998). The Job Content Questionnaire (JCQ): An instrument for internationally comparative assessments of psychosocial job characteristics. Journal of Occupational Health Psychology, 3, 322.

50.

Kautz

Gittell

Weinberg

Lusenhop

Wright

(2007). Patient benefits from participating in an integrated delivery system: Impact on coordination of care. Health Care Management Review, 32, 284–294.

51.

Kenaszchuk

Reeves

Nicholas

Zwarenstein

(2010). Validity and reliability of a multiple-group measurement scale for interprofessional collaboration. BMC Health Services Research, 10, 83. doi:10.1186/1472-6963-10-83

52.

Kivimaki

Elovainio

(1999). A short version of the Team Climate Inventory: Development and psychometric properties. Journal of Occupational and Organizational Psychology, 72, 241–246.

53.

Korner

Wirtz

M. A.

(2013). Development and psychometric properties of a scale for measuring internal participation from a patient and health care professional perspective. BMC Health Services Research, 13, 374. doi:10.1186/1472-6963-13-374

54.

Kralewski

Dowd

B. E.

Kaissi

Curoe

Rockwood

(2005). Measuring the culture of medical group practices. Health Care Management Review, 30, 184–193.

55.

Kreindler

Larson

Carluzzo

Gbemudu

Struthers

… Fisher

S. (2012). Interpretations of integration in early accountable care organizations. Milbank Quarterly, 90, 457–483.

56.

Kristensen

T. S.

Borritz

Villadsen

Christensen

K. B.

(2005). The Copenhagen Burnout Inventory: A new tool for the assessment of burnout. Work & Stress, 19, 192–207.

57.

Lemieux-Charles

Murray

Baker

G. R.

Barnsley

Tasa

Ibrahim

S. A.

(2002). The effects of quality improvement practices on team effectiveness: A mediational model. Journal of Organizational Behavior, 23, 533–553.

58.

Levac

Colquhoun

O’Brien

(2010). Scoping studies: Advancing the methodology. Implementation Science, 5, 69. doi:10.1186/1748-5908-5-69

59.

Lewis

(2003). Measuring transactive memory systems in the field: Scale development and validation. Journal of Applied Psychology, 88, 587–604.

60.

Lichtenstein

Alexander

J. A.

Jinnett

Ullman

(1997). Embedded intergroup relations in interdisciplinary teams effects on perceptions of level of team integration. Journal of Applied Behavioral Science, 33, 413–434.

61.

Lin

Wan

(1999). Analysis of integrated health care networks’ performance: A contingency-strategic management perspective. Journal of Medical Systems, 23, 467–485.

62.

Ling

Brereton

Conklin

Newbould

Roland

(2012). Barriers and facilitators to integrating care: Experiences from the English Integrated Care Pilots. International Journal of Integrated Care, 12, e129.

63.

Low

Yap

Brodaty

(2011). A systematic review of different models of home and community care services for older persons. BMC Health Services Research, 11, 93. doi:10.1186/1472-6963-11-93

64.

Maharaj

. (2010). Organizational culture, absorptive capacity, and the change process: Influences on the fidelity of implementation of integrated dual disorder treatment in community-based mental health organizations (Dissertation). Retrieved from http://aladinrc.wrlc.org/handle/1961/9209

65.

Martin-Rodriguez

L. S.

Beaulieu

D’Amour

Ferrada-Videla

(2005). The determinants of successful collaboration: A review of theoretical and empirical studies. Journal of Interprofessional Care, 19, 132–147.

66.

McCormack

McCarthy

Wright

Coffey

(2009). Development and testing of the Context Assessment Index (CAI). Worldviews on Evidence-Based Nursing, 6, 27–35.

67.

Mechanic

Perloff

Zinner

Razavi

Keane

(2012). Evaluation of HHS delivery system reform efforts and Affordable Care Act provisions: Consolidated evaluation design recommendations. Retrieved from http://aspe.hhs.gov/health/reports/2013/EvaluationOfHHsDelivery/rpt_evaluationDelivery.pdf

68.

Mikkelsen

Grønhaug

(1999). Measuring organizational learning climate: A cross-national replication and instrument validation study among public sector employees. Review of Public Personnel Administration, 19, 31–44.

69.

Millward

Ramsay

(1998). Measuring team performance from a cognitive and motivational perspective: A pilot study of an evaluation tool. Guildford, England: Centre for Employee Research, University of Surrey.

70.

Minvielle

Sicotte

Champagne

Contandriopoulos

A. P.

Jeantet

Préaubert

… Richard

(2008). Hospital performance: Competing or shared values? Health Policy, 87, 8–19.

71.

Moorman

R. H.

(1991). Relationship between organizational justice and organizational citizenship behaviors: Do fairness perceptions influence employee citizenship? Journal of Applied Psychology, 76, 845.

72.

Mowday

R. T.

Steers

R. M.

Porter

L. W.

(1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14, 224–247.

73.

Nuno-Solinis

Zabalegui

I. B.

Arce

R. S.

Rodríguez

L. S. M.

Polanco

N. T.

(2013). Development of a questionnaire to assess interprofessional collaboration between two different care levels. International Journal of Integrated Care, 13, e015.

74.

Ohman-Strickland

P. A.

John Orzano

Nutting

P. A.

Perry Dickinson

Scott-Cawiezell

Hahn

… Crabtree

B. F.

(2007). Measuring organizational attributes of primary care practices: Development of a new instrument. Health Services Research, 42, 1257–1273.

75.

Orchard

C. A.

King

G. A.

Khalili

Bezzina

M. B.

(2012). Assessment of interprofessional team collaboration scale (AITCS): Development and testing of the instrument. Journal of Continuing Education in the Health Professions, 32, 58–67.

76.

Oreg

(2003). Resistance to change: Developing an individual differences measure. Journal of Applied Psychology, 88, 680.

77.

Ouwens

Hermens

Hulscher

Merkx

van den Hoogen

Grol

… Marres

H. A.

(2009). Impact of an integrated care program for patients with head and neck cancer on the quality of care. Head and Neck, 31, 902–910.

78.

Pare

Sicotte

Poba-Nzaou

Balouzakis

(2011). Clinicians’ perceptions of organizational readiness for change in the context of clinical information system projects: Insights from two cross-sectional surveys. Implementation Science, 6, 15. doi:10.1186/1748-5908-6-15

79.

Patterson

M. G.

West

M. A.

Shackleton

V. J.

Dawson

J. F.

Lawthom

Maitlis

… Wallace

A. M.

(2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior, 26, 379–408.

80.

Petticrew

Roberts

(2005). Systematic reviews in the social sciences: A practical guide. London, England: Blackwell.

81.

Pham

(2010). Good neighbors: How will the patient-centered medical home relate to the rest of the health-care delivery system? Journal of General Internal Medicine, 25, 630–634.

82.

Porter

Steers

Mowday

Boullian

(1974). Organizational commitment, job satisfaction and turnover among psychiatric technicians. Journal of Applied Psychology, 59, 603–609.

83.

Sanchez

Grandes

Cortada

Pombo

Balague

Calderon

(2009). Modelling innovative interventions for optimising healthylifestyle promotion in primary health care: “Prescribe Vida Saludable” phase I research protocol. BMC Health Services Research, 9, 103. doi:10.1186/1472-6963-9-103

84.

Schroder

Medves

Paterson

Byrnes

Chapman

O’Riordan

… Kelly

(2011). Development and pilot testing of the collaborative practice assessment tool. Journal of Interprofessional Care, 25, 189–195.

85.

Scott

Mannion

Davies

Marshall

(2003). The quantitative measurement of organizational culture in health care: A review of the available instruments. Health Services Research, 38, 923–945.

86.

Seys

Deneckere

Sermeus

Van Gerven

Panella

Bruyneel

… Vanhaecht

(2013). The Care Process Self-Evaluation Tool: A valid and reliable instrument for measuring care process organization of health care teams. BMC Health Services Research, 13, 325. doi:10.1186/1472-6963-13-325

87.

Shortell

S. M.

Gillies

Anderson

Mitchell

Morgan

(1993). Creating organized delivery systems: The barriers and facilitators. Hospital and Health Services Administration, 38, 447–466.

88.

Shortell

S. M.

O’Brien

J. L.

Carman

J. M.

Foster

R. W.

Hughes

E. F.

Boerstler

O’Connor

E. J.

(1995). Assessing the impact of continuous quality improvement/total quality management: Concept versus implementation. Health Services Research, 30, 377.

89.

Shortell

S. M.

Jones

R. H.

Rademaker

A. W.

Gillies

R. R.

Dranove

D. S.

Hughes

E. F. X.

… Huang

C. F.

(2000). Assessing the impact of total quality management and organizational culture on multiple outcomes of care for coronary artery bypass graft surgery patients. Medical Care, 38, 207–217.

90.

Shortell

S. M.

Marsteller

Lin

Pearson

S.-Y.

Mendel

… Rosen

(2004). The role of perceived team effectiveness in improving chronic illness care. Medical Care, 42, 1040.

91.

Singer

S. J.

Burgers

Friedberg

Rosenthal

Leape

Schneider

(2011). Defining and measuring integrated patient care: Promoting the next frontier in health care delivery. Medical Care Research and Review, 68, 112–127.

92.

Singer

S. J.

Moore

S. C.

Meterko

Williams

(2012). Development of a short-form learning organization survey: The LOS-27. Medical Care Research and Review, 69, 432–459. doi:10.1177/1077558712448135

93.

Solberg

L. I.

Asche

S. E.

Shortell

S. M.

Gillies

R. R.

Taylor

Pawlson

L. G.

… Young

M. R.

(2009). Is integration in large medical groups associated with quality? American Journal of Managed Care, 15, e34–e41.

94.

Spreitzer

G. M.

(1995). Psychological empowerment in the workplace: Dimensions, measurement, and validation. Academy of Management Journal, 38, 1442–1465.

95.

Stamatakis

K. A.

McQueen

Filler

Boland

Dreisinger

Brownson

R. C.

Luke

D. A.

(2012). Measurement properties of a novel survey to assess stages of organizational readiness for evidence-based interventions in community chronic disease prevention settings. Implementation Science, 7, 65. doi:10.1186/1748-5908-7-65

96.

Stewart

Georgiou

Westbrook

(2013). Successfully integrating aged care services: A review of the evidence and tools emerging from a long-term care program. International Journal of Integrated Care, 13, e003.

97.

Strandberg-Larsen

Schiotz

Silver

Frolich

Andersen

Graetz

… Hsu

(2010). Is the Kaiser Permanente model superior in terms of clinical integration? A comparative study of Kaiser Permanente, Northern California and the Danish healthcare system. BMC Health Services Research, 10, 91. doi:10.1186/1472-6963-10-91

98.

Suter

Oelke

Adair

Armitage

(2009). Ten key principles for successful health systems integration. Healthcare Quarterly, 13, 16–23.

99.

Teece

D. J.

Pisano

Shuen

(1990). Firm capabilities, resources, and the concept of strategy (CCC Working Paper 90-8). Berkeley, CA: Center for Research in Management.

100.

Vanhaecht

Bellemans

De Witte

Diya

Lesaffre

Sermeus

(2010). Does the organization of care processes affect outcomes in patients undergoing total joint replacement? Journal of Evaluation in Clinical Practice, 16, 121–128.

101.

Vanhaecht

De Witte

Depreitere

Van Zelm

De Bleser

Proost

Sermeus

(2007). Development and validation of a Care Process Self-Evaluation Tool. Health Services Management Research, 20, 189–202.

102.

Wagner

De Bakker

D. H.

Groenewegen

P. P.

(1999). A measuring instrument for evaluation of quality systems. International Journal for Quality in Health Care, 11, 119–130.

103.

Wan

Lin

(2001). Integration and the performance of healthcare networks: Do integration strategies enhance efficiency, profitability, and image? International Journal of Integrated Care, 1, e39.

104.

Watson

Clarke

Swallow

Forster

(2005). Exploratory factor analysis of the research and development culture index among qualified nurses. Journal of Clinical Nursing, 14, 1042–1047.

105.

Weiss

Anderson

Lasker

(2002). Making the most of collaboration: Exploring the relationship between partnership synergy and partnership functioning. Health Education & Behavior, 29, 683–698.

106.

Williams

Sullivan

(2009). Faces of integration. International Journal of Integrated Care, 9. Retrieved from http://www.ijic.org/index.php/ijic/article/view/509/1016

107.

Wilkens

London

(2006). Relationships between climate, process, and performance in continuous quality improvement groups. Journal of Vocational Behavior, 69, 510–523.

108.

Wodchis

W. P.

Dixon

Anderson

Goodwin

(2015). Integrating care for older people with complex needs: Key insights and lessons from a seven-country cross-case analysis. International Journal of Integrated Care, 15, e021.

109.

Woolf

S. H.

Johnson

R. E.

(2005). The break-even point: When medical advances are less important than improving the fidelity with which they are delivered. Annals of Famiy Medicine, 3, 545–552.

110.

Zammuto

Krakower

(1991). Quantitative and qualitative studies of organizational culture. In Pasmore

W. A.

Woodman

R. W.

(Eds.), Research in organizational change and development (Vol. 5, pp. 83–114). Greenwich, CT: JAI Press.