The Implementation of Performance Management Systems in U.S. Physician Organizations

Abstract

Performance management systems (PMSYS) aid in improving the quality and efficiency of care, but little is known about factors that influence more robust PMSYS among physician organizations. Using a nationally representative survey of U.S. medical practices, we examined the extent to which organizational capabilities and external factors were associated with more developed PMSYS. Linear regression estimated the relative impact of these factors on PMSYS. On average, practices implemented a minority (32 points out of 100) of the PMSYS processes assessed. Practices evaluated (p < .01) or financially incentivized by external entities (p < .01), receiving data from health plans (p < .01), participating in an accountable care organization (p < .01), affiliating with an independent practice association and/or physician–hospital organization (p < .01), and using health information technology (p < .01) and chronic disease registries (p < .01) to greater degrees had more robust PMSYS. PMSYS of medical practices are underdeveloped, although both external incentives and organizational capabilities may support PMSYS development.

Keywords

performance management systems external incentives health information technology accountable care organizations physician organizations

Introduction

The measurement and management of physician performance has been high on the agenda of policy makers, payers, and consumer groups for decades. Wennberg et al. (Fisher et al., 2003; Wennberg, Freeman, Shelton, & Bubolz, 1989) discovered considerable small area variation in utilization of services, expenditures, and quality and outcomes of care and the Institute of Medicine (2001) highlighted major deficits in care and underscored the central importance of performance measurement to improving quality and outcomes. More recently, the Affordable Care Act in 2010 and the Medicare Access and CHIP Reauthorization Act of 2015 aim to foster physician performance measurement through incentivizing practice transitions from fee-for-service reimbursement to performance and value-based payment models, such as the Merit-Based Incentive Payment System and Alternative Payment Models (Centers for Medicare & Medicaid Services [CMS], 2015).

Studies indicate a positive association between external reporting requirements, such as evaluation by external entities, public reporting or pay for performance (P4P), and the clinical and operational performance of physician organizations (Casalino et al., 2003; Shortell et al., 2005), providing some empirical support for the positive impact of external incentives. How to effectively use external incentives to cultivate performance measurement and improvement, however, is less clear. Reservations about the proliferation of performance measures have been expressed because clinicians can become overloaded with different, nonoverlapping measures required by payers and other stakeholders, leading to measurement fatigue (Cassel et al., 2014; Higgins, Veselovskiy, & McKown, 2013; Meyer et al., 2012) and high measurement costs (Casalino et al., 2016). Because it is difficult to measure important aspects of care, the usefulness of P4P programs—especially programs that require individual physician-level performance measurement (Campbell, Reeves, Kontopantelis, Sibbald, & Roland, 2009; Cassel & Jain, 2012)—has been questioned. In particular, the lack of evidence-based measures for many specialties, sample size constraints, and attribution problems which have raised concerns about the potential adverse effects of measuring and incentivizing performance (Chien & Rosenthal, 2013; Hofer et al., 1999; Landon, Normand, Blumenthal, & Daley, 2003).

The extent to which external performance measurement requirements compete with the development of internal performance management systems (PMSYS) remains unclear. Shortell et al. (2005) hypothesized a positive relationship between external reporting requirements and organizational performance because organizations involved in external incentive programs have the ability to make performance comparisons that would motivate internal improvement efforts. Others argue that external reporting requirements (Meyer et al., 2012) can distract from the development and use of measures for internal performance management (Farmer, Black, & Bonow, 2013; Landon et al., 2003). Importantly, linking external performance measures to financial incentives may result in gaming strategies, such as upcoding of desirable conduct (Haut, Pronovost, & Schneider, 2012), downcoding of adverse events (McNair, Luft, & Bindman, 2009; Wright, 2012) or “sham” improvements (Meyer et al., 2012). External measurement and incentive programs can also result in focusing on the wrong areas for improvement, as the external measures may not be specific to the needs of an individual organization or patient served (Meyer et al., 2012) and pose a risk to dilute physicians’ intrinsic motivation (Cassel & Jain, 2012; Petersen, Woodard, Urech, Daw, & Sookanan, 2006).

In spite of these challenges, the use of performance measures to internally stimulate improvement and manage performance is emphasized as a crucial success factor of high-value health care organizations (Bohmer, 2011; Mechanic & Zinner, 2012; Porter, 2010; Wagner et al., 2001). Yet the current discussion of PMSYS is predominantly focused on satisfying external reporting requirements. Little is known about the extent to which U.S. physician organizations have implemented PMSYS internally, and which factors are associated with greater PMSYS implementation. We therefore focus on this issue in our study.

New Contributions

The distinct contributions of our research are threefold. First, we develop a conceptual model of the influence of external factors and organizational capabilities on the implementation of PMSYS and performance outcomes. Guided by this conceptual model, we construct a multi-item PMSYS index from the third National Survey of Physician Organizations (NSPO3), a nationally representative sample of nongovernmental U.S. physician organizations, including physician practices and medical groups (n = 1,398, response rate = 49.7%). Second, we shed light on the extent of implementation of internal PMSYS in a nationally representative sample of organizations, enabling a national assessment of the extent of PMSYS adoption in U.S. medical practices. Third, our study addresses the understudied issue of which factors facilitate and impeded the implementation of internal performance measurement and management practices in physician organizations. In addition to the influence of external reporting requirements and P4P, we emphasize delivery system reform strategies, including practice affiliation with accountable care organizations (ACOs; Davis et al., 2010; McClellan, McKethan, Lewis, Roski, & Fisher, 2010), and the promotion of “meaningful use” of health information technology (HIT) functionality (Blumenthal, 2011a, 2011b; Heisey-Grove, Danehy, Consolazio, Lynch, & Mostashari, 2014; Lynch et al., 2014). Because our study assesses PMSYS implementation in U.S. physician organizations and identifies factors related to more robust PMSYS, the findings have important implications for policy decision makers and health care managers aiming to support the development of PMSYS.

Conceptual Model

We define PMSYS as a “set of formalized, mostly quantifiable practices of reflexive control, that organizations use to improve efficiency and effectiveness” (Pimperl, 2015, p63). To develop our conceptual model of PMSYS implementation, we draw on conceptual models focused on the implementation of evidence-based care management developed by Rittenhouse et al. (2010) and Shortell et al. (2005; Figure 1) and previously published PMSYS-specific conceptual design and implementation criteria (Adair et al., 2003; Hudson, Smart, & Bourne, 2001; Neely, Gregory, & Platts, 1995; Pimperl, 2015). As depicted in Figure 1, high performance of physician organizations is driven by the robustness of their internal PMSYS (Bohmer, 2011; Mechanic & Zinner, 2012; Porter, 2010; Wagner et al., 2001). Core components of PMSYS are drawn from Adair et al. (2003), Hudson et al. (2001), Neely et al. (1995), and Pimperl (2015) and structured into two categories: (a) conceptual criteria for the design of the system of measures and (b) PMSYS process model criteria. Conceptual criteria include criteria for how PMSYS measures are designed, such as balanced and comprehensive, aligned with strategy, and simple to understand and use. The PMSYS process model criteria focus on the processes used to manage performance, including the involvement of key stakeholders, ensuring fast and accurate feedback, and organizationally integrating the PMSYS. The PMSYS robustness criteria are detailed in Figure 1. For our empirical study of PMSYS, we focus on a crucial subset of the PMSYS robustness criteria: the comprehensiveness of the set of measures used, the provision of feedback, the organizational integration of the PMSYS into the information technology (IT)-system (also a prerequisite for an efficient PMSYS), and the regular review, update, and facilitation of continuous quality improvement (QI), because these PMSYS features are central to U.S. physician organizations and can be reliably measured using surveys of organizational leaders (Marsden et al., 2006). We construct a multi-item PMSYS index from NSPO3 with four subcomponents addressing this subset of PMSYS criteria: (a) performance feedback for chronic conditions; (b) and/or for preventive services both([a and b] addressing the provision of feedback as well as indirectly the comprehensiveness of the set of measures criteria); (c) PMSYS integration into the (IT) system (addressing the organizational integration of PMSYS and indirectly the PMSYS efficiency criteria); and (d) regular review, update, and established continuous QI (Figure 1).

Figure 1.

Conceptual framework: The influence of organizational capabilities and external factors on the implementation of performance management systems (PMSYS) and performance outcomes.

We hypothesized that the implementation of PMSYS is affected by external factors and organizational capabilities.

External Influences on PMSYS Implementation

External factors have long been identified to have a crucial impact on organizations (Aldrich, 2008; Lawrence & Lorsch, 1967); especially in a strongly regulated environment as the health care market (Shortell et al., 2005). There is also specific evidence about the importance of the environment on the adoption of PMSYS (Franco-Santos & Bourne, 2005; Garengo, Biazzo, & Bititci, 2005; Hoque, 2004; Julnes & Holzer, 2001). We posited that external evaluation by health plans or other entities, P4P incentives and public reporting of performance data would be positively associated with the implementation of PMSYS among physician organizations, acknowledging the mixed evidence to date about this positive association (Farmer et al., 2013; Fung, Lim, Mattke, Damberg, & Shekelle, 2008; Landon et al., 2003; Meyer et al., 2012; Rittenhouse et al., 2010; Shortell et al., 2005; Werner & Asch, 2005; Wiley et al., 2015).

We also hypothesized a positive association of the provision of data by health plans and robust PMSYS, as these data may allow physician organizations to implement feedback reports independent from their own electronic health record (EHR) or chronic disease registries. Health plan data submission requirements can support the scope of practices’ performance management efforts, as health plan data may also include additional data, such as out-of-network utilization, that are missing in the physician organizations’ own EHR.

Furthermore, practice participation in available national or regional collaborative quality demonstration projects may also aid in implementing PMSYS (Nadeem, Olin, Hill, Hoagwood, & Horwitz, 2013) because these initiatives emphasize using data to drive performance improvement.

Finally, payer mix, the mix of racial and ethnic minority patients served, and regional influences may influence the adoption of PMSYS. Previous studies have found that practices with a large percentage of patients from minorities and/or a low-income background have lower payment levels and fewer resources to invest in the development of care management processes (Bach, Pham, Schrag, Tate, & Hargraves, 2004; Friedberg, Coltin, Safran, Dresser, & Schneider, 2010; Reschovsky & O’Malley, 2008; Varkey et al., 2009) and the adoption of PMSYS is likely more difficult for these practices.

Organizational Capabilities and PMSYS Implementation

Organizational capabilities affect how well an organization can capitalize on external incentives, resources, and opportunities (Aldrich, 2008; Barney, 1991; Peteraf, 1993). We distinguish measures of organizational capabilities using two categories: organizational characteristics and internal capabilities. Organizational characteristics represent general structural elements of the organization, including ACO, independent practice association (IPA), and physician–hospital organization (PHO) affiliation. These affiliations have been found to be associated with greater use of care management processes (Casalino et al., 2013; Nembhard, 2012; Rundall et al., 2002; Shortell et al., 2014) and may also support PMSYS implementation. Similarly, practices owned by hospital, health systems or health maintenance organizations, larger practices, and multispecialty practices may also have greater resources to invest in robust PMSYS and respond to external incentives (Rittenhouse et al., 2011; Wiley et al., 2015).

Internal capabilities are technologies or processes established by the physician organization, that we assumed would have an influence on the adoption of a PMSYS. HIT functionality can positively affect the use of care management processes and organizational performance (Buntin, Burke, Hoaglin, & Blumenthal, 2011; Casalino et al., 2003; Holroyd-Leduc, Lorenzetti, Straus, Sykes, & Quan, 2011; Kern et al., 2013). When developing a robust PMSYS, HIT functionality is critical because it provides the technological foundation for the integration, collection, and analysis of performance data, allowing to implement decision support systems, such as reminders. In addition, Cassel et al. (2014) highlight the importance of chronic disease registries as important data-collection infrastructure facilitating the implementation of PMSYS.

Method

Data and Sample

We used data from the third iteration (Wave 3) of the largest NSPO3. The sample, methods, and administration of NSPO3 is already detailed elsewhere (Rittenhouse, Casalino, Gillies, Shortell, & Lau, 2008; Rittenhouse et al., 2011; Wiley et al., 2015).

Briefly, NSPO3 was conducted with a lead physician or lead administrator of a nationally representative sample of physician practices and medical groups. Data on physician organization characteristics, chronic care management processes, ACO participation, and related variables were collected via a 40-minute phone survey from January 2012 to November 2013. Respondents were paid $200 for their participation (Wiley et al., 2015).

The survey sample was derived from respondents to two earlier surveys (Rittenhouse et al., 2008; Rittenhouse et al., 2011) and an additional random sample, from comprehensive listings of physician organizations available from the IMS Healthcare Organizational Services database as of May 2011, stratified by practice size, mix of specialties, and geographic location. The study focused on physician organizations caring for major chronic illnesses including asthma, coronary heart failure, depression, and diabetes. Thus, only organizations with a signiﬁcant proportion of primary care providers (family physicians, general internists, and general practitioners), cardiologists, endocrinologists, or pulmonologists were eligible for the study. Practices with fewer than 20 physicians were eligible to participate as long as 40% or more of their physicians were in these specialties. Practices with 20 or more physicians were eligible if at least 30% were in the focal specialties. Academic faculty practices and practices affiliated with governmental entities were excluded.

The adjusted response rate was 50% (Wiley et al., 2015). From the 1,398 responding practices, we excluded organizations with missing values in the measures used in this study, resulting in an analytic sample of 1,328 practices. Taking into account the complex sampling design of NSPO3, population ratio-adjusted weights were determined based on sampling probabilities with poststratification adjustments (Little, 1993; Wiley et al., 2015).

Measures

Outcome Measure: PMSYS Robustness Index

We constructed a composite PMSYS robustness index from 20 items of NSPO3, guided by a crucial subset of the PMSYS robustness criteria outlined in Figure 1. Using principal factors analysis, we identified—using the scree test (D’Agostino & Russell, 2005)—a four-factor solution that we aggregated by the arithmetic mean to the composite PMSYS index (range = 0-100 points). For ease of interpretation, we did not use the separate factor scores in calculating the composite PMSYS index, but a 0- to 100-point summary index. The PMSYS robustness index reliably demonstrated internal consistency (unweighted Cronbach’s [1951] α = .84), greater than .70, the accepted norm for assessing the internal consistency of indices (Boermans & Kattenberg, 2011).

The four subcomponents of the composite PMSYS index are as follows: (a) performance feedback for chronic conditions (α = .85); (b) performance feedback for preventive services (α = .71); (c) PMSYS integration into the IT system (α = .87); (d) regular review, update, and established continuous QI processes (α = .60). Three of four subindices have acceptable levels of internal consistency reliability (α > .70). Some evidence (Andersson, Eriksson, & Torstensson, 2006; Hughes, 2008; Salah, Rahim, & Carretero, 2010) suggests the complementary benefits of using multiple QI tools. The regular review, update, and established continuous QI processes subcomponent, however, had modest reliability (α = .60). Cronbach’s α between .50 and .70 are generally considered modest and a Cronbach’s α below .50 is generally considered too low to reliably construct a composite measure (Boermans & Kattenberg, 2011). As a result, we calculated the regular review, update, and established continuous QI processes subcomponent as a binary variable (100 points = 1 or more QI methods, 0 points = no QI) as opposed to using the arithmetic mean of items. The detailed calculation of the overall PMSYS index is detailed in Table 1.

Table 1.

Construction of the Performance Management Systems (PMSYS) Robustness Index Score From Specific NSPO3 Questionnaire Items.

PMSYS robustness index subcomponents	Question content/criteria	Scoring rules
a) Performance feedback for chronic conditions	Practice/IPA/PHO provides performance feedback to physicians on the quality of their care for patients with the following chronic conditions: • asthma • congestive heart failure • depression • diabetes	If “yes” to: 4 items = 100 3 items = 75 2 items = 50 1 item = 25
b) Performance feedback for preventive services	Practice/IPA/PHO provides performance feedback to physicians on the quality of their care for the following preventive services: • breast cancer screening • tobacco cessation interventions	If “yes” to: 2 items = 100 1 item = 50
c) PMSYS integration into the IT system	Practice uses the electronic medical record • to collect data for clinical quality measures • for potential drug interactions • for prompts and reminders • for alerts on abnormal test results	If “yes” to: 4 items = 100 3 items = 75 2 items = 50 1 item = 25
d) Regular review, update, and established continuous quality improvement process	Practice uses the following formal and systematic quality improvement system: • Plan-Do-Study-Act (PDSA) • Lean production techniques • Six Sigma • Any other formal and systematic quality improvement systems.	If “yes” to: 1 or more items = 100
Composite PMSYS Robustness Index (Subcomponent a + b + c + d)/4

Note. NSPO3 = third National Survey of Physician Organizations; IPA = independent practice association: PHO = physician–hospital organization.

External Factors

Our main independent external factors variables for this study are external evaluation by health plans or other entities, P4P incentives, public reporting of performance data, the provision of data by health plans, and QI collaborative. An external evaluation index (range: 0-2) was constructed as proposed by Rittenhouse et al. (2010), awarding an organization 1 point for clinical quality and 1 point for patient satisfaction evaluation by external entities such as health insurance plans. A P4P index and public reporting index was created based on prior work (Rittenhouse et al., 2010; Wiley et al., 2015). The P4P index (range: 0-3) accounts for the opportunity of the practice to receive additional income from external entities based on (a) clinical quality scores (Healthcare Effectiveness Data and Information Set), (b) use of IT, and (c) the efficient use of resources. The public reporting index (range: 0-2) was constructed for each practice based on whether their health plans publically report practice data on patient satisfaction and clinical quality. The provision of data by health plans was measured as dichotomous variable that assessed if health plans provide data to physicians on quality of preventive care and/or care for patients with chronic illness (yes/no). Practice participation in QI collaboratives (ever) was also included as dichotomous variable.

We also controlled for payer mix, racial/ethnic minority patient mix, and regional influences, which could potentially confound the estimated relationship of the main independent external variables and PMSYS implementation. Payer mix was measured as the proportion of patients who were insured by Medicaid or were low income and had no insurance. Minority patient mix included measures of the proportion of African American patients and the proportion of patients with limited English proficiency. In order to account for geographic variation in our regression models, we also included nine dummy variables capturing census regions as control variables.

Organizational Characteristics

Our main independent organizational variables are ACO participation and IPA or PHO affiliation, measured as dichotomous variables (yes/no). Taking into account the potentially confounding effect of practice ownership, practice size, and specialty mix, we also controlled for these variables in regression models.

Internal Capabilities

The main internal organizational capabilities variables that we hypothesized to be positively associated with more robust PMSYS were HIT functionality and the use of chronic disease registries. A HIT functionality index (range: 0-12; α = .83) was calculated for each organization, based on the sum of 12 dichotomous questions in regard to whether a majority of physicians use the EHR for patient’s medication, problem lists, progress notes, and to provide patients with clinical summaries. The index also included other HIT functions including electronic access to hospital discharge summaries and/or clinical information on patient emergency room visits at the main hospital, as well as access to lab results from the main lab and pharmacy record of prescriptions filled by patients, transmitting prescriptions directly to pharmacies, and whether a majority of physicians were communicating with patients via e-mail and/or allowed for patients to view their medical record online and/or change or update it. In addition, a dichotomous variable was constructed to denote use of chronic disease registries: A practice was awarded 1 point if they implemented registries for more than two out of the four chronic diseases, which included asthma, congestive heart failure, depression, and diabetes, registries in place.

Analysis

We first calculated the four mean subcomponent scores for the PMSYS robustness index and mean scores for the overall PMSYS index. We then conducted bivariate analyses using adjusted Wald tests for each independent and control variable, including external evaluation by health plans or other entities, P4P incentives, public reporting, ACO affiliation, HIT functionality, and so on.

Five multivariable regression models were estimated. The association between the overall PMSYS robustness index and external factors and organizational capabilities was analyzed using linear regression. The other four models used the four subcomponent indices of the composite PMSYS robustness measure as dependent variables. Linear regression was also used in these cases, with one exception: for the binary regular review, update, and established continuous QI processes subcomponent index a logistic regression model was applied (StataCorp, 2013).

All continuous variables were standardized in regression analyses to enable consistent interpretation of regression coefficients. All analyses also were weighted to account for the national population of practices represented in the IMS Healthcare Organizational Services database, adjusted for clustering in the survey sample and the probability of selection and nonresponse variables. A number of sensitivity analyses were done in relation to the specification of the PMSYS index, such as using the factor scores instead of the arithmetic means to aggregate the subcomponents of the PMSYS index for the overall composite measure. All analyses were performed using Stata Version 13.1 (StataCorp, 2013).

Results

Extent of Implementation of Internal PMSYS in U.S. Physician Organizations

On average, U.S. physician organizations implemented few (32 points out of 100; interquartile range [IQR] = 0-50 points) of the PMSYS processes measured in the composite PMSYS index (Table 2). Although low on average, the PMSYS index score varied considerably among physician organizations. For example, 26% had not implemented any PMSYS process (0 points), 32% had most of the PMSYS processes (≥50 points); and only 1% implemented all of the PMSYS processes measured (100 points) in the composite index (Figure 2).

Table 2.

Implementation of Performance Management Practices in Physician Organizations: Main Independent Variables.

Variables	n	PMSYS index (ranges for composite index and subcomponents = 0-100 points)
		Physician organization/IPO/PHO provides performance feedback for
		Composite PMSYS index			chronic conditions			Preventive services			PMSYS integration into IT-system			Review, update, and QI
		Mean	(IQR)	p	Mean	(IQR)	p	Mean	(IQR)	p	Mean	(IQR)	p	Mean	(IQR)	p
External factors
Evaluated by external entities on measures of patient satisfaction and clinical quality
0	215	9.7	(0-13)	^b,a	4.2	(0-0)	^b,a	6.4	(0-0)	^b,a	25.1	(0-50)	^b,a	3.0	(0-0)	^b,a
1	339	27.9	(0-44)	^b,k	20.3	(0-25)	^b,k	27.8	(0-50)	^b,k	41.6	(0-75)	^b,k	21.8	(0-0)	^k
2	774	40.0	(13-69)	^a,k	36.5	(0-75)	^a,k	47.5	(0-100)	^a,k	50.4	(0-100)	^a,k	25.4	(0-100)	^k
Pay-for-performance index (range = 0-3)
0	583	23.5	(0-38)	^b,c,a	18.2	(0-0)	^c,a	25.7	(0-50)	^b,c,a	37.5	(0-75)	^b,c,a	12.7	(0-0)	^b,c
1	333	31.1	(0-50)	^b,c,k	31.8	(0-75)	^c,k	35.1	(0-100)	^b,c,k	40.1	(0-100)	^b,c,k	17.3	(0-0)	^b,c
2	277	43.1	(13-69)	^a,c,k	22.4	(0-50)	^c	51.6	(0-100)	^a,c,k	51.1	(0-100)	^a,c,k	47.1	(0-100)	^a,k
3	135	62.9	(50-75)	^a,b,k	61.0	(25-100)	^a,b,k	65.8	(50-100)	^a,b,k	81.7	(75-100)	^a,b,k	43.0	(0-100)	^a,k
Public reporting index (range = 0-2)
0	563	23.4	(0-44)	^b,a	16.1	(0-0)	^b,a	25.2	(0-50)	^b,a	30.0	(0-50)	^b,a	22.1	(0-0)	^a
1	239	38.8	(19-63)	^k	29.0	(0-50)	^b,k	33.5	(0-50)	^b,k	66.0	(50-100)	^k	26.8	(0-100)	^k
2	526	38.9	(6-63)	^k	40.1	(0-75)	^a,k	50.2	(0-100)	^a,k	50.3	(0-100)	^k	15.0	(0-0)
Health plans provide data on the quality of preventive care
0) No	367	20.1	(0-31)	^a	17.0	(0-25)	^a	17.8	(0-0)	^a	34.9	(0-75)	^a	10.9	(0-0)	^a
1) Yes	961	34.7	(6-56)	^k	29.2	(0-75)	^k	39.9	(0-100)	^k	46.2	(0-100)	^k	23.5	(0-0)	^k
Quality improvement collaboratives
0) No	983	27.9	(0-50)	^a	24.7	(0-50)		33.3	(0-100)		39.3	(0-75)	^a	14.3	(0-0)	^a
1) Yes	345	54.2	(19-75)	^k	37.9	(0-75)		45.8	(0-100)		71.6	(50-100)	^k	61.4	(0-100)	^k
Organizational characteristics
Currently participating in an ACO
0) No	1,012	25.4	(0-44)	^a	17.4	(0-25)	^a	27.8	(0-50)	^a	38.2	(0-75)	^a	18.2	(0-0)	^a
1) Yes	316	56.6	(50-75)	^k	63.9	(25-100)	^k	64.8	(50-100)	^k	66.4	(25-100)	^k	31.2	(0-100)	^k
Significant proportion of patients received through an IPA or PHO
0) No	1,070	28.8	(0-50)	^a	19.6	(0-25)	^a	29.8	(0-50)	^a	44.5	(0-100)	^a	21.3	(0-0)
1) Yes	258	43.5	(25-50)	^k	57.5	(0-100)	^k	58.8	(0-100)	^k	39.9	(0-75)	^k	17.9	(0-0)
Internal capabilities
HIT index (range = 0-12)
1) 0-5	404	16.7	(0-38)	^b,c	21.9	(0-25)	^b	25.4	(0-50)	^b,c	9.4	(0-0)	^b,c	10.1	(0-0)	^b,c
2) 6-8	398	39.5	(13-69)	^a,c	34.1	(0-75)	^a	36.9	(0-100)	^a,c	64.7	(50-100)	^a,c	22.4	(0-0)	^a,c
3) 9-12	526	50.4	(25-75)	^a,b	24.7	(0-50)		52.1	(0-100)	^a,b	84.3	(75-100)	^a,b	40.5	(0-100)	^a,b
Electronic chronic disease registry
0) No	1,022	26.1	(0-44)	^a	22.1	(0-25)	^a	29.4	(0-50)	^a	37.8	(0-75)	^a	15.2	(0-0)	^a
1) Yes	306	72.6	(63-88)	^k	60.2	(50-75)	^k	78.0	(50-100)	^k	88.9	(75-100)	^k	63.2	(0-100)	^k
Total	1,328	31.5	(0-50)		26.5	(0-50)		35.0	(0-100)		43.7	(0-100)		20.7	(0-0)

Note. PMSYS = performance management systems; IT = information technology; QI = quality improvement; IPO = independent practice organization; PHO = physician–hospital organization; IQR = interquartile range; ACO = accountable care organization. All analyses use weighted data.

Statistically different mean than practices from Category 1 (p < .05). ^bStatistically different mean than practices from Category 2 (p < .05). ^cStatistically different mean than practices from Category 3 (p < .05). ^dStatistically different mean than practices from Category 0 (p < .05).

Figure 2.

Distribution of PMSYS index scores among 1,328 U.S. physician organizations.

On average, physician organizations implemented 27% (IQR = 0% to 50%) of the performance feedback processes for chronic conditions, 35% (IQR = 0% to 100%) of performance feedback processes for preventive services processes, 44% (IQR = 0% to 100%) of IT-related PMSYS processes, and 21% (IQR = 0% to 0%) had implemented at least one or more regular review, update, and continuous QI processes (Table 2). A majority of organizations (62%) had not implemented a performance feedback process for any of the four chronic conditions measured or either of the two preventive services measured (56%). Many organizations (43%) had not implemented any of the four ITs related PMSYS processes and 79% did not have regular review, update, and continuous QI processes (data not shown). Few organizations implemented performance feedback processes for all four of the chronic conditions measured (16%) and both preventive services measured (26%). Only 27% had implemented all four ITs related PMSYS processes and a mere 1% implemented all four regular review, update, and continuous QI processes (data not shown).

Factors Contributing to the Adoption of Internal PMSYS

Descriptive Analysis

External factors, organizational characteristics, and internal capabilities are outlined in Table 2 in relation to the composite PMSYS index and its subcomponents. Most external factors, factors related to organizational characteristics, and internal capabilities were associated with PMSYS index scores.

External Factors

Physician organizations had a significantly (p < .05) more developed PMSYS when evaluated by health plans or other entities (highest external evaluation score: 40 vs. no external evaluation: 10 points), confronted with P4P incentives (highest P4P score: 63 vs. no P4P: 24 points) or public reporting of performance data (highest public reporting score: 40 vs. no public reporting: 23 points), or when they received data by health plans (35 vs. 20 points) or took part in QI collaboratives (54 vs. 28 points).

Organizational Characteristics

Compared with physician organizations not affiliated with ACOs or with IPAs or PHOs, those affiliated had a significantly (p < .05) higher PMSYS index score (ACO: 57 vs. 25 points; independent practice organization/PHO: 44 vs. 29 points).

Internal Capabilities

Physician organizations had a significantly (p < .05) higher PMSYS index score, when having greater HIT-functionality (highest HIT score category: 50 vs. lowest HIT score category: 17 points) or using electronic chronic disease registries (73 vs. 26 points).

Similar results for the external factor and organizational capability variables could be found for all PMSYS subcomponents; with one notable exception. Greater use of public reporting requirements was not associated with more regular review, update, and established continuous QI processes (Table 2).

Adjusted Analyses

In adjusted analyses (Table 3), physician organizations evaluated by health plans or other entities (p < .01), incentivized by P4P (p < .01), that received data from health plans (p < .01), with an ACO (p < .01), or IPA or PHO affiliation (p < .01), with greater HIT functionality (p < .01) and chronic disease registries (p < .01) had higher PMSYS index scores. Notable is that ACO affiliation is associated with an additional 16 points on the PMSYS index and having at least two chronic disease registries was associated with an additional 19 points on the PMSYS index. In adjusted analyses, publically reporting clinical and patient experience performance data and QI collaboratives were not associated with PMSYS.

Table 3.

Main Independent Factors Associated With Scores On the Performance Management Systems (PMSYS) Index.

Factor	Regression coefficient^a				Odds ratio^b
	Composite PMSYS index	PO/IPA/PHO feedback for		PMSYS-IT integration	Review, update, and QI
	Composite PMSYS index	Chronic diseases	Preventive services	PMSYS-IT integration	Review, update, and QI
External factors
Evaluated by external entities (range = 0-2)	4.5**	4.7**	6.6**	0.1	2.3*
Pay-for-performance index (range = 0-3)	1.6**	−2.2	0.9	3.6*	1.4
Public reporting index (range = 0-2)	−0.5	1.3	1.3	6.5**	0.3*
Health plans provide data	7.0**	3.6	10.1**	−0.7	7.7
QI collaboratives	3.9	1.2	−0.9	−4.2	5.3**
Organizational characteristics
Currently participating in an ACO	15.7**	37.1**	19.3**	12.4**	0.4
IPA/PHO affiliation	7.3**	22.7**	10.2**	−2.5	1.0
Internal capabilities
HIT index (range = 0-12)	9.7**	0.3	4.7**	31.1**	1.4**
Electronic chronic disease registry	18.8**	17.4*	23.5**	6.8**	7.9**
Observations	1,328	1,328	1,328	1,328	1,328
R ²	0.60	0.45	0.38	0.69	—
F test model	22,642**	13,200**	7,719**	25,739**	1,581**

Note. HIT = health information technology; IPA = independent practice association; PHO = physician–hospital organization; PO = physician organization; QI = quality improvement; ACO = accountable care organization. Data were adjusted by weights based on sampling probabilities and poststratification comparisons. All estimates are complex sample design estimates computed using the survey procedure in Stata/IC, version 13.1. For more details about the factors, see “Measures.” We also controlled for payer and the mix of racial and ethnic minority patients, region, ownership, practice size and specialty mix (data not shown).

Linear regression. ^bLogistic regression.

p < .5. **p < .01.

The analyses of subcomponents of the PMSYS index had similar patterns as the overall index, with some notable differences. In contrast to the positive association of external evaluation (p < .01) or the provision of performance data by health plans (p < .01 to p < .10) and three subcomponents of the PMSYS index, no such associations were found for the fourth subcomponent: PMSYS-IT-integration. Instead, public reporting (p < .05) and P4P incentives (p < .05) were significantly positively related to more developed PMSYS-IT-integration, but not the other PMSYS subcomponents. In addition, physician organizations with greater public reporting requirements had less developed (p < .05) review, update, and QI processes compared with practices less involved in public reporting. ACO affiliation was positively (p < .01) associated with higher scores on all PMSYS subcomponents, with the exception of review, update, and QI. Practices affiliated with an IPA or PHO implemented feedback for chronic diseases and preventive services to significantly greater degrees (p < .01) than unaffiliated practices, but no such association was found for PMSYS-IT integration and review, update, and QI. In contrast to our hypothesis, taking part in a QI collaborative was not significantly associated with a greater PMSYS score and was only positively associated (p < .01) with the review, update, and QI processes index.

Finally, physician organizations with greater HIT functionality were more likely to have higher scores on all PMSYS index subcomponents, but these associations were not statistically significant (p > .10) for the performance feedback for chronic conditions subcomponent.

Several control variables were significantly related to the PMSYS index score in adjusted analyses. Physician organizations with a high proportion of revenue from Medicaid and uninsured low-income patients (p < .05), those from specific regions (e.g., WestSouthCentral, p < .01), and multispecialty practices (p < .01) had higher PMSYS index scores (data not shown). In contrast, practices with a higher proportion of patients with limited English proficiency (p < .01), those with a size of three to seven physician practices (compared with one to two physician practices; p < .01), and practices with less than 33% primary care physicians (p < .01) had less developed PMSYS (data not shown).

All associations of the composite PMSYS index and the subcomponents to the main independent variables are outlined in Table 3. A number of sensitivity analyses were conducted in relation to the specification of the PMSYS index, but none of them substantially changed the results or conclusions reported here (data not shown).

Discussion

These are the first national data characterizing the implementation of PMSYS in U.S. physician organizations. Six findings are of particular importance. First, most physician organizations in the United States have underdeveloped PMSYS. On average, physician organizations in the United States only implemented few (32 points out of 100) of the PMSYS processes examined. Regular review, update, and continuous QI processes were especially underdeveloped. The vast majority of practices (79%) did not use any QI methods yet. Only 10% used one, 7% used two, 3% used three, and only 1% four QI methods assessed; which may reflect the limited evidence about the benefits of combining QI methods (Hughes, 2008). These findings highlight that there is considerable room for improving how performance measurement activities are supported by continuous performance improvement processes (regular review, update, and QI), so that performance is not only measured but also managed. Even ACO-affiliated physician organizations implemented a modest number (57 points out of 100, on average) of the measured PMSYS processes, in spite of the central nature of performance measurement and management to achieving shared savings and quality goals of ACOs.

Second, we assessed factors associated with the implementation of PMSYS and found that greater implementation of PMSYS processes was associated with external factors, organizational characteristics, as well as internal capabilities. As hypothesized, physician organizations that have more developed PMSYS tend to have greater external incentives, including evaluation by health plans or other entities, receipt of performance data by health plans, and performance-based financial incentives (P4P). Physician organizations affiliated with an ACO and/or IPA or PHO, those with greater HIT functionality and electronic chronic disease registries implemented tend to have more developed PMSYS. Results of this study are consistent with previous empirical research that emphasizes the importance of these factors on the implementation of care management processes (Buntin et al., 2011; Cassel et al., 2014; Phipps-Taylor & Shortell, 2016; Rittenhouse et al., 2011; Wiley et al., 2015).

Third, participating in an ACO appears to be a strong differentiator of physician organizations with robust versus underdeveloped PMSYS. ACO affiliation can support the development of internal organizational capabilities if ACOs have the capacity to provide technical assistance to practices or provide centralized resources for supporting performance measurement activities. Thus, further strengthening of ACO initiatives may support the implementation of PMSYS among affiliated practices. Importantly, ACOs are currently incentivized to exclude underdeveloped physician practices, as value-based payment models emphasize enrolled or physician-attributed populations. Expansion of geographically oriented payment and delivery system reform models, such as Accountable Care Communities in the United States (Casalino, Erb, Joshi, & Shortell, 2015; Tipirneni, Vickery, & Ehlinger, 2015; Zusman et al., 2014) or the German ACO Gesundes Kinzigtal (Barnes, Unruh, Chukmaitov, & van Ginneken, 2014), can emphasize the inclusion of underperforming practices in their region and leverage their existing system infrastructure to improve the PMSYS of these practices. Incentives for practices to join ACOs or other integrated networks may also strengthen internal organizational capabilities in general.

Fourth, our findings highlight that electronic chronic disease registries are foundational for PMSYS in physician practices. Our results indicate that only about one fourth of U.S. medical practices have electronic chronic disease registries and also only about one third have highly developed HIT systems. As both are positively associated with more robust PMSYS, policy initiatives to support the implementation and use of HIT and electronic chronic disease registries, such as the CMS (2016) Clinical Practice Transformation Initiative, are warranted for practices that do not have the resources to develop these capabilities through networks or other means.

Fifth, in contrast to other studies (Wiley et al., 2015), we found no relationship between public reporting requirements and PMSYS implementation in general. Public reporting was only positively associated with greater PMSYS-IT-integration and not other dimensions of PMSYS. Public reporting was even negatively associated with internal review, update, and continuous QI. Furthermore, greater P4P was also only positively associated with PMSYS-IT-integration. P4P and public reporting requirements may foster the technical adoption of practices relevant to meeting external requirements for reimbursement, but not the cultural changes necessary to also make performance measurement and continuous improvement to an integral part of the organizational culture. By contrast, evaluation and the provision of performance data by health plans or other entities are positively associated with internal review, update, and continuous QI. These mixed results related to the impact of P4P and external requirements are consistent with the perspective that there are unintended consequences of using performance-based financial incentives in health care (Campbell et al., 2009; Cassel & Jain, 2012) and that too much emphasis on external incentives may stifle the development of PMSYS among practices, as external requirements may compete for resources with internal QI (Cassel et al., 2014; Meyer et al., 2012). This also highlights the challenge of achieving the right balance of emphases on responding to external requirements and developing the internal capacities to respond to them.

Finally, contrary to our hypothesis, practices that participated in QI collaboratives did not have more robust PMSYS compared with practices that had never participated in QI collaboratives. Taking part in QI collaboratives only was associated with a higher score on the review, update, and QI index. It may be that practices more likely to participate in QI collaboratives are doing so because they aim to improve their internal capabilities through these initiatives. As a result, those with less developed PMSYS may be more likely to participate. Interorganizational learning may translate into more robust internal QI processes, explaining the higher scores of QI collaborative participants on the review, update, and QI index. These organizations, however, may face difficulty in implementing relatively more resource-intensive performance feedback systems and HIT functionality. Joining ACOs or other networks may also be an internal strategy for these practices to gain access to the resources required to implement robust PMSYS. Continued support of external initiatives to aid in the expansion of HIT and use of electronic chronic disease registries is warranted to advance PMSYS implementation.

Limitations

Our results should be considered in light of important limitations. First, the NSPO3 survey has the same limitations as other self-reported assessments, including potential limited sample representativeness and social desirability bias (Wiley et al., 2015). Previous analyses of the NSPO3 survey, however, find only minor differences between respondent versus nonrespondent organizations, and highlight internal consistency reliability and predictive validity of the questions we analyzed (McHugh et al., 2016; Ramsay, Shortell, Casalino, Rodriguez, & Rittenhouse, 2016).

Second, our unique national data allowed for measurement of critical dimensions of PMSYS implementation, but all PMSYS components were not assessed. PMSYS criteria (Figure 1), such as the flexibility or ease of use of the PMSYS, the alignment of the PMSYS with strategy, the extent of key stakeholders involvement in the PMSYS process, strategic planning and control systems, and budgeting and resource allocation issues related to PMSYS would be important to assess in future research. Moreover, a more comprehensive set of performance measures, such as productivity, access or efficiency measures or additional preventive services measures, would have strengthened the internal validity of our findings. In spite of these unmeasured PMSYS variables, the PMSYS index we used captures crucial components of the PMSYS processes needed for physician organizations to successfully manage and improve performance. Finally, the cross-sectional nature of the data limits our ability to establish causal relationships. Nevertheless, the results identify important relationships between PMSYS, organizational capabilities, and external influences that can be examined further when longitudinal data are available.

Implications for Policy and Practice

ACOs, IPAs/PHOs, and other networks appear to enable the implementation of PMSYS to improve organizational performance. The further strengthening of these networks may promote internal capabilities of participating practices, including the development of robust PMSYS. PMSYS are likely to become increasingly important for health care providers and systems given the coming implementation of the Medicare Access and CHIP Reauthorization Act of 2015 value-based payment models that will apply to all physician practices. Reinforcing emerging geographical population-based models, such as Accountable Care Communities, could offer an opportunity to realign the financial incentives for ACOs and other networks to also include underdeveloped practices, as they then are not solely responsible for their physician-attributed populations, but the whole regional population. Policy initiatives that support the implementation of HIT, electronic chronic disease registries, and PMSYS, such as the CMS (2016) Clinical Practice Transformation Initiative, should be promoted for practices that do not have the resources or capabilities to implement PMSYS through networks or other means. Given the potential burden of external reporting on PMSYS implementation, the harmonization of measures and other requirements between payers and other entities should receive high priority in policy and practice.

Footnotes

Acknowledgements

The authors thank Lawrence P. Casalino and Diane Rittenhouse for their contributions in helping to develop and field the NSPO survey, Kennon R. Copeland for constructing survey weights, and Patricia P. Ramsay for analytic support.

Authors’ Note

The statements, findings, conclusions, views, and opinions contained and expressed in this article are based in part on data obtained under license from the following IMS Health information services: Healthcare Organizational Services (2007), IMS Health Inc. All rights reserved. The statements, findings, conclusions, views, and opinions contained and expressed herein are not necessarily those of IMS Health Inc. or any of its affiliated or subsidiary entities. University of California, Berkeley Committee for the Protection of Human Subjects protocol #2014-06-6480, Weill Cornell Medical College Human Research Protections Programs protocol #EXE2011-035. The views presented here are those of the authors and not necessarily those of The Commonwealth Fund, its directors, officers, or staff.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The National Study of Physician Organizations III was funded by the Robert Wood Johnson Foundation (Award No. 68847). Dr. Pimperl was supported by The Commonwealth Fund (10.13039/100000905), a private independent foundation based in New York City. Dr. Schmittdiel received support from the Health Delivery Systems Center for Diabetes Translational Research (P30 DK092924).

References

Adair

E. C.

Simpson

Birdsell

M. J.

Omelchuk

Casebeer

L. A.

Gardiner

H. P.

. . . Beausejour

(2003). Performance measurement systems in health and mental health services: Models, practices and effectiveness: A state of the science review. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.195.2219&rep=rep1&type=pdf

Aldrich

(2008). Organizations and environments. Stanford, CA: Stanford Business Books.

Andersson

Eriksson

Torstensson

(2006). Similarities and differences between TQM, six sigma and lean. TQM Journal, 18, 282-296. doi:10.1108/09544780610660004

Bach

P. B.

Pham

H. H.

Schrag

Tate

R. C.

Hargraves

J. L.

(2004). Primary care physicians who treat blacks and whites. New England Journal of Medicine, 351, 575-584. doi:10.1056/NEJMsa040609

Barnes

A. J.

Unruh

Chukmaitov

van Ginneken

(2014). Accountable Care Organizations in the USA: Types, developments and challenges. Health Policy, 118, 1-7. doi:10.1016/j.healthpol.2014.07.019

Barney

(1991). Firm resources and sustained competitive advantage. Journal of Management, 17, 99-120.

Blumenthal

(2011a). Implementation of the Federal Health Information Technology initiative. New England Journal of Medicine, 365, 2426-2431. doi:10.1056/NEJMsr1112158

Blumenthal

(2011b). Wiring the health system: Origins and provisions of a new federal program. New England Journal of Medicine, 365, 2323-2329. doi:10.1056/NEJMsr1110507

Boermans

Kattenberg

(2011). Estimating reliability coefficients with heterogeneous item weightings using Stata: A factor based approach. Retrieved from http://econpapers.repec.org/paper/usetkiwps/1119.htm

10.

Bohmer

R. M.

(2011). The four habits of high-value health care organizations. New England Journal of Medicine, 365, 2045-2047.

11.

Buntin

M. B.

Burke

M. F.

Hoaglin

M. C.

Blumenthal

(2011). The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Affairs, 30, 464-471. doi.10.1377/hlthaff.2011.0178

12.

Campbell

S. M.

Reeves

Kontopantelis

Sibbald

Roland

(2009). Effects of pay for performance on the quality of primary care in England. New England Journal of Medicine, 361, 368-378. doi:10.1056/NEJMsa0807651

13.

Casalino

L. P.

Erb

Joshi

M. S.

Shortell

S. M.

(2015). Accountable care organizations and population health organizations. Journal of Health Politics, Policy and Law, 40, 821-837. doi:10.1215/03616878-3150074

14.

Casalino

L. P.

Gans

Weber

Cea

Tuchovsky

Bishop

T. F.

. . . Evenson

T. B.

(2016). US physician practices spend more than $15.4 billion annually to report quality measures. Health Affairs, 35, 401-406. doi:10.1377/hlthaff.2015.1258

15.

Casalino

L. P.

Gillies

R. R.

Shortell

S. M.

Schmittdiel

Bodenheimer

Robinson

Wang

M. C.

(2003). External incentives, information technology, and organized processes to improve health care quality for patients with chronic diseases. Journal of the American Medical Association, 289, 434-441. doi:10.1001/jama.289.4.434

16.

Casalino

L. P.

F. M.

Ryan

A. M.

Copeland

Rittenhouse

D. R.

Ramsay

P. P.

Shortell

S. M.

(2013). Independent practice associations and physician-hospital organizations can improve care management for smaller practices. Health Affairs, 32, 1376-1382. doi:10.1377/hlthaff.2013.0205

17.

Cassel

C. K.

Conway

P. H.

Delbanco

S. F.

Jha

A. K.

Saunders

R. S.

Lee

T. H.

(2014). Getting more performance from performance measurement. New England Journal of Medicine, 371, 2145-2147. doi:10.1056/NEJMp1408345

18.

Cassel

C. K.

Jain

S. H.

(2012). Assessing individual physician performance: Does measurement suppress motivation? Journal of the American Medical Association, 307, 2595-2596. doi:10.1001/jama.2012.6382

19.

Centers for Medicare & Medicaid Services. (2015). CMS Quality Measure Development Plan: Supporting the transition to the Merit-based Incentive Payment System (MIPS) and Alternative Payment Models (APMs) (DRAFT). Retrieved from https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment-Instruments/Value-Based-Programs/MACRA-MIPS-and-APMs/Draft-CMS-Quality-Measure-Development-Plan-MDP.pdf

20.

Centers for Medicare & Medicaid Services. (2016). Transforming Clinical Practice Initiative. Retrieved from https://innovation.cms.gov/initiatives/Transforming-Clinical-Practices/

21.

Chien

A. T.

Rosenthal

M. B.

(2013). Medicare’s physician value-based payment modifier: Will the tectonic shift create waves? New England Journal of Medicine, 369, 2076-2078. doi:10.1056/NEJMp1311957

22.

Cronbach

L. J.

(1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. doi:10.1007/BF02310555

23.

D’Agostino

R. B.

Russell

H. K.

(2005). Scree test. In Armitage

Colton

(Eds.), Encyclopedia of biostatistics (Vol. 7). New York, NY: John Wiley. Available at: http://onlinelibrary.wiley.com/doi/10.1002/0470011815.b2a10082/abstract?userIsAuthenticated=false&deniedAccessCustomisedMessage=

24.

Davis

Guterman

Collins

S. R.

Stremikis

Rustgi

Nuzum

(2010). Starting on the path to a high performance health system: Analysis of the payment and system reform provisions in the Patient Protection and Affordable Care Act of 2010. Retrieved from http://www.commonwealthfund.org/~/media/files/publications/fund-report/2010/sep/1442_davis_payment-and-system-reform_923v2.pdf

25.

Farmer

S. A.

Black

Bonow

R. O.

(2013). Tension between quality measurement, public quality reporting, and pay for performance. Journal of the American Medical Association, 309, 349-350.

26.

Fisher

E. S.

Wennberg

D. E.

Stukel

T. A.

Gottlieb

D. J.

Lucas

F. L.

Pinder

É. L.

(2003). The implications of regional variations in Medicare spending: Part 2: Health outcomes and satisfaction with care. Annals of Internal Medicine, 138, 288-298. doi:10.7326/0003-4819-138-4-200302180-00007

27.

Franco-Santos

Bourne

(2005). An examination of the literature relating to issues affecting how companies manage through measures. Production Planning & Control, 16, 114-124.

28.

Friedberg

M. W.

Coltin

K. L.

Safran

D. G.

Dresser

Schneider

E. C.

(2010). Medical home capabilities of primary care practices that serve sociodemographically vulnerable neighborhoods. Archives of Internal Medicine, 170, 938-944. doi:10.1001/archinternmed.2010.110

29.

Fung

C. H.

Lim

Y.-W.

Mattke

Damberg

Shekelle

P. G.

(2008). Systematic review: The evidence that publishing patient care performance data improves quality of care. Annals of Internal Medicine, 148, 111-123. doi:10.7326/0003-4819-148-2-200801150-00006

30.

Garengo

Biazzo

Bititci

U. S.

(2005). Performance measurement systems in SMEs: A review for a research agenda. International Journal of Management Reviews, 7, 25-47. doi:10.1111/j.1468-2370.2005.00105.x

31.

Haut

E. R.

Pronovost

Schneider

(2012). Limitations of administrative databases. Journal of the American Medical Association, 307, 2589-2590. doi:10.1001/jama.2012.6626

32.

Heisey-Grove

Danehy

L.-N.

Consolazio

Lynch

Mostashari

(2014). A national study of challenges to electronic health record adoption and meaningful use. Medical Care, 52, 144-148. doi:10.1097/MLR.0000000000000038

33.

Higgins

Veselovskiy

McKown

(2013). Provider performance measures in private and public programs: Achieving meaningful alignment with flexibility to innovate. Health Affairs, 32, 1453-1461. doi:10.1377/hlthaff.2013.0007

34.

Hofer

Hayward

R. A.

Greenfield

Wagner

E. H.

Kaplan

S. H.

Manning

W. G.

(1999). The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. Journal of the American Medical Association, 281, 2098-2105. doi:10.1001/jama.281.22.2098

35.

Holroyd-Leduc

J. M.

Lorenzetti

Straus

S. E.

Sykes

Quan

(2011). The impact of the electronic medical record on structure, process, and outcomes within primary care: A systematic review of the evidence. Journal of the American Medical Informatics Association, 18, 732-737. doi:10.1136/amiajnl-2010-000019

36.

Hoque

(2004). A contingency model of the association between strategy, environmental uncertainty and performance measurement: Impact on organizational performance. International Business Review, 13, 485-502. doi:10.1016/j.ibusrev.2004.04.003

37.

Hudson

Smart

Bourne

(2001). Theory and practice in SME performance measurement systems. International Journal of Operations & Production Management, 21, 1096-1115.

38.

Hughes

R. G.

(2008). Tools and strategies for quality improvement and patient safety. In Hughes

R. G.

(Ed.), Patient safety and quality: An evidence-based handbook for nurses. Rockville, MD: Agency for Healthcare Research and Quality (US). Retrieved from http://www.ncbi.nlm.nih.gov/books/NBK2682/

39.

Institute of Medicine. (2001). Crossing the Quality Chasm: A new health system for the 21st century. Washington, DC: National Academies Press.

40.

Julnes

P. L.

Holzer

(2001). Promoting the utilization of performance measures in public organizations: An empirical study of factors affecting adoption and implementation. Public Administration Review, 61, 693-708. doi:10.1111/0033-3352.00140

41.

Kern

L. M.

Barrón

Dhopeshwarkar

R. V.

Edwards

Kaushal

, & HITEC Investigators. (2013). Electronic health records and ambulatory quality of care. Journal of General Internal Medicine, 28, 496-503. doi:10.1007/s11606-012-2237-8

42.

Landon

Normand

Blumenthal

Daley

(2003). Physician clinical performance assessment: Prospects and barriers. Journal of the American Medical Association, 290, 1183-1189. doi:10.1001/jama.290.9.1183

43.

Lawrence

P. R.

Lorsch

J. W.

(1967). Differentiation and integration in complex organizations. Administrative Science Quarterly, 12, 1-47.

44.

Little

R. J. A.

(1993). Post-stratification: A modeler’s perspective. Journal of the American Statistical Association, 88, 1001-1012. doi:10.1080/01621459.1993.10476368

45.

Lynch

Kendall

Shanks

Haque

Jones

Wanis

M. G.

. . . Mostashari

(2014). The Health IT Regional Extension Center program: Evolution and lessons for health care transformation. Health Services Research, 49, 421-437. doi:10.1111/1475-6773.12140

46.

Marsden

P. V.

Landon

B. E.

Wilson

I. B.

McInnes

Hirschhorn

L. R.

Ding

Cleary

P. D.

(2006). The reliability of survey assessments of characteristics of medical clinics. Health Services Research, 41, 265-283. doi:10.1111/j.1475-6773.2005.00480.x

47.

McClellan

McKethan

A. N.

Lewis

J. L.

Roski

Fisher

E. S.

(2010). A national strategy to put Accountable Care into practice. Health Affairs, 29, 982-990. doi:10.1377/hlthaff.2010.0194

48.

McHugh

Shi

Ramsay

P. P.

Harvey

J. B.

Casalino

L. P.

Shortell

S. M.

Alexander

J. A.

(2016). Patient-centered medical home adoption: Results from aligning forces for quality. Health Affairs, 35, 141-149. doi:10.1377/hlthaff.2015.0495

49.

McNair

P. D.

Luft

H. S.

Bindman

A. B.

(2009). Medicare’s policy not to pay for treating hospital-acquired conditions: The impact. Health Affairs, 28, 1485-1493. doi:10.1377/hlthaff.28.5.1485

50.

Mechanic

Zinner

D. E.

(2012). Many large medical groups will need to acquire new skills and tools to be ready for payment reform. Health Affairs, 31, 1984-1992. doi:10.1377/hlthaff.2012.0127

51.

Meyer

G. S.

Nelson

E. C.

Pryor

D. B.

James

Swensen

S. J.

Kaplan

G. S.

. . . Hunt

G. C.

(2012). More quality measures versus measuring what matters: A call for balance and parsimony. BMJ Quality & Safety, 21, 964-968. doi:10.1136/bmjqs-2012-001081

52.

Nadeem

Olin

S. S.

Hill

L. C.

Hoagwood

K. E.

Horwitz

S. M.

(2013). Understanding the components of quality improvement collaboratives: A systematic literature review. Milbank Quarterly, 91, 354-394. doi:10.1111/milq.12016

53.

Neely

Gregory

Platts

(1995). Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 15(4), 80-116.

54.

Nembhard

I. M.

(2012). All teach, all learn, all improve? The role of interorganizational learning in quality improvement collaboratives. Health Care Management Review, 37, 154-164. doi:10.1097/HMR.0b013e31822af831

55.

Peteraf

M. A.

(1993). The cornerstones of competitive advantage: A resource-based view. Strategic Management Journal, 14, 179-191.

56.

Petersen

L. A.

Woodard

L. D.

Urech

Daw

Sookanan

(2006). Does pay-for-performance improve the quality of health care? Annals of Internal Medicine, 145, 265-272. doi:10.7326/0003-4819-145-4-200608150-00006

57.

Phipps-Taylor

Shortell

S. M.

(2016). More than money: Motivating physician behavior change in accountable care organizations. Milbank Quarterly, 94, 832-861.

58.

Pimperl

(2015). Performance management Systeme für Netzwerke im Gesundheitswesen: Entwicklung einer Soll-Konzeption. Taunusstein, Hesse, Germany: Driesen.

59.

Porter

M. E.

(2010). What is value in health care? New England Journal of Medicine, 363, 2477-2481.

60.

Ramsay

P. P.

Shortell

S. M.

Casalino

L. P.

Rodriguez

H. P.

Rittenhouse

D. R.

(2016). A longitudinal study of medical practices’ treatment of patients who use tobacco. American Journal of Preventive Medicine, 50, 328-335. doi:10.1016/j.amepre.2015.07.005

61.

Reschovsky

J. D.

O’Malley

A. S.

(2008). Do primary care physicians treating minority patients report problems delivering high-quality care? Health Affairs, 27, w222-w231. doi:10.1377/hlthaff.27.3.w222

62.

Rittenhouse

D. R.

Casalino

L. P.

Gillies

R. R.

Shortell

S. M.

Lau

(2008). Measuring the medical home infrastructure in large medical groups. Health Affairs, 27, 1246-1258.

63.

Rittenhouse

D. R.

Casalino

L. P.

Shortell

S. M.

McClellan

S. R.

Gillies

R. R.

Alexander

J. A.

Drum

M. L.

(2011). Small and medium-size physician practices use few patient-centered medical home processes. Health Affairs, 30, 1575-1584. doi:10.1377/hlthaff.2010.1210

64.

Rittenhouse

D. R.

Shortell

S. M.

Gillies

R. R.

Casalino

L. P.

Robinson

J. C.

McCurdy

R. K.

Siddique

(2010). Improving chronic illness care: Findings from a National Study of Care Management Processes in large physician practices. Medical Care Research and Review, 67, 301-320. doi:10.1177/1077558709353324

65.

Rundall

T. G.

Shortell

S. M.

Wang

M. C.

Casalino

Bodenheimer

Gillies

R. R.

. . . Robinson

J. C.

(2002). As good as it gets? Chronic care management in nine leading US physician organisations. British Medical Journal, 325, 958-961. doi:10.1136/bmj.325.7370.958

66.

Salah

Rahim

Carretero

J. A.

(2010). The integration of Six Sigma and lean management. International Journal of Lean Six Sigma, 1, 249-274. doi.10.1108/20401461011075035

67.

Shortell

S. M.

Mcclellan

S. R.

Ramsay

P. P.

Casalino

L. P.

Ryan

A. M.

Copeland

K. R.

(2014). Physician practice participation in Accountable Care Organizations: The emergence of the unicorn. Health Services Research, 49, 1519-1536.

68.

Shortell

S. M.

Schmittdiel

Wang

M. C.

Gillies

R. R.

Casalino

L. P.

. . . Rundall

T. G.

(2005). An empirical assessment of high-performing medical groups: Results from a national study. Medical Care Research and Review, 62, 407-434.

69.

StataCorp. (2013). Svy estimation. In Stata survey data reference manual: Release 13 (pp. 89-101). College Station, TX: Stata Press.

70.

Tipirneni

Vickery

K. D.

Ehlinger

E. P.

(2015). Accountable communities for health: Moving from providing accountable care to creating health. Annals of Family Medicine, 13, 367-369. doi:10.1370/afm.1813

71.

Varkey

Manwell

Williams

Ibrahim

Brown

Bobula

. . . Linzer

(2009). Separate and unequal: Clinics where minority and nonminority patients receive primary care. Archives of Internal Medicine, 169, 243-250. doi:10.1001/archinternmed.2008.559

72.

Wagner

E. H.

Austin

B. T.

Davis

Hindmarsh

Schaefer

Bonomi

(2001). Improving chronic illness care: Translating evidence into action. Health Affairs, 20(6), 64-78. doi:10.1377/hlthaff.20.6.64

73.

Wennberg

J. E.

Freeman

J. L.

Shelton

R. M.

Bubolz

T. A.

(1989). Hospital use and mortality among Medicare beneficiaries in Boston and New Haven. New England Journal of Medicine, 321, 1168-1173. doi:10.1056/NEJM198910263211706

74.

Werner

R. M.

Asch

D. A.

(2005). The unintended consequences of publicly reporting quality information. Journal of the American Medical Association, 293, 1239-1244.

75.

Wiley

J. A.

Rittenhouse

D. R.

Shortell

S. M.

Casalino

L. P.

Ramsay

P. P.

Bibi

. . . Alexander

J. A.

(2015). Managing chronic illness: Physician practices increased the use of care management and medical home processes. Health Affairs, 34, 78-86. doi:10.1377/hlthaff.2014.0404

76.

Wright

(2012). Few adverse events in hospitals were reported to state adverse event reporting systems. Washington, DC: Office of Inspector General.

77.

Zusman

E. E.

Carr

S. J.

Robinson

Kasirye

Zell

Miller

W. J.

. . . Williams

(2014). Moving toward implementation: The potential for accountable care organizations and private–public partnerships to advance active neighborhood design. Preventive Medicine, 69, S98-S101. doi:10.1016/j.ypmed.2014.08.009