Linking Stakeholder Involvement to Policy Performance: Nonlinear Effects in Dutch Local Government Policy Making

Abstract

Current research in public management generally hypothesizes that the involvement of external stakeholders by governments positively affects the performance of policies. Recent research, however, has demonstrated diminishing returns of involvement on performance, as well as different effects of involvement for different types of stakeholder organizations. The present article combines these insights. We distinguish between professional and client-interest stakeholder organizations, and assess the effect of their involvement on policy performance in terms of client outcomes. The hypotheses are tested using a combined longitudinal data set consisting of a representative sample of 69 Dutch local governments and 3,434 clients of the Social Support Act, which aims to increase the independent functioning of individuals with mental or physical impediments. Multilevel analyses show that only the involvement of professional stakeholder organizations is related to policy performance, with negative returns on policy performance at higher levels of involvement.

Keywords

stakeholder involvement managerial networking policy performance local government

Introduction

The past two decades of public management research have shown that a great degree of interdependence exists among local governments and nongovernmental actors in the provision of public services (Agranoff, 2003; Walker & Andrews, 2015), as part of a more general and growing interest in collaborative and networked arrangements (Kapucu, Hu, & Khosa, 2014; Moynihan, Provan, & Lemaire, 2012). Local governments increasingly engage in interactive policy-making activities to enable nongovernmental stakeholders to voice their opinions and concerns in the decision-making process (Denters, van Heffen, Huisman, & Klok, 2003; Edelenbos & Klijn, 2006) and to coordinate their activities in inter-organizational service delivery (Thomas, Poister, & Ertas, 2010; Walker, O’Toole, & Meier, 2007). This article examines whether and how stakeholder involvement in local governments’ policy making affects the performance of local policies.

Stakeholder involvement by governments and public agencies has been referred to as “collaborative governance” (Ansell & Gash, 2008), inter-organizational collaboration (Vangen, Hayes, & Cornforth, 2015), “public participation” (Rydin & Pennington, 2000), “deliberative democracy” (Feldman, Khademian, Ingram, & Schnieder, 2006), or “interactive decision-making” (Edelenbos & Klijn, 2006). Its practice has been documented in various policy sectors, such as environmental policy (Flynn & Kroeger, 2003), transportation (Thomas & Poister, 2009), and urban development (Agranoff, 2003). “Stakeholders” are nongovernmental actors who can affect, and who are affected by, the performance of policies (Scholl, 2001). They include nonprofit and private-sector organizations, interest groups, other governments, or individual citizens. Stakeholder involvement is typically government initiated, consensus based, and collectively organized, and it offers stakeholders the opportunity to influence policies (Ansell & Gash, 2008). The process itself may take on formalized arrangements, in which stakeholders obtain actual procedural or legislative power, for example, through official committees in negotiated rulemaking (Lavertu & Weimer, 2011). The process can also be organized more informally, for example, through the use of open-access platforms, study groups, citizen panels, or professional focus groups (Bingham, Nabatchi, & O’Leary, 2005; Tatenhove, van Edelenbos, & Klok, 2010).

The goal of stakeholder involvement is to increase the performance of policies in terms of the service outputs and outcomes for the client population whose conditions these policies target, such as mental health patients (Provan & Milward, 1995), drug abusers (Percival, 2009), or veterans (Keiser & Miller, 2010). Studies mostly focus on process-oriented performance indicators, such as litigation rates and the duration of rulemaking (Coglianese, 1997) or the satisfaction of stakeholders with the policy-making process (Gilliam et al., 2002). Alternatively, a growing literature on performance management in the public sector studies stakeholder involvement in terms of the effectiveness of public programs, as measured by service outputs and outcomes—for example, client satisfaction or the speed and reliability of service delivery (Boyne & Walker, 2010; Meier & O’Toole, 2003; O’Toole & Meier, 1999; Schalk, Torenvlied, & Allen, 2010).

Although the majority of these empirical studies find a positive link between stakeholder involvement and performance, the evidence is inconsistent. In particular, recent research has drawn attention to the costs that may condition any positive effect of stakeholder involvement on service outputs and outcomes. There are costs involved in maintaining relationships with stakeholders (Provan & Sydow, 2008), including time and opportunity costs, as well as the increased decision-making time needed to reach agreements (Agranoff, 2006; Coglianese, 1997). Also, the effect of stakeholder involvement on policy performance may be different for different types of stakeholders due to differences in their access to information and resources, their power, and their specific interests (Moynihan & Pandey, 2005; Torenvlied, Akkerman, Meier, & O’Toole, 2013; Walker, Boyne, & Brewer, 2010). Walker et al. (2010), for example, found no overall effect of external networking on local government performance, but they found differential effects when examining political officials (negative effect), user groups (positive effect), and professional organizations (no effect) separately. Thus, it appears that the effect of stakeholder involvement on policy performance may be nonlinear and stakeholder-specific.

In this article, we explore this hypothesis. Our research context is the implementation of the Social Support Act (SSA) in the Netherlands (in Dutch: “Wet Maatschappelijke Ondersteuning,” WMO). The SSA context offers a unique testing ground to examine the relationship between involvement and performance in a systematic and comparative way (cf. Ansell & Gash, 2008, p. 562), by combining two important features. First, all local governments are required to attain the same general SSA goals, which relate to increasing their citizens’ level of independent functioning, in particular those with physical or mental impediments. Second, local governments have been assigned financial and statutory responsibility for the formulation and implementation of local policies to attain these SSA goals. As a consequence, policy performance—in terms of attaining SSA goals—is comparable across local governments, while local discretion in policy making will likely cause variation in the degree to which local governments involve stakeholders in the process.

For the SSA context, we identify two types of stakeholders that are likewise perceived as separate types of partners by public managers in the SSA context (De Klerk, Gilsing, & Timmermans, 2010). First, professional organizations are those partner organizations that do not represent the interests of the target population but deliver services to them, for example, house-cleaning services or transportation. Professional organizations are legal entities and may be for-profit, nonprofit, or semipublic. Second, client-interest organizations are those organizations that represent the target population. These organizations may or may not be legal entities, but they are always nonprofit. Their primary interest is in representing their clients by influencing policies, although they have become increasingly involved in delivering services as well through coproduction arrangements (Bovaird, 2007). Examples in the present context are interest groups for the elderly and for informal caregivers.

Empirically, we test our expectations by conducting multilevel analyses on a unique longitudinal and multi-actor data set consisting of a representative sample of 69 key public managers in Dutch local governments, as well as an additional representative sample of 3,434 individual SSA clients. This data set allows us to evaluate policy performance in terms of actual client outcomes (cf. Boyne, 2003) that are directly related to SSA goals. With clients’ evaluations of their own conditions, we prevent same-source and social-desirability bias compared with policy performance as perceived by governments and stakeholder participants (cf. J. Freeman & Langbein, 2000; Klijn, Steijn, & Edelenbos, 2010).

The Effect of Stakeholder Involvement on Policy Performance

From the vast literature in public administration, it has become clear that there is no single dominant form of organizing stakeholder involvement, and wide variations can be observed in terms of formation, process, and outcome characteristics of collaborative arrangements (Ansell, 2015). A key question for public management then, is under which conditions the active involvement of nongovernmental partners produces the desirable policy outcomes. It is beyond the scope of this article to develop a fully integrated theoretical framework on these conditions. Rather, we elaborate on the various mechanisms put forward in the literature that link the scope of collaboration in terms of the number of different types of stakeholders that are involved by (local) governments to policy performance. Our goal is to develop testable expectations on the effect of stakeholder involvement in the SSA context that may stimulate further theoretical development on this relationship.

There are a number of arguments as to way involving more stakeholders would lead to better service delivery. First of all, involving more stakeholders ensures access to information and resources. From the perspective of the individual public manager, stakeholder involvement constitutes a key managerial task that critically affects program performance (Agranoff, 2003). An important branch of the performance-management literature (Akkerman & Torenvlied, 2011; Boyne, Meier, O’Toole, & Walker, 2006; Hill & Lynn, 2005; O’Toole & Meier, 2004) focuses on stakeholder involvement as the individual networking behavior of public managers. The core explanation underlying the positive association between managerial networking and performance is that public managers who maintain more frequent contact with key stakeholders are better able to acquire resources for the public agency, to reduce informational uncertainties related to political, economic, and technical demands, and to manage unexpected external events (O’Toole & Meier, 1999).

From the perspective of collaborative governance in policy making (Ansell & Gash, 2008; Edelenbos & Klijn, 2006; Furlong & Kerwin, 2005; Huxham & Vangen, 2000; Lavertu & Weimer, 2011), access to information and resources likewise links stakeholder involvement to improved performance. This literature argues that—in addition to bilateral information exchange—the collective nature of stakeholder involvement implies that governments and stakeholders may simultaneously collect, analyze, and combine information from different sources in a process that further facilitates learning (Damgaard & Torfing, 2010; Lipshitz, Popper, & Oz, 1996; Moynihan, 2005).

Moreover, involving different types of organizations—community organizations, interest groups, professional service deliverers, and so on—increases the diversity of information available to solve policy problems. Stakeholders may provide evidence-based information (Head, 2008) derived from their specific core tasks (e.g., health care or social work). This evidence may include performance data for intervention methods and programs related to a policy’s target population, general trends in socio-demographic and economic conditions, or information on best practices (Nutley, Walter, & Davies, 2007; Sullivan & Skelcher, 2002). Second, stakeholders may sound an alert when programs fail or problems occur during service delivery, and they can provide performance feedback for their specific client groups (Walker et al., 2010). Finally, by providing information on their daily operations, tasks, goals, and competencies, stakeholder organizations enable governments to improve the inter-organizational coordination of service delivery, particularly by reducing the likelihood of blind spots (a lack of needed client services) and duplication of services (Provan & Milward, 1995).

Finally, increasing the scope of involvement may build policy support among stakeholders. Governments are dependent on stakeholder organizations because the performance of policies depends on stakeholders’ cooperation and contribution to policy design and implementation. Such cooperation is not self-evident. Although all stakeholders benefit from effective policies (e.g., through improved client services), stakeholder organizations have their own priorities in pursuing various policy options. These priorities derive from their particular understanding of the problem at hand, their perceived need to cooperate, their trust in other participants, their willingness to invest resources, and so on (Head, 2008; Vangen & Huxham, 2012). Stakeholder involvement in platforms, consultation meetings, and projects offers governments the opportunity to build support for policies through argumentation, deliberation, and interest mediation (J. Freeman & Langbein, 2000; Nutt & Backoff, 2002; Provan & Kenis, 2008). Even when policies do not reflect stakeholders’ interests, the perception of being taken seriously is likely to enhance policy support (Newig & Fritsch, 2009).

Although few scholars would argue that there are no benefits to stakeholder involvement, and empirical studies on balance find a positive linear effect on performance, students of stakeholder involvement have identified various barriers that must be overcome to make such collaboration effective. In general, several arguments are put forward as why involving more partners in the collaborative process may in fact negatively affect performance. First, the benefits of access to information and resources are likely to decrease with additional stakeholders, as limits can and will be reached in terms of the usefulness of information and resources that can be obtained from additional partners (Hicklin, O’Toole, & Meier, 2008). Duplication of information becomes more likely, or new information may add unnecessary complexity. At the same time, there are limits to the time, energy, and financial resources of the public managers who organize stakeholder involvement (Provan & Sydow, 2008). Public managers must divide their attention across all their relationships. Consequently, the quality of the relationship with each individual partner may diminish. Opportunity costs exist as well because investments in external networking divert time and resources away from other important managerial tasks related, for instance, to the internal management of the public agency (Favero, Meier, & O’Toole, 2014).

Furthermore, reaching compromises and accounting for a wider variety of interests is more difficult when a larger variety of stakeholders is involved (Bryson, Crosby, & Stone, 2006). Partners can become frustrated with the process (J. Freeman & Langbein, 2000), and trust is likely to diminish (Berardo, 2009). Likewise, real innovative solutions are harder to achieve between many partners, as innovations are typically radical deviations from the status quo that are unlikely to elicit general support (Coglianese, 1997).

In sum, the literature emphasizes both positive and negative effects of access to information, diversity among participant organizations, and involving stakeholders in the decision-making process. Building on this observation, Vangen and Huxham (2012) have argued that differences in organizations’ goals in fact lie at the heart of a paradox in the collaborative management and network governance literature. This “goals paradox” follows from the implicit assumption in the literature that both congruence and diversity in organizations’ goals positively and negatively affect performance. Goal congruence fosters commitment among participants, and explicit acknowledgment of collective goals is expected to be a necessary condition for effective service delivery. Too much homogeneity in organizations’ goals, however, can lead to competitiveness among participants and reluctance to collaborate. Similarly, diversity in organizational goals is assumed to correlate with diversity in organizational sources of information and resources, and, consequently, to stimulate innovative solutions to policy problems through synergies between these sources. At the same time, too much diversity is expected to result in conflict and non-cooperative behavior.

The most likely conclusion to be drawn from this review is that returns of stakeholder involvement on performance may be positive, but diminishing. It is likely that at low levels of involvement, the marginal benefits exceed the marginal costs of additional partners. At higher levels, however, the additional advantages in terms of access to information, diversity of partners, and policy support may not outweigh the costs. If so, it remains an empirical, and probably context-specific, question how many stakeholders should be involved to maximize performance. We formulate the following hypothesis:

Hypothesis 1: The positive effect of stakeholder involvement on policy performance decreases with the involvement of additional organizations.

Research Context: The SSA

Over the last decade, the Netherlands has seen major decentralization in a number of policy areas (Gilsing, 2007). One of the most recent shifts in central-to-local authority is marked by the SSA. The SSA is an example of so-called “framework settings” that are gaining momentum in the Netherlands (Koppenjan, Kars, & van der Voort, 2009). In this system, the central government establishes the goals and the desired social effects of a policy, while local authorities and their administrations enjoy considerable freedom in formulating and applying the policy.¹

The principal aim of the SSA is to facilitate the independent functioning and social participation of individuals, in particular those who experience physical or mental impediments. These individuals constitute the client population in the present study. Independent functioning is here understood according to three key criteria: being able to manage a household and perform daily routines such as grocery shopping (physical self-reliance), maintaining a personal social network of friends and family (social contacts), and engaging actively in social activities such as volunteering or sports (social participation; Gilsing, Tuynman, van der Veer, & Iedema, 2010). These three subgoals of the SSA constitute the dependent variables in our analysis.

In this article, we focus on the direct relationship between stakeholder involvement and policy performance. We do not attempt to analyze in detail how the involvement of stakeholders affects the development and provision of specific types of services. Considering the wide scope of the SSA, this would be a nearly impossible task. The SSA covers many different subdomains of social policy and a wide range of services. SSA clients characteristically face problems that are complex and multifaceted. For instance, clients who are physically disabled are less able to run a household but are also more likely to be socially isolated and have a lower income as a consequence of being unemployed. The types of services provided under the umbrella of “social support” are therefore manifold. They include household services, informal care, transportation, social activities, adapted housing for disabled individuals, and so on.

Addressing the needs of SSA clients and providing a wide range of services require the coordination of different types of organizations in multidisciplinary diagnostic teams, coordinating platforms, or case-centered projects (Schalk, 2013; cf. Alter, 1990). The SSA encourages, but does not formally require, local governments to develop and maintain relations with local actors at all stages of policy making, assuming that stakeholder involvement will increase policy performance. Specifically, the SSA envisages local governments as policy brokers or “lead organizations” (Provan & Kenis, 2008) that facilitate the exchange of expertise and coordinate interaction.

The specific way in which local governments organize the interactive process of stakeholder involvement in practice can be characterized by an intermediate level of formality (Van Houten, Schalk, & Tuynman, 2010). That is, stakeholder organizations do not have actual political decision-making power, such as veto rights or voting power, but local governments establish official and long-term forums of negotiation, roundtables, or bilateral relations. . The responsible public manager organizes and presides over a forum and enjoys discretion in developing policies and managing the process of inter-organizational service delivery. Nevertheless, the local Council must eventually ratify policies and budget plans.

Thus, stakeholder involvement in local SSA policy making is typically consultative, and organized by public managers. In addition, the involvement of professional and client-interest organizations is a largely separate process: In 95% of all municipalities, local governments have established a separate platform (the ‘SSA Council’) for client-interest organizations, in which only these organizations participate (without professional organizations’ involvement).

Apart from this empirical separation, these two central types of stakeholders also have different core tasks, resources, and incentives to participate in collaborative local policy making. Professional organizations provide governments with access to specialized expertise related to their core organizational tasks (e.g., intramural health care, transportation services, education) as well as to financial or other resources to deliver organization-specific services. In terms of valuable information, they possess performance feedback from their own clientele and evidence-based information on their own methods and best practices. In addition, they have the professional expertise to develop and propose new intervention methods and programs based on their own experiences. Likewise, because professional organizations play an active role in the actual implementation process, generating policy support among these organizations in the early stages of the policy-making process should facilitate their cooperation and coordination. At the same time, however, professional organizations are likely to be heterogeneous due to their particular specializations, and each will likely place a strong emphasis on its own self-interest and survival.

The valuable information that client-interest organizations possess primarily refers to performance feedback in terms of alerting governments when problems occur in practice, as client-interest organizations often maintain strong contacts with the bureaucrats and service providers who work with the clients (Lipsky, 2010; Needham, 2008). At the same time, SSA clients are heterogeneous, varying from the elderly to homeless people, to physically disabled individuals. So even if client-interest organizations can be assumed to behave more idealistically than professional organizations (R. E. Freeman & Reed, 1983), differences in interests between representative organizations of different client groups may still exist, as well as contestation over preferred policies. Empirical evidence for the positive effect of client-interest organization involvement has been found in many studies (Keiser & Miller, 2010; Needham, 2008), though not all of them (Moynihan & Pandey, 2005). Because of the distinct natures of professional and client-interest organizations, we assess Hypothesis 1 separately for these two types of stakeholders in the analyses that follow.

Research Design and Data

To test the hypothesis, we use a unique multi-actor and multilevel data set. The combined data set consists of two nationally representative samples of (a) local governments and (b) SSA clients nested within local governments. The data were collected by the Netherlands Institute for Social Research (in Dutch: “Sociaal en Cultureel Planbureau,” SCP) in 2008 and 2009 as part of a large-scale evaluation project commissioned by the Dutch Ministry of Health, Welfare, and Sports. The goal of these extensive surveys was to evaluate the SSA both in terms of its governance by local governments and the attainment of its social goals at the client level (De Klerk et al., 2010).

The first sample is a full population sample of local public managers. For each municipality in the Netherlands (N = 443), the public manager who is the key coordinator of the local SSA policy-making process was sent a written or internet questionnaire. These questionnaires were sent in the first quarter of 2008 and addressed the process of SSA policy making for the year 2007. Of the 443 public managers contacted, 383 responded, constituting a response rate of 83%. The questionnaire inquired into collaborative practices as well as the content of policies, local SSA goals, and various municipal characteristics.

The second sample is a stratified random sample of SSA clients conducted in the first quarter of 2009. The sample was stratified to ensure that a sufficient number of “G31” municipalities, that is, the 31 largest municipalities in the Netherlands, would be included.² Respondents were sampled from the total population of individuals who applied for individual SSA services in the first quarter of 2008. Of the 5,535 clients in 81 municipalities who were contacted, responses were obtained for 4,055, a response rate of 73%. The surveys were conducted through extensive 45-min personal interviews that addressed the health and health-related problems of SSA clients, social networks, and general socioeconomic conditions.

An important advantage of the sample is that the data on the independent and dependent variables were obtained from different sources: public managers and SSA clients. Therefore, same-source and social-desirability bias is avoided (Spector & Branninck, 2010). Furthermore, the data are longitudinal though not panel data: Stakeholder involvement (the public manager sample) is measured for the year 2007, whereas client outcomes (the SSA client sample) are measured for the year 2008, but client conditions were not measured for 2007. The actual change in client conditions therefore cannot be assessed. Nevertheless, the time lapse between the stakeholder involvement variables (2007) and the actual performance variables (2008) mitigates the risk of reversed causality for these variables and constitutes a significant improvement on cross-sectional studies of stakeholder involvement and performance (cf. Walker et al., 2010). A related characteristic of the sample is that there are no past-performance indicators against which client outcomes in 2008 can be evaluated because the goals of the SSA were finalized in 2007.³

The Independent Functioning of SSA Clients

The SSA aims to improve the level of independent functioning of clients who experience physical or mental impediments in terms of their physical self-reliance, social contacts, and social participation. We operationalize these three social subgoals of the SSA using three different sum scales. The first, physical self-reliance, is a sum scale among several items that refer to two key dimensions of independent functioning described in the SSA legal document (TK, 2004/2005). These items relate strictly to an individual’s physical ability to (a) run a household and (b) to move in and around the house to perform daily tasks (see Appendix Table A1; Cronbach’s α = .72). The two other dependent variables relate to two different aspects of what the SSA regards as the social aspect of independent functioning, namely, (a) maintaining an extensive personal network of family and friends and (b) participating in social activities and being an active member of society. We operationalized social contacts as a sum scale among items that measure how often a client has contact with different types of social groups. In addition, we operationalized social participation as a sum scale among different social activities, such as cultural activities (e.g., going to the theater) and participation in voluntary associations (e.g., sports clubs).

The appendix lists the items used for the different variable constructs at the individual level.⁴ Tables 1 and 2 present the descriptive statistics and correlations for all variables used in the analysis.⁵ Table 1 demonstrates that the degree of physical self-reliance among SSA clients is high, whereas the levels of social contacts and especially social participation are much lower (when comparing the mean values relative to the scale maximum).

Table 1.

Descriptive Statistics for Municipal-Level and Client-Level Variables.

Variables	n	M	SD	Minimum	Maximum
Municipal level
1. Professional involvement	69	14.8	5.2	0.0	20.0
2. Client-interest involvement	69	5.9	2.4	0.0	11.0
3. Political support^a	69	35.8	9.6	20.0	59.5
4. SSA budget (per capita)	69	70.7	17.7	45.1	124.7.2
5. Goal specificity	69	11.3	6.2	0.0	21.0
Client level
6. Physical self-reliance	3,340	8.1	2.0	0.0	12.0
7. Social contacts	3,343	12.4	4.2	0.0	24.0
8. Social participation	3,343	4.2	3.4	0.0	18.0
9. Education	3,343	3.6	1.8	1.0	8.0
10. Household income	3,343	2.3	1.0	1.0	5.0
11. Living alone	3,343	0.5		0.0	1.0
12. Age	3,343	68.0	17.4	2.0	105.0
13. Physical difficulty	3,343	6.0	4.7	0.0	16.0
14. Gender (1 = male)	3,343	0.3		0.0	1.0
15. Informal care	3,343	0.6		0.0	1.0
16. Perceived meeting opportunities	3,343	3.0	0.6	1.0	4.0

Note. Public manager (2008) and SSA client (2009) samples, Netherlands Institute for Social Research (SCP). SSA = Social Support Act.

Data obtained from the Electoral Council (in Dutch: “Kiesraad”).

Table 2.

Correlations for Municipal-Level and Client-Level Variables.

Variables	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Municipal level
1. Professional involvement	1.00
2. Client-interest involvement	.35**	1.00
3. Political support^a	−.22	−.15	1.00
4. SSA budget (per capita)	.21	.06	−.47**	1.00
5. Goal specificity	.35**	.42**	−.24**	.11	1.00
Client level
6. Physical self-reliance	.05	.06	−.03	−.25**	.07	1.00
7. Social contacts	.00	.00	.06	−.27**	.08	.18**	1.00
8. Social participation	−.24**	−.08	.03	−.42**	.02	.22**	.45**	1.00
9. Education	−.05	−.17	−.15	−.02	−.11	.08**	.09**	.17**	1.00
10. Household income	−.11	−.13	.00	−.17	−.15	.04	.11**	.11**	.33**	1.00
11. Living alone	.13	.07	−.19	.19	.21	.09**^b	−.03^b	.06**^b	−.03^b	−.40**^b	1.00
12. Age	−.19	−.01	.23	−.01	.09	.05**	.04**	−.05**	−.11**	−.14**	.29**^b	1.00
13. Physical difficulty	.11	.10	.14	.19	.00	−.40**	−.16**	−.28**	−.07**	−.01	−.11**^b	−.01	1.00
14. Gender (1 = male)	−.06	.24**	−.02	.03	.16	.02^b	−.01^b	−.04**^b	.11**^b	.13**^b	−.14**^c	−.03	.07**	1.00
15. Informal care	−.09	−.12	.06	−.15	−.15	−.14**^b	.08**^b	−.07**^b	−.05**^b	.12**^b	−.21**^c	−.09**^b	.27**^b	.01^c	1.00
16. Perceived meeting opportunities	−.08	−.12	.05	−.17	.08	.34**	.33**	.28**	.04**	.09**	−.00^b	.07**	−.20**	.01^b	−.04**^b

Note. Public manager (2008) and SSA client (2009) samples, Netherlands Institute for Social Research (SCP). Mean value of client-level variables used for correlations with municipal-level variables. SSA = Social Support Act.

The Electoral Council (in Dutch: “Kiesraad”).

Point-biserial correlation for dichotomous and interval variables.

Phi-correlation for dichotomous variables.

p < .05.

Professional and Client-Interest Involvement

In the Public Manager Questionnaire, public managers were asked whether each of 20 different functional types of professional organizations and 11 different functional types of client-interest organizations were involved in the local SSA policy-making process. For client-interest organizations, involvement refers to attending the periodically convening collective SSA-Council (in Dutch: “WMO-Raad”). Much care was taken in constructing the list of organizations. Many roundtable sessions were conducted with representatives from stakeholder organizations in the field across the Netherlands: local governments, professional organizations, and client-interest organizations. The information obtained from these sessions resulted in two lists of organizations that are (a) exhaustive in the sense that all relevant functional organizations are represented and (b) meaningful to public managers in terms of the categorization of these organizations. Furthermore, the lists were cross-validated with the organizations mentioned in the documents issued by the Ministry of Health, Welfare, and Sports, which were designed to advise local governments and their partners regarding different aspects of the SSA.

In the analyses below, we use a sum variable for both types of stakeholders, professional involvement and client-interest involvement. Although great care was taken in constructing the (exhaustive) list of types of organizations, these network measures are more general than some measures in the literature. For example, other network studies take into account contact frequency (e.g., Meier & O’Toole, 2003), and different types of relationship content (e.g., Provan & Lemaire, 2014). In fact, the question on professional involvement did make a distinction between three different categories of content: inform, ask for information, and formally ask for permission. However, as our aim is to explore whether networking in general affects policy performance differently for the two key types of stakeholders, we do not construct separate network scales for the different types of content in the present analysis.⁶

In addition, a recent development in the measurement of network relations is to assess whether public managers distinguish between different dimensions of networking (cf. Torenvlied et al., 2013). Using Mokken (1971) scaling, these studies have empirically demonstrated that, rather than simply networking more or less, public managers distinguish between groups of network partners. For the SSA data, we have likewise assessed whether public managers distinguish between subgroups of professional stakeholder and client-interest organizations. The Mokken analysis on the 20 professional involvement (dichotomous) items shows a clear single networking dimension for professional involvement (H > .51 for all items; where item loadings of >.3 are acceptable [cf. Mokken, 1971]). The analysis on the 11 client-interest items instead shows two separate networking dimensions (H > .36 for all items in both scales): one dimension groups together organizations for social security clients, elderly clients, and those suffering from domestic violence. The other items are grouped together in the second scale, except for individual citizens, which was a nonscalable item. Substantively, these scales are hard to interpret. One would, for example, expect elderly client organizations to be grouped together with voluntary work organizations. For robustness, we have nonetheless executed the causal analyses presented below for the two separate scales as well. The effects of the two separate client-interest networking dimensions are the same compared with the effects of the single dimension presented in the analyses in terms of significance.⁷

Table 3 displays the percentage of public managers who indicate that they have a relationship with each type of organization. Most types of professional organizations are nonprofits or semipublic organizations that fall under different levels of government. The list also includes private-sector organizations (e.g., transportation companies and home-care organizations). From Tables 1 and 3, it can be concluded that local governments are generally active in terms of stakeholder involvement but more so with respect to professional organizations: Table 1 indicates that local governments on average involve nearly 15 out of 20 different types of professional organizations and nearly six out of 11 types of client-interest organizations. Table 3 shows that local governments in particular involve home-care organizations (which typically provide household services), welfare, voluntary work, and informal care professional organizations. The client-interest organizations that are most involved represent the elderly and disabled SSA clients. Arguably, these types of organizations’ greater degree of involvement reflects local governments’ perceived importance of their expertise and resources for successful local implementation of the SSA.

Table 3.

Percentage of Public Managers Indicating the Involvement of Professional and Client-Interest (‘SSA Council’) Organizations in SSA Policy Making (N = 69).

Professional organizations	%	Client-interest organizations	%
Regional governments	38	Informal care clients	62
Transportation companies	33	Voluntary work clients	65
Housing corporations	87	Disabled clients	91
Health care service coordination office	70	Social security clients	52
CIZ, SSA client eligibility indication office	71	Elderly clients	91
Home care organizations	91	Patient associations	58
GGD, primary health care	87	Local neighborhood platforms	42
Residential care homes	84	Homeless, disabled and clients suffering from domestic violence	26
GGZ, primary health care	86	Migrant associations	19
Organizations for disabled clients	87	Individual citizens	42
Financial-aid organizations (excluding banks)	49	Youth associations	36
Welfare organizations	93
Community shelters	70
BJZ, youth health care coordination office	75
Schools	62
Police departments	51
Religious organizations	72
Informal care organizations	94
Voluntary work organizations	93
MEE, administrative aid office	87

Note. Public manager sample (2008), Netherlands Institute for Social Research (SCP). SSA = Social Support Act; CIZ = Centrum Indicatiestelling Zord; GGD = Gemeentelijke Gezondheidsdienst; GGZ = Geestelijke Gezondheidszorg; BJZ = Bureau Jeugdzorg.

Political Support

The literature suggests various factors that constitute opportunities for, and constraints on, stakeholder involvement and its effectiveness (Ansell & Gash, 2008). We incorporate three key control variables at the municipal level. First, the support of elected officials is imperative to the effective management of stakeholder involvement. Empirical studies have demonstrated the positive effect of political support on performance (Moynihan & Pandey, 2005).

To capture political support, we use a measure that defines the percentage of center-right to right-wing party votes as a share of total votes for the municipal Council, which is the local legislative body in the Netherlands. In general, right-wing parties favor the reduction of social security and welfare expenditures. The relative dominance of right-wing parties in local politics should therefore be an indication of the difficulty that public managers experience in designing effective SSA policies and establishing trust and support vis-à-vis local stakeholders.

In the Netherlands, local elections for the municipal council are held every 4 years. The 2006 elections determined the distribution of seats in the municipal councils responsible for developing SSA policies. Local election data were obtained from the Electoral Council (in Dutch: “Kiesraad”), which is the government agency responsible for the election process at all levels of Dutch government. Per municipality, the number of votes is available for each political party. There are eight political parties that also participate in the national elections. These parties can be classified on a left-to-right scale, the most important dimensions of which are redistribution and welfare-state issues (Aarts & Van Der Kolk, 2006).⁸ On average, center-right to right-wing parties comprise 35.8% of the municipal council seats in the sample (see Table 1).⁹

SSA Budget

Our second control variable captures the financial resources of local governments. As O’Toole and Meier (2004) argue, any study on public performance should control for resource capacity. A higher resource capacity is generally assumed to positively affect performance. Additional personnel and monetary resources enable local communities to develop and provide more effective (although not necessarily more efficient) services (Turrini, Cristofoli, Frosini, & Nasi, 2010). Resource capacity has been operationalized in various ways, such as per capita mental health spending (Provan & Milward, 1995), state government funding (Meier & O’Toole, 2003), and other indicators, depending on the specific policy context.

In the Netherlands, local governments are financially dependent upon the national government. In the present research context, the key indicator for local resource capacity is the funding that the national government specifically provides to local governments for SSA implementation. The variable SSA budget captures this source of funding, measured in euros per citizen per year for 2008. Importantly, this budget was established using a formula based on numerous general socioeconomic, financial, and demographic factors, for example, the percentage of low-income households. In general, the SSA budget is lower when municipalities as a whole are characterized by more resources and fewer constraints. Thus, a low SSA budget is in fact an indicator of high community resource capacity.

Goal Specificity

A final important control variable at the municipal level concerns the specificity of SSA policy goals in local implementation programs. These local policy goals are instrumental in attaining the ultimate policy goal of independent functioning. They may refer to budget allocations, eligibility criteria, program development, and so on. Local governments that have clearly defined goals are assumed to be more motivated to perform well (Wright, 2001) and to be able to cooperate more effectively, as their goals and tasks can be more easily communicated and synchronized (Moynihan & Pandey, 2005). The variable goal specificity is a sum scale that captures the extent to which specific goals (both qualitative and quantitative) have been formulated for different subdomains of the SSA (see Appendix Table A4; Cronbach’s α = .88).

Client-Level Control Variables

Apart from the factors that potentially condition the effectiveness of stakeholder involvement at the municipal level, it is important to take into account individual, client-level characteristics that relate to policy performance. Most studies on public performance account for variations in client characteristics that indicate the difficulty of service delivery (Meier & O’Toole, 2003; Provan & Milward, 1995). Specifically, some SSA clients may find independent functioning to be more difficult than others. If the individual-level factors that stimulate or inhibit independent functioning are systematic across municipalities, these factors constitute composition effects that must be controlled for in our analysis.

In the present context, we control for a number of client-level characteristics that are expected to relate to policy performance in terms of independent functioning. The three performance indicators are expected to be positively related to education (ranging from 1 = no education to 8 = a university degree), household income (ranging from 1 = less than 1,000 euros net per month to 5 = over 3,000 euros net per month), informal care (a dummy variable indicating whether a client receives nonprofessional help from family, friends, and so on to facilitate their independent functioning), and perceived meeting opportunities (a 4-point variable indicating whether a client judges the opportunities to meet other people and maintain social contacts as 1 = highly inadequate to 4 = highly adequate). The variable perceived meeting opportunities also captures the perceived personal need to engage in social activities to some extent. In addition, all indicators of independent functioning are expected to be negatively related to living alone, age, and physical difficulty. The variable physical difficulty captures the degree to which clients experience physical difficulties in performing daily tasks. It is a client-level sum variable of eight three-category items that ask whether an SSA client is physically able—prior to any service delivery and without help from others—to perform a certain activity related to independent functioning (see Appendix Table A5; Cronbach’s α = .91). Finally, gender is incorporated as a control variable with no expected direction in its effect.

Analysis and Results

Below, we test Hypothesis 1 by estimating the effects of the involvement of professional and client-interest organizations separately on three indicators of independent functioning: physical self-reliance, social contacts, and social participation. We evaluate three multilevel generalized linear models for each dependent variable. The main effects model includes only the main effects of professional and client-interest organizations’ involvement and the client-level controls. Second, the diminishing returns model adds a quadratic term for each type of stakeholder to test for diminishing returns. Finally, the full model adds the municipal-level control variables to either the main effects model or the diminishing returns model, depending on which model best fits the data.

The multilevel design offers two distinct advantages. First, it allows for an assessment of the extent to which individual SSA clients’ independent functioning is correlated with the fact that they live in the same municipality (as characterized by the degree of stakeholder involvement, among other municipal-level factors) as compared with the myriad individual-level factors that may likewise affect their level of independent functioning. Second, multilevel models allow for more statistically efficient models compared with ordinary regression with aggregated client-level variables because of their enhanced power and reduced likelihood of producing biased standard errors (Hox, 2002).

A methodological complication of the analysis concerns the stratification of the SSA client sample on G31 municipalities. To account for the unequal selection probability of G31 municipalities (municipal level) and of SSA clients living in G31 municipalities (client level), the estimates are corrected with two-level sampling weights (Asparouhov, 2004). This method implies that we do not include municipality size (i.e., the number of inhabitants) as a separate control variable in the multivariate analyses (municipal size is strongly correlated with the G31 variable (ρ_pb = .88; p < .05). All variables except dummy variables were grand mean centered to facilitate the interpretation of coefficients (Snijders & Bosker, 1999).

Multivariate Results

Table 4 shows the results of the multilevel analyses. A first observation is that the intraclass correlations (not shown in Table 4) are significant for all three dependent variables: 11.1% for physical self-reliance, 12.0% for social contacts, and 4.5% for social participation.¹⁰ These findings indicate that there is a significant amount of between-municipality variation in independent functioning that can be attributed to municipal-level factors.

Table 4.

Multilevel Regression Analysis of the Independent Functioning of SSA Clients in Dutch Municipalities.

	Physical self-reliance			Social contacts			Social participation
	Main effects model	Diminishing returns model	Full model	Main effects model	Diminishing returns model	Full model	Main effects model	Diminishing returns model	Full model
Municipal level
Professional involvement	.021* (.013)	−.008 (.025)	.023* (.013)	.007 (.040)	−.100* (.062)	−.097 (.064)	−.040* (.021)	−.070** (.036)	−.034* (.019)
Client-interest involvement	.020 (.020)	.027 (.023)	.002 (.020)	.046 (.071)	.053 (.068)	−.002 (.064)	.040 (.035)	.042 (.039)	.008 (.032)
Political support			−.003 (.008)			−.002 (.022)			−.008 (.012)
SSA budget (per capita)			−.009** (.004)			−.016 (.022)			−.020*** (.006)
Goal specificity			.011 (.013)			.047 (.040)			.015 (.018)
Professional involvement squared		−.003 (.002)			−.012** (.006)	−.012 ** (.005)		−.003 (.004)
Client-interest involvement squared		.002 (.006)			−.007 (.017)	−.006 (.016)		−.002 (.010)
Client level
Education	.037* (.021)	.037* (.022)	.038* (.022)	.146*** (.049)	.145*** (.049)	.146*** (.048)	.240*** (.035)	.240*** (.035)	.241*** (.035)
Household income	.029 (.036)	.028 (.036)	.028 (.037)	.259*** (.069)	.256*** (.069)	.255*** (.070)	.229*** (.072)	.227** (.072)	.225*** (.072)
Living alone	.155** (.078)	.154** (.077)	.155** (.078)	−.028 (.123)	−.032 (.123)	−.031 (.123)	.526*** (.161)	.524*** (.161)	.530*** (.161)
Age	.003 (.002)	.003 (.002)	.003 (.002)	.014** (.006)	.014** (.006)	.014*** (.006)	−.014*** (.004)	−.014 *** (.004)	−.014*** (.004)
Physical difficulty	−.148*** (.012)	−.148*** (.012)	−.147*** (.012)	−.112*** (.016)	−.112*** (.016)	−.112** (.016)	−.166*** (.015)	−.167*** (.015)	−.164*** (.015)
Gender (1 = male)	.184*** (.064)	.184*** (.064)	.184*** (.064)	−.337** (.158)	−.337** (.159)	−.337** (.159)	−.428*** (.103)	−.428*** (.103)	−.425*** (.102)
Informal care	−.070 (.078)	−.068 (.077)	−.073 (.077)	.984*** (.144)	.989*** (.144)	.987*** (.144)	−.051 (.123)	−.049 (.123)	−.057 (.124)
Perceived meeting opportunities	.740*** (.081)	.740 *** (.081)	.740*** (.081)	1.901 *** (.195)	1.898 *** (.137)	1.896 *** (.137)	1.244 *** (.098)	1.242 *** (.098)	1.240 *** (.098)
Constant	8.012*** (.103)	8.080*** (.122)	8.077*** (.140)	11.905*** (.195)	12.232*** (.268)	12.505*** (.412)	4.070*** (.186)	4.164*** (.231)	4.140*** (.231)
Variance components
Municipal-level residual variance	0.260*** (.058)	.249*** (.055)	.240*** (.059)	1.806*** (.359)	1.648*** (.367)	1.516*** (.343)	.484*** (.121)	.471*** (.112)	.384*** (.096)
Client-level residual variance	2.625*** (.148)	2.625*** (.148)	2.625*** (.148)	13.042*** (.429)	13.042*** (.429)	13.042*** (.429)	9.405*** (.297)	9.405*** (.297)	9.404*** (.297)
n (municipal level)	69	69	69	69	69	69	69	69	69
n (client level)	3,340	3,340	3,340	3,343	3,343	3,343	3,343	3,343	3,343
Wald test (df)	280.8 (10)***^a	2.6 (2)^b	8.94 (3)*^b	526.7 (10)***^a	5.0 (2)*^b	6.6 (3)*^c	675.8 (10)***^a	1.0 (2)^b	14.2 (3)***^b

Note. Public manager (2008) and SSA client (2009) samples, Netherlands Institute for Social Research (SCP). Unstandardized coefficients (standard errors) reported. All variables except dichotomous variables were grand mean centered. Standard errors were adjusted for stratification on G31 membership. SSA = Social Support Act.

Compared with empty model.

Compared with main effects model.

Compared with negative returns model.

p < .10. **p < .05. ***p < .01.

Hypothesis 1 states that the positive effect of stakeholder involvement on policy performance decreases with the involvement of additional organizations. The hypothesis is only partly confirmed. First, the effect of professional involvement on social contacts is indeed inverted U shaped: the interaction term is negative and significant in the full model (β = −.012; p < .05). This finding indicates that at higher levels of professional involvement, the effect diminishes, whereas it is positive at lower levels. Second, the results for physical self-reliance and social participation only partly confirm the hypothesis: the effect of professional involvement on physical self-reliance is linear and positive (β = .023; p < .10), and the effect on social participation is linear and negative (β = −.034; p < .10). And finally, Hypothesis 1 is not confirmed for client-interest organizations: their inclusion by local public managers has no effect on policy performance at all in terms of all three indicators of independent functioning.

To illustrate the interpretation of effects, let us consider the effect of professional involvement on social contacts in the full model. The slope for the effect can be computed for any value by taking the first derivative of the equation and substituting values. Because the estimates reported in Table 4 are grand mean centered, and the average for professional involvement is 14.8 (see Table 1), the estimates in the full model show that at the average level of professional involvement, a one-unit increase (i.e., one additional type of organization involved in local policy making) is associated with a decrease in the average level of social contacts of (−.097 −.024 × 14.8 =) −.45. This finding is substantial, given the 20-point maximum range in professional involvement.

The findings should be interpreted with caution. Figure 1 plots the bivariate relationships between professional stakeholder involvement and each of the dependent variables. Figure 1 shows that the professional involvement variable is skewed, with many local public managers involving many types of organizations in policy making. This finding means that with few cases at the lower end of the involvement scale, the relationship depicted in this area of the graph is unstable, and thus we are hesitant to draw strong conclusions.

Figure 1.

The independent functioning of SSA clients in terms of their (a) physical self-reliance, (b) social contacts, and (c) social participation as a function of the Dutch local governments’ involvement of professional organizations.

The most robust conclusion is that for social contacts and social participation, the returns of professional organizations’ involvement for performance are not only diminishing but even negative at higher levels of involvement. To illustrate the robustness of this result even further, we fitted a regression line in Figure 1 for only those municipalities that included more than 10 types of organizations (the scale median) based on the full model. With even less statistical power, the coefficients are negative and significant in both cases.

Finally, we briefly turn to the key findings for the control variables. At the municipal level, only the SSA budget that local governments receive affects performance. The effect is negative on physical self-reliance (β = −.009; p < .05) and social participation (β = −.020; p < .01). Given the fact that local governments receive a higher budget based on the degree to which they face “problematic” socioeconomic and demographic conditions, these results suggest that either the budget is not sufficient, or local governments do not use it as efficiently as possible. In addition, at the individual level, most effects are as expected. Perhaps surprisingly, access to informal care only improves social contacts (β = .984; p < .01). Also, SSA clients who live alone participate more (β = .530; p < .01) and are more physically self-reliant (β = .155; p < .01), which suggests that living alone is not so much a risk factor for this sample, but rather indicates high independence.

Conclusion and Discussion

The preceding sections presented a systematic analysis of policy performance—in terms of client outcomes—in Dutch local government policy making. The key hypothesis predicted a positive effect of the involvement of stakeholder organizations on policy performance, with diminishing returns at higher levels of involvement. Based on the analysis, we can draw a number of conclusions. First, the results indicate that the involvement of professional organizations is related to policy performance, whereas the collective involvement of client-interest organizations is not.¹¹ A possible context-specific explanation for the absence of a direct effect for client-interest involvement is that the separate platform for client-interest organizations (the SSA-Council) has primarily been established by local governments to legitimize the local SSA policy-making process (cf. Edelenbos & Klijn, 2006; Pierre, 2000), whereas the impact of client-interest organizations on policy change and their involvement in actual service delivery remains limited (Tatenhove et al., 2010). Nevertheless, this result is surprising and deserves further investigation, especially because most studies actually find that client-interest organizations—because of their expected lack of survival motives and idealistic predisposition—can be especially beneficial for governments to involve (Keiser & Miller, 2010; Needham, 2008).

Second, there is a consistent observation across the two different “social” indicators of independent functioning, that is, the social contacts and social participation of SSA clients: Too much involvement of professional organizations negatively affects policy performance. This observation is confirmed by the presence of a negative interaction effect on social contacts and a negative effect on social participation. This negative effect of professional involvement at higher levels of involvement indicates that Dutch local governments currently engage in too much involvement, given the high average level of professional involvement (see Table 1). One cautionary note must be made with respect to this conclusion. Because the data are not true panel data, it is possible that the unobserved client conditions of 2007 caused local governments to increase stakeholder involvement in the same year. Although we can reasonably assume that the client-level control variables account for these prior conditions to some extent, only true panel data can exclude this type of reversed causality.

Third, the effect of professional involvement differs across different sub-indicators of policy performance. The results suggest that professional involvement actually positively affects the “physical” conditions of SSA clients (although the effect is weak), while it negatively affects their “social” conditions at higher levels of involvement. The simplest explanation for this difference is that social problems are more complex, while solutions and interventions are presumably more numerous and ambiguous, thus leaving more room for dispute among service delivery organizations and local governments. Another explanation may be that the policy problems addressed in the local collaborative platforms first addressed the physical conditions of SSA clients, leaving social problems to be addressed later in the process. The SCP’s general evaluation of the SSA concludes that the social contacts of SSA clients are an underdeveloped area of local policy making (De Klerk et al., 2010).

In summary, the analysis presented here contributes to the current debate over stakeholder involvement by providing empirical evidence for the expectation of decreasing returns for stakeholder involvement at higher levels of involvement (Hicklin et al., 2008) and the dependency of the effect of stakeholder involvement on the types of stakeholder organizations involved (cf. Torenvlied et al., 2013; Walker et al., 2010). This article likewise makes an important methodological contribution. It is one of the few studies that uses representative, longitudinal data to analyze the effects of stakeholder involvement on policy performance at the client level, across local settings that operate under the same regulatory program (Ansell & Gash, 2008; for exceptions, see, for example, Meier & O’Toole, 2003; O’Toole & Meier, 1999).

This article did not intend to reveal all possible mechanisms that may underlie a systematic analysis of stakeholder involvement and policy performance. An important avenue for future research is to further develop theory on the complex relationship between the scope of stakeholder involvement, taking into account the interrelatedness of the SSA goals, and such factors as goal conflict (cf. Vangen & Huxham, 2012), the managerial strategies of participating organizations (Walker, O’Toole, & Meier, 2007), the behavior of street-level employees responsible for the actual delivery of services within participant organizations (Lipsky, 2010), prior cooperation and conflict between partners (Ansell & Gash, 2008), and leadership strategies (Huxham & Vangen, 2000).

Another avenue for future research concerns the use of alternative, and more fine-grained, measures of network relations to study the relationship between stakeholder involvement and (local) policy performance. The measures used in the present analysis did not capture potentially relevant theoretical aspects of network management that may shed a different light on the results. For example, we did not capture who started the relationship, nor did we measure the strength of relationships and the actual resources (e.g., expertise, financial aid) and constraints (e.g., free riding behavior) that local organizations bring to the policy-making process. Also, hybrid methods of free recall and roster rating of partners can provide a more accurate picture of a network than the simple roster technique we used for types of partners (Henry, Lubell, & McCoy, 2012). And finally, more fine-grained involvement measures can also improve our theoretical understanding of why public managers differentiate between networking behavior for different groups of actors (cf. Torenvlied et al., 2013).

The results have important practical implications for public managers. The enduring emphasis that researchers and practitioners have placed on the benefits of networking with external partner organizations (Kapucu et al., 2014) may have prompted managers to become overly focused on managing their relationships with external organizations. In the case of the SSA, the central government’s encouragement (De Klerk et al., 2010) to include as many stakeholders as possible in the SSA policy-making process may be ill-advised. The analysis presented here suggests that there are limits to the effectiveness of professional involvement, while the involvement of client-interest organizations may have no effect on policy performance at all.

Footnotes

Appendix

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

Author Biography

Jelmer Schalk is an assistant professor in the Institute of Public Administration of Leiden University. His fields of interest are public management and inter-organizational networks, with a focus on organizational effectiveness and government performance. His work has appeared in The Journal of Public Administration Research and Theory, The American Review of Public Administration, and Local Government Studies.

References

Aarts

Van Der Kolk

(2006). Understanding the Dutch “no”: The euro, the east, and the elite. Political Science & Politics, 39, 243-246.

Agranoff

(2003). Leveraging networks: A guide for public managers working across organizations. Washington, DC: IBM Endowment for the Business of Government.

Agranoff

(2006). Inside collaborative networks: Ten lessons for public managers. Public Administration Review, 66, 56-65.

Akkerman

Torenvlied

(2011). Managing the agency environment: Effects of network ambition on agency performance. Public Management Review, 13, 159-174.

Alter

(1990). An exploratory study of conflict and coordination in interorganizational service delivery systems. Academy of Management Journal, 33, 478-502.

Ansell

(2015). When collaborative governance scales up: Lessons from global public health about compound collaboration. Policy & Politics, 43, 391.

Ansell

Gash

(2008). Collaborative governance in theory and practice. Journal of Public Administration Research and Theory, 18, 543-571.

Asparouhov

(2004). Weighting for unequal probability of selection in multilevel modeling. Retrieved from http://www.statmodel.com/

Berardo

(2009). Generalized trust in multi-organizational policy arenas: Studying its emergence from a network perspective. Political Research Quarterly, 62, 178-189.

10.

Bingham

L. B.

Nabatchi

O’Leary

(2005). The new governance: Practices and processes for stakeholder and citizen participation in the work of government. Public Administration Review, 65, 547-558.

11.

Bovaird

(2007). Beyond engagement and participation: User and community coproduction of public services. Public Administration Review, 67, 846-860.

12.

Boyne

G. A.

(2003). What is public service improvement? Public Administration, 81, 221-228.

13.

Boyne

G. A.

Meier

K. J.

O’Toole

L. J.

Jr. Walker

R. M.

(2006). Public service performance: Perspectives on measurement and management. Cambridge, UK: Cambridge University Press.

14.

Boyne

G. A.

Walker

R. M.

(2010). Strategic management and public service performance: The way ahead. Public Administration Review, 70, S185-S192.

15.

Bryson

J. M.

Crosby

B. C.

Stone

M. M.

(2006). The design and implementation of cross-sector collaborations: Propositions from the literature. Public Administration Review, 66, 44-55.

16.

Coglianese

(1997). Assessing consensus: The promise and performance of negotiated rulemaking. Duke Law Journal, 46, 1255-1349.

17.

Damgaard

Torfing

(2010). Network governance of active employment policy: The Danish experience. Journal of European Social Policy, 20, 248-262.

18.

De Klerk

Gilsing

Timmermans

(Eds.). (2010). Op weg met de Wmo: Evaluatie van de Wet maatschappelijke ondersteuning 2007-2009 [Ahead with the SSA: Evaluation of the Social Support Act 2007-2009]. The Hague, The Netherlands: SCP.

19.

Denters

van Heffen

Huisman

Klok

P. J.

(Eds.). (2003). The rise of interactive governance and quasi markets. Dordrecht, The Netherlands: Kluwer Academic Press.

20.

Edelenbos

Klijn

E. H.

(2006). Managing stakeholder involvement in decision making: A comparative analysis of six interactive processes in the Netherlands. Journal of Public Administration Research and Theory, 16, 417-446.

21.

Favero

Meier

K. J.

O’Toole

L. J.

Jr. (2014). Goals, trust, participation, and feedback: Linking internal management with performance outcomes. Journal of Public Administration Research and Theory. Advance online publication. doi:10.1093/jopart/muu044

22.

Feldman

M. S.

Khademian

A. M.

Ingram

Schnieder

A. S.

(2006). Ways of knowing and inclusive management practices. Public Administration Review, 66, 89-99.

23.

Flynn

Kroeger

(2003). Can policy learning really improve implementation? Evidence from Irish responses to the Water Framework Directive. Environmental Policy and Governance, 3, 150-163.

24.

Freeman

Langbein

L. I.

(2000). Regulatory negotiation and the legitimacy benefit. NYU Environmental Law Journal, 9, 60-151.

25.

Freeman

R. E.

Reed

D. L.

(1983). Stockholders and stakeholders: A new perspective on corporate governance. California Management Review, 25, 88-106.

26.

Furlong

S. R.

Kerwin

C. M.

(2005). Interest group participation in rule making: A decade of change. Journal of Public Administration Research and Theory, 15, 353-370.

27.

Gilliam

Davis

Barrington

Lacson

Uhl

Phoenix

(2002). The value of engaging stakeholders in planning and implementing evaluations. AIDS Education and Prevention, 14(Suppl. A), 5-17.

28.

Gilsing

(2007). Intergovernmental relations and the effectiveness of local governance: The case of Dutch youth policy. International Review of Administrative Sciences, 73, 45-64.

29.

Gilsing

Tuynman

van der Veer

Iedema

(2010). De hoofdlijnen van het gemeentelijk Wmo-beleid. In de Klerk

Gilsing

Timmermans

(Eds.), Op weg met de Wmo: Evaluatie van de Wet maatschappelijke ondersteuning 2007-2009 [Outline of the municipal SSA-policy. Ahead with the SSA: Evaluation of the Social Support Act 2007-2009] (pp. 46-68). The Hague, The Netherlands: SCP.

30.

Head

B. W.

(2008). Three lenses of evidence-based policy. Australian Journal of Public Administration, 67, 1-11.

31.

Henry

A. D.

Lubell

McCoy

(2012). Survey-based measurement of public management and policy networks. Journal of Policy Analysis and Management, 31, 432-452.

32.

Hicklin

O’Toole

L. J.

Jr. Meier

K. M.

(2008). Serpents in the sand: Managerial networking and nonlinear influences on organizational performance. Journal of Public Administration and Theory, 18, 253-273.

33.

Hill

C. J.

Lynn

L. E.

Jr. (2005). Is hierarchical governance in decline? Evidence from empirical research. Journal of Public Administration Research and Theory, 15, 173-195.

34.

Hox

(2002). Multilevel analysis: Techniques and applications. Mahwah, NJ: Lawrence Erlbaum.

35.

Huxham

Vangen

(2000). Leadership in the shaping and implementation of collaboration agendas: How things happen in a (not quite) joined-up world. Academy of Management Journal, 43, 1159-1175.

36.

Kamer

(2004/2005). Nieuwe regels betreffende maatschappelijke ondersteuning (Wet maatschappelijke ondersteuning) (Memorie van Toelichting, Tweede Kamer, vergaderjaar 2004/2005, 30131, nr. 3) [New regulations regarding social support’ Social Support Act; Explanatory Memorandum, Second Chamber (Lower House)].

37.

Kapucu

Khosa

(2014). The state of network research in public administration. Administration & Society. Advance online publication. doi: 10.1177/0275074015577110

38.

Keiser

L. R.

Miller

S. M.

(2010). The impact of organized interests on eligibility determination: The case of veterans’ disability compensation. Journal of Public Administration Research and Theory, 20, 505-531.

39.

Klijn

E. H.

Steijn

Edelenbos

(2010). The impact of network management on outcomes in governance networks. Public Administration, 88, 1063-1082.

40.

Koppenjan

Kars

van der Voort

(2009). Vertical politics in horizontal policy networks: Framework setting as coupling arrangement. The Policy Studies Journal, 37, 769-792.

41.

Lavertu

Weimer

D. L.

(2011). Federal advisory committees, policy expertise, and the approval of drugs and medical devices at the FDA. Journal of Public Administration Research and Theory, 21, 211-237.

42.

Lipshitz

Popper

(1996). Building learning organizations: The design and implementation of organizational learning mechanisms. Journal of Applied Behavioral Science, 32, 292-305.

43.

Lipsky

(2010). Street-level bureaucracy: Dilemmas of the individual in public services. New York, NY: Russell Sage.

44.

Meier

K. J.

O’Toole

L. J.

Jr. (2003). Public management and educational performance: The impact of managerial networking. Public Administration Review, 63, 689-699.

45.

Mokken

R. J.

(1971). A theory and procedure of scale analysis. With applications in political research.

46.

Hague, The Netherlands: Mouton.

47.

Moynihan

D. P.

(2005). Goal-based learning and the future of performance management. Public Administration Review, 65, 203-216.

48.

Moynihan

D. P.

Pandey

S. K.

(2005). Testing how management matters in an era of government by performance management. Journal of Public Administration Research and Theory, 15, 421-439.

49.

Moynihan

D. P.

Provan

K. G.

Lemaire

R. H.

(2012). Core concepts and ideas for understanding public sector organizational networks: Using research to inform scholarship and practice. Public Administration Review, 72, 638-648.

50.

Needham

(2008). Realising the potential of co-production: Negotiating improvements in public services. Social Policy and Society, 7, 221-231.

51.

Newig

Fritsch

(2009). Environmental governance: Participatory, multi-level—and effective? Environmental Policy and Governance, 19, 197-214.

52.

Nutley

S. M.

Walter

Davies

H. T. O.

(2007). Using evidence: How research can inform public services. Bristol, UK: Policy Press.

53.

Nutt

Backoff

R. W.

(2002). Why decisions fail. San Francisco, CA: Jossey-Bass.

54.

O’Toole

L. J.

Jr. Meier

K. J.

(1999). Modeling the impact of public management: The implications of structural context. Journal of Public Administration Research and Theory, 9, 505-526.

55.

O’Toole

L. J.

Jr. Meier

K. J.

(2004). Public management in intergovernmental networks: Matching structural networks and managerial networking. Journal of Public Administration Research and Theory, 14, 469-494.

56.

Percival

G. L.

(2009). Exploring the influence of local policy networks on the implementation of drug policy reform: The case of California’s Substance Abuse and Crime Prevention Act. Journal of Public Administration Research and Theory, 19, 795-815.

57.

Pierre

(Ed.). (2000). Debating governance. Oxford, UK: Oxford University Press.

58.

Provan

K. G.

Kenis

(2008). Modes of network governance: Structure, management, and effectiveness. Journal of Public Administration Research and Theory, 18, 229-252.

59.

Provan

K. G.

Milward

H. B.

(1995). A preliminary theory of interorganizational network effectiveness: A comparative study of four community mental health systems. Administrative Science Quarterly, 40, 1-33.

60.

Provan

K. G.

Sydow

(2008). Evaluating inter-organizational relationships. In Cropper

Ebers

Huxham

Ring

P. S.

(Eds.), The Oxford handbook of inter-organizational relations (pp. 691-716). Oxford, UK: Oxford University Press.

61.

Rovan

K. G

Lemaire

R. H.

(2015). Positional embeddedness in a community source software development project network: The importance of relationship intensity. R&D Management, 45(5), 440–457.

62.

Rydin

Pennington

(2000). Public participation and local environmental planning: The collective action problem and the potential of social capital. Local Environment, 5, 153-169.

63.

Schalk

(2013). Interorganisational relations and goal consensus: An exploratory study in two Dutch service delivery networks. Local Government Studies, 39, 853-877.

64.

Schalk

Torenvlied

Allen

(2010). Network embeddedness and public agency performance: The strength of strong ties in Dutch higher education. Journal of Public Administration Research and Theory, 20, 629-653.

65.

Scholl

H. J.

(2001). Applying stakeholder theory to e-government: Benefits and limits. In Schmid

Stanoevska-Slabeva

Tschamme

(Eds.), Proceedings of the first IFIP conference on e-commerce, e-business and e-government (pp. 735-747). London, England: Kluwer Academic.

66.

Snijders

T. A. B.

Bosker

R. J.

(1999). Multilevel analysis: An introduction to basic and advanced multilevel modelling. London, England: Sage.

67.

Spector

P. E.

Branninck

M. T.

(2010). Common method issues: An introduction to the feature topic in organizational research methods. Organizational Research Methods, 13, 403-406.

68.

Sullivan

Skelcher

(2002). Working across boundaries: Collaboration in public services. Houndmills, UK: Palgrave Macmillan.

69.

Tatenhove

van Edelenbos

Klok

P. J.

(2010). Power and interactive policy-making: A comparative study of power and influence in 8 interactive projects in the Netherlands. Public Administration, 88, 609-626.

70.

Thomas

J. C.

Poister

T. H.

(2009). Thinking about stakeholders of public agencies: The Georgia Department of Transportation stakeholder audit. Public Organization Review, 9, 67-82.

71.

Thomas

J. C.

Poister

T. H.

Ertas

(2010). Customer, partner, principal: Local government perspectives on state agency performance in Georgia. Journal of Public Administration Research and Theory, 20, 779-799.

72.

Torenvlied

Akkerman

Meier

K. J.

O’Toole

L. J.

Jr. (2013). The multiple dimensions of managerial networking. American Review of Public Administration, 43, 251-272.

73.

Turrini

Cristofoli

Frosini

Nasi

(2010). Networking literature about determinants of network effectiveness. Public Administration, 88, 528-550.

74.

Vangen

Hayes

J. P.

Cornforth

(2015). Governing cross-sector, inter-organizational collaborations. Public Management Review, 17, 1237-1260.

75.

Vangen

Huxham

(2012). The tangled web: Unraveling the principle of common goals in collaborations. Journal of Public Administration Research and Theory, 22, 731-760.

76.

Van Houten

Schalk

Tuynman

(2010). Samenwerking en sturing in gemeentelijke beleidsprocessen. In de Klerk

Gilsing

Timmermans

(Eds.), Op weg met de Wmo: Evaluatie van de Wet maatschappelijke ondersteuning 2007-2009 [Collaboration and governance in municipal policy processes. Ahead with the SSA: Evaluation of the Social Support Act 2007-2009] (pp. 101-124). The Hague, The Netherlands: Netherlands Institute for Social Research.

77.

Walker

R. M.

Andrews

(2015). Local government management and performance: A review of evidence. Journal of Public Administration Research and Theory, 25, 101-133.

78.

Walker

R. M.

Boyne

G. A.

Brewer

G. A.

(Eds.). (2010). Public management and performance: Research directions. Cambridge, UK: Cambridge University Press.

79.

Walker

R. M.

O’Toole

L. J.

Jr. Meier

K. J.

(2007). It’s where you are that matters: The networking behaviour of English local government officers. Public Administration, 85, 739-756.

80.

Wright

B. E.

(2001). Public sector work motivation: Review of current literature and a revised conceptual model. Journal of Public Administration Research and Theory, 11, 559-586.