Abstract
Quebec's Municipal Management Indicators embodies what Hood (2007) described as an intelligence regime. This research tries to determine if the design of the municipal intelligence performance regime in Quebec, Canada, delivered the expected results. To answer that question, publicly available official documents, minutes of meetings, and survey data are used. The story of Quebec's regime offers a counter-example to Pollitt and colleagues’ (2010) theory that once in place, performance regimes follow a logic of escalation. The municipal intelligence regime in Quebec never moved from formative to summative; from intelligence to targets and rankings. The experience in that Canadian province offers support to Hood's (2007) model about the shortcomings of intelligence regimes.
Points for practitioners
The case study of a performance regime details an effort with few demands on participants. It is argued that the documented shortcomings are the result of the strategic path initially taken by decision makers, not the result of their later decisions and adjustments. Shielded from public scrutiny and without sanctions from the provincial government, most municipal managers chose not to use the indicators, not to include them in budgets and annual reports, not to compare themselves to others, and not to set targets for themselves. In a mandated regime with bottom-up and voluntary approaches, most municipalities effectively opted out.
Introduction
How can we design a performance management regime that achieves the expected objectives? Many American and British studies include proposed steps that should be followed to achieve benefits expected from performance management at the initiative (Likierman, 1993: 15–22; Roberts, 1996: 376–377; Audit Commission, 2000: 17; Jackson, 2005: 35–36) and system level (Ghobadian and Ashworth, 1994: 50; Straight, 2000: 513). Many more studies offer descriptive and critical assessments of national/federal/provincial performance management frameworks such as the American Government Performance and Results Act and the Program Assessment Rating Tool (Radin, 1998, 2000, 2006; Gueorguieva et al., 2009; Kasdin, 2010), the French Loi organique sur les lois de finances (Mazouz and Rochet, 2005; Calmette, 2006; Trosa, 2006; Corbett, 2010), or Quebec's Loi sur l’administration publique (Bourgault, 2004; Mazouz and Rochet, 2005; Gilbert, 2009). Although there is a vast literature on the British and American experiences, there is a paucity of studies documenting performance regimes at the local level in French-speaking jurisdictions. 1
Performance measurement frameworks covering many comparable entities are often referred to as performance regimes. A performance regime is defined ‘as the range of institutional actors who can influence the performance of an organisation through the use of formal authority, resources control or information coupled with the way in which these institutions actually use the powers available to them …’ (Talbot and Wiggan, 2010: 62). The actual performance interventions, or levers, used to influence the performance of public organizations can take many forms: performance contracts, imposed targets, or comparative ‘league tables’ (Talbot, 2008: 1570). The levers described by Talbot (2008: 1570) overlap with the three different uses of performance indicators identified by Hood (2007: 100–101). The different uses of indicators originate from the intended effects of performance regimes. Performance indicators used as targets and rankings would be utilized to pursue performance improvements, respectively through raising a limited number of standards and ‘sweating and stretching'. Performance indicators can also be used as intelligence, when the goal of the performance regime is to develop learning capacities and diagnostic powers (Hood, 2007: 101).
The purpose of this study is threefold. First, it seeks to describe a performance measurement regime at the municipal level in a Canadian and Francophone context. We know little about performance management in these two settings. Second, it offers empirical support for the theoretical expectations (Hood, 2007: 96) of an intelligence regime. Third, it offers an example disputing Pollitt and colleagues’ (2010: 19, 26) recent theory of policy punctuation. These authors assert that once in place, the technical and political factors endogenous to performance measurement regimes follow a logic of escalation. The Quebec case will not invalidate the possibility that in some sectors, like Pollitt and colleagues’ (2010) case studies of health services, it is possible that quantitative performance measurement systems can exert a logic of their own, leading to an escalation toward complexity. However, the Quebec case will show that for a municipal benchmarking regime, the opposite may be observed, even after ten years. All in all, this study's contributions are to present a chronicle of a municipal performance regime's implementation, and to offer an empirical testing ground for two theories concerning performance regimes. This article does not aim at to be a normative and prescriptive case study.
The rest of this article is organized as follows. First, the data, methods and rationale will be presented. Second, Quebec's Municipal Management Indicators will be introduced. This section will include the design of this performance regime. The evolution of the benchmarking regime's objectives will be laid out. Third, an analysis will be put forward to see if the regime's objectives were achieved. Then, the implications of the results of this intelligence regime will be discussed.
Data, methods, and rationale
Two sources of data are used in this study. First, an overview of Quebec's mandatory municipal performance regime will be presented. Publically available official documents, and minutes of the meetings of the Partners on Municipal Management Indicators Committee's (hereafter Committee, as it is known at present) meetings from 1999 to 2010 constitute the data used for this approach. There are meeting minutes for all of the 83 meetings held by the Committee from the consultation and design stage of the Municipal Management Indicators in 1999, until 2010. The objectives are contained either in publically available official documents or in the minutes of meetings. This archival data also include Committee members’ evaluations of the system at different times during its existence.
Quebec municipalities by population, in 2010, including survey participation
Source: Adapted from MAMROT (2010c).
Quebec's Municipal Management Indicators
Continuity in the design of Quebec's Municipal Management Indicators regime
Most of the Municipal Management Indicators’ features have remained constant since the beginning of its planning stage in 1999, its implementation in 2004, all the way through to 2010. The Committee is the governing body of the Municipal Management Indicators regime. It is a collegial and consensual umbrella organization of professional associations at the municipal level, local elected officials, and the Ministry of Municipal Affairs, Regions and Territorial Use (MAMROT in French, hereafter Ministry). From the very first meeting in October 1999, it was decided that the Committee would work on a collegial and consensual basis. It was agreed that the Ministry could act as a mediator and a tie-breaker (Table sur les indicateurs de performance municipaux, 1999a: 7), even if it is the financier of the regime.
Using Kuhlmann's (2010: 333–334) typology of performance systems in Europe, Quebec's Municipal Management Indicators regime is ‘external’, as it applies to many organizations, and ‘horizontal’, since it includes organizations at the same administrative level. The inclusion of all municipalities makes external benchmarking possible. In this regime, the measurement of performance is ‘obligatory’. The collection and transmission of a predetermined set of indicators is mandatory for all municipalities in the province. The methods of design and implementation can be described as being ‘pragmatic’. Many of the indicators originate from data available in financial documents that were already collected by municipalities. The performance regime also takes into account the respective context of the municipalities. The categorization scheme classifies municipalities into 13 categories, using the number of residents and the presence/absence of water treatment facilities. The regime in Quebec also features a contextualization tool in the form of influential factors. These factors are predetermined influences offered by the Ministry and the Committee. Influential factors can be summoned by managers to explain the values of the indicators. These factors are used in the transmission of the indicators’ values to the Ministry.
The degree of diffusion is close to 100 percent. The collection and transmission of the indicators are mandatory, but no action is taken against municipalities who choose not to comply with the performance regime. With respect to collection and transmission, the implementation is uniform. Municipalities have until the end of September of the following year to transmit their data (MAMROT, 2010b). This is to say that for the 2009 values of the indicators, municipalities had until 30 September 2010 to share their data with the Ministry. In terms of use and reporting, the implementation varies.
As for steering, the performance regime is mostly ‘bottom-up’ in its approach, thanks to the collaborative structures of its governing body. The Municipal Management Indicators are ‘locally steered’ by the numerous municipal professional organizations that have participated in the decision-making process from the beginning. Also in terms of steering, Quebec's performance regime does not focus on control by the Ministry, but rather on learning. The inclusion of the measures in management activities, also known as performance management, and in official documents is voluntary for municipalities. The Ministry does not impose sanctions or rewards for good or bad values of the indicators.
As for transparency, the regime is not open to the public or to municipalities. Since its implementation, the values for the indicators are released in an anonymous aggregated form with quartiles for the 13 categories in a report available on a website. 2 Since the Fall of 2009, municipal managers can log onto a password-protected web portal to have the opportunity to compare their results with other municipalities. Once logged in, the information available to managers is also presented in an anonymous aggregated form. Regarding the validity of the data, by design (TIPM, 2002c: 8), there are no performance audits to determine whether the data transmitted by the municipalities to the Ministry are accurate.
Quebec's Municipal Management Indicators as an intelligence regime
Theoretical expectations of targets, rankings, and Intelligence Regimes
Source: Hood (2007: 96).
Hood (2007: 101) theorized that the limits of intelligence regimes are a lack of transparency and clear incentives. This is precisely what is observed in Quebec's regime. Hood's predictions of what would constitute an intelligence regime apply well to Quebec's Municipal Management Indicators.
Quebec's Municipal Management Indicators as a stable intelligence regime
The story of Quebec's performance regime presented in the next section is a counter-example to Pollitt and colleagues’ (2010: 19) ‘logic of escalation’ model. According to this model, we could expect from a performance regime: (a) a multiplication of indicators, (b) a drift from formative to summative indicators, (c) a connection to incentives and sanctions, (d) a complication of indicators, (e) a diffusion of regime ownership, and (f) an external reluctance to trust the performance regime (Pollitt et al., 2010: 19–25). What is expected from Pollitt and colleagues’ (2010: 20) logic of escalation model is that, ‘(t)here is a tendency for the uses of PIs to shift from the formative to the summative, that is, from information aimed at identifying possible areas for local managerial attention to indicators that are seen to define performance through expression as targets or “league tables”.’ To use Pollitt and colleagues’ (2010) models with Hood's (2007) terminology, this is to say that the ‘logic of escalation’ model posits that the intelligence regime will evolve into rankings and/or target regimes over time. The upcoming description of the changes in Quebec's municipal performance regime does not corroborate this model.
Changes in the design of Quebec's Municipal Management Indicators regime
The features of the Municipal Management Indicators that have changed since its implementation are the number of mandatory indicators, the appellation of the indicators, and the number and scope of the objectives. The archival data of the Committee's meetings reveal that the number of mandatory indicators has been the object of much debate. The final recommendation made by the Committee to the Ministry was to launch the program with ten indicators (TIPM, 2002a: 4, 2002b: 6). The mandatory indicators came from a pool of 99 indicators that were validated by municipal managers (Corporation des Officiers Municipaux Agréés du Québec, 2002: 73), during consultations and pilot projects. The Ministry finally settled on 19 indicators when the regime was implemented (Ministère des Affaires municipales, du Sport et du Loisir, 2004: 8). It is reported that during two days of consultation with municipal managers in November 2004 and January 2005, the consensus of the managers who were consulted was to increase the number of mandatory indicators (Centre de promotion de l’excellence en gestion municipal, 2005: 5–6). The projection was that 35 indicators would be added in 2008 to the 19 indicators that were mandatory in 2005 (CPEGM, 2005: 15). The suggestions collected during the consultations were not implemented. For the year 2007, the number of mandatory indicators was dropped from 19 to 14.
Quebec municipal benchmarking mandatory performance indicators in 2004 and 2010
Source: Adapted from Boyne (2002).
Data from Table 3 suggest that the modifications of the measures in 2007, which were still in effect in 2010, resulted in a scaling down of the initial version of the regime. Nine mandatory indicators were dropped (five of which were finance indicators), while four new ones were added to cover the human resource function. However, the dimensions covered by the indicators did not change noticeably. The modifications in the set of mandatory indicators have not translated into an expansion of indicators outside of dimensions of efficiency and service outcomes.
The logic of escalation model suggests that a few years into a performance regime, concerns for data validity lead to the complication of indicators (Pollitt et al., 2010: 23). Data validity has been a recurrent concern for the regime. As we have seen, many indicators have to do with efficiency and cost per unit. Scholars and practitioners, including a scholar from Quebec, warn that cost and financial indicators are perceived as being too lagging and backward looking, not predictive enough, foster short-term or incorrect behavior, and lack actionability for management actions (Henri, 2006: 82). Data validity threats from accounting differences were documented in early consultations with managers (Table sur les indicateurs de gestion municipaux, 2004b: 2, CPEGM, 2005: 5), by Committee members (Groupe de travail sur la valorisation et la promotion des indicateurs de gestion municipaux, 2008a: 4; Comité des partenaires des indicateurs de gestion municipaux, 2009: 2), and in a report documenting cost imputation disparities for certain indicators (Bellavance et al., 2008: 47).
The complications of indicators can take the form of standardization or characteristics/risk-adjustment of indicators. Statistically adjusting indicators for characteristics of municipalities was suggested by an academic member of the Committee in a demonstration (TIPM, 2003c: 6), and included in an official report to the Ministry (CPEGM, 2004a: 20). Statically adjusted indicators were not a feature of the regime in 2010.
The complication of indicators can also manifest itself in composite indices replacing simple indicators. Given the limited number of indicators, the aggregation of many indicators in indices has not been an issue.
Evolution of the name for the indicators
In changes, one of the facts that set the tone early on was how the set of indicators is called. From 1999 to 2003, for all activities, meetings, consultations, and pilot projects, the indicators were referred to as ‘performance indicators’. When the Ministry made public the language of its upcoming law enabling the performance regime, the ‘performance’ appellation had disappeared. The indicators were now ‘management indicators’. One of the professional associations active in the Committee protested in a public document about the upcoming law (COMAQ, 2003: 3). This association argued that the appellation ‘performance indicators’ had been used for four years in the Committee's internal and outreach activities. It is used in the public sector in Quebec and abroad (COMAQ, 2003: 4). The regime's indicators were to be ‘management indicators’, not ‘performance’ ones. The move away from performance itself and defining performance was shared by the Committee in the following months. The agreement among Committee members in 2004 was that ‘the results of the indicators do not count, but their interpretation does; especially their link to the municipality's objectives’ (TIGM, 2004b: 5).
Evolution of the performance regime's objectives
The objectives of the performance regime in Quebec changed in number and in scope before the implementation of 2004, at implementation, and in the seven years after it. At the policy formulation stage, during the second meeting of the Committee in November 1999, five ideas that were seen as important for the measurement of performance were put forward (TIPM, 1999b: 3). First, an informational tool should be developed. Second, this tool should be geared for decision-making. Third, services that could be improved would be singled out. Fourth, all resources for municipal organization would be evaluated. Fifth, best management practices would be identified. These five ideas essentially cover the first (projected) official objectives of the regime (COMAQ, 2000: 6–7). Table 4 summarizes the official objectives of the performance regime from 2000 to 2010. The 2004 objectives are highlighted, as they were the ones in effect for the official launch of the regime.
The evolution of official objectives for Quebec's Municipal Management Indicators regime, ranked
Source: COMAQ, 2000: 6–7; TIPM, 2001: 11–12; COMAQ, 2002: 62; MAMSL, 2004: 4; MAMR, 2008; MAMR, 2009, 2010.
Types of objectives for Quebec's Municipal Management Indicators regime at implementation (2004) and in 2010, unranked
At least two elements differentiate the 2010 objectives of the performance regime from those of 2004. The first difference is the number of objectives. The 2010 goals of the regime were more limited than the ones put forward at the formulation and implementation phases. Second, objectives that have been dropped since the launch of the program were primarily related to reporting and accountability and performance improvement. Objectives related to use were mostly left untouched.
Analysis: Are the objectives reached?
There has not been a formal evaluation of the performance regime ordered by either the Ministry or the Committee to assess whether the performance management reform implemented in 2004 delivered the expected results. In lieu of a professional evaluation, subjective reviews of members of the Committee were offered. Also, in 2009, at the request of the Committee, the Ministry supported an evaluative self-administered survey on the use of performance measurement. Even if the use of performance information was the prime focus of this survey, some questions related to accountability and reporting were included. No questions were formulated to evaluate the enhancement of performance.
Use
Archival data from the Committee shed some light on the use of indicators. In January 2008, members of the Committee were asked to express their association's take on the performance regime. One evaluation by a Committee member recorded in the meeting minutes was that if the indicators are to become a real management issue, they should be taken out of finance departments (GTVPIGM, 2008a: 3). His point was that they were not really used at this point in time. During the fourth Committee meeting of 2008, members expressed the view that the participation of the majority of managers is limited to collecting the data for the mandatory indicators (GTVPIGM, 2008b: 2). According to the Committee members, few municipal managers actually use the indicators by making them their own and analyzing them (GTVPIGM, 2008b: 2). Even for symbolic uses, ‘… the indicators are used very seldom as tools supporting decision making’ (GTVPIGM, 2008b: 3). The verdict that the indicators are not used, for the most part, was stated again during the first Committee meeting of 2009 by representatives of professional associations, and by a high-ranking government officials (CPIGM, 2009: 3).
The results of the 2009 survey complement the picture painted by the archival data. Municipal managers taking the survey were asked about the general utilization level of the indicators. This kind of question does not discriminate between symbolic and instrumental uses of performance information. Approximately 55 percent of respondents indicated that the indicators are never used in their municipality. Indicators were seldom used in 41 percent of municipalities. Around 5 percent of municipalities stated that this information is used often. One respondent stated that that the indicators were used very often. Around 45 percent of municipalities would be users of indicators.
To evaluate the extent to which municipal managers use indicators for instrumental purposes, respondents were asked to identify management and budgeting activities where indicators are included. The most widespread management activity involving indicators is an ‘evaluation to establish underlying reasons for results’. Approximately 13 percent of municipalities in the sample say they used the indicators for that purpose. Indicators would be used to ‘prepare budgets, including resource allocations or discussion of resources reallocations’ in 7 percent of municipalities. ‘Managing operations or routine decisions (e.g. scheduling activities)’ and ‘establishing contracts for services (e.g. snow removal)’ would be used as municipal management indicators for around 4 and 2 percent of municipalities in Quebec.
Relative frequency of comparison levels being used and targets set by municipalities
In addition, the survey data for Table 6 help us assess if the objective of ‘enabling information sharing between comparable municipalities’ was achieved. Even in the presence of external comparisons, few managers that have compared their results claim to have taken advantage of this shared information.
Reporting and accountability
The responsibility of municipalities to be transparent to their citizens was supposed to be achieved by letting them know that the municipality's results meet their objectives (CPEGM, 2005: 19; TIGM, 2005: 3). As early as 2000, it was acknowledged in meetings of the Committee that the majority of municipalities do not have a strategic plan or objectives (TIPM, 2000a: 5). Inspired by solutions adopted in Ontario, Australia, and in the United States (through GASB and ICMA), the regime established its own objectives (TIPM, 2000a: 5).
The archival data from the Committee meeting revealed that a debate went on about potential regime features for accountability to citizens, accountability to council members, and the mandatory transmission of data to the Ministry. The options considered by the Committee were to publicize the indicators with the financial statement of the Treasurer, with the Mayor's speech on the financial situation, or with the Mayor's comments published around 30 June of the following year (TIPM, 2002a: 3–4). The recommendation of the Committee to the Ministry was to make the deposit of the indicators at a council meeting mandatory, and to also make mandatory the inclusion of the indicators in the Mayor's speech on the financial situation (COMAQ, 2002: 69–70; TIPM, 2002b: 5–6). Contrary to the Committee's recommendation, the delay for data transmission was pushed back to 30 September of the following year (TIPM, 2002e: 3–4). Also, to give time to elected officials and managers to familiarize themselves with the indicators, the results for the year 2004 could be kept private (TIPM, 2002c: 7). It was posited that ‘reporting would be rather simple in the beginning, only to become more elaborate with time’ (CPEGM, 2005: 5).
In terms of accountability, what was deemed important for the Committee was that ‘citizens should have in their possession clear information with which to judge their council objectives’ attainment’ (TIGM, 2005: 3), and also that elected officials should be allowed flexibility in their exchanges with citizens (TIGM, 2005: 3). The obligation to include the indicators in the Mayor's speech on the financial situation was later withdrawn by the Ministry. A public document released by the Ministry explained that making the values of the indicators available during a meeting of the municipal council constitute reporting (Ministère des Affaires municipales et Régions, 2007: 3). ‘It remains the sole element regarding “citizen reporting”’ (MAMR, 2007: 3). The standard format for presenting the values of the indicators is created from a template generated by the transmission of data to the Ministry. It features the values for the indicators for the municipality for the last three years available. No comparative data from other municipalities are offered. Citizens seeking to make comparisons between their municipalities and others needed to peruse the internet for other municipalities divulging the values of the indicators or attend another municipality's council meeting.
Later, the assessment by the Committee was that, because of the low level of interests by citizens and elected officials in the values of the indicators (Groupe de travail sur les indicateurs de gestion municipaux, 2007: 6), most municipalities limit themselves to depositing a report about the indicators at a council meeting, and in rare cases make this report available on their website (GTVPIGM, 2008b: 3). An admission was made by the Committee to the effect that the design of the system had not enabled the attainment of the initial objectives regarding accountability to citizens (GTIGM, 2007: 6).
One item that is indicative of the trajectory of Quebec's performance regime has to do with indicators covering the dimension of responsiveness (see Table 3). Citizen satisfaction indicators, were debated favorably (COMAQ, 2002: 22, 24; TIPM, 2002d: 5, 2003c: 4; CPEGM, 2004b: 26) and against (COMAQ, 2000: 46; TIPM, 2003a: 4) regarding their mandatory inclusion in the launch of the regime. This would have flowed out of the ‘reinforcing the commitment of municipal administration toward the public’ and the ‘enabling elected officials and public servants to better understand the evolution of their organizations’ quality of services’ 2004 objectives. Citizen satisfaction indicators did not make it to the regime's launch. Instead, a citizen satisfaction field guide and a survey template were offered to municipalities by the Committee in 2004. The use of this tool was voluntary. Four years after offering this option to municipalities, the Committee's assessment on the use of that complementary tool was that it had not been used (GTVPIGM, 2008b: 4).
The 2009 survey data complement the picture painted by the archival data. There is a reporting obligation to deposit the values of the indicators in a council meeting. However, there is no enforcement of that obligation. We do not know the proportion of municipalities that comply with this obligation. When asked if the indicators were reported to elected officials, or were used to report to ‘citizens, citizen groups or to inform the media (e.g. the mayoral report on financial health)’, 58 and 28 percent, respectively, of managers who do use the indicators indicated that they fulfill these obligations. Participants in the survey were also asked if the mandatory management indicators have ever been included in their municipality's budget and in the annual report on the financial situation. The results fluctuated between 15 and 6 percent for the annual budget, and 18 and 8 percent for the annual report on the financial situation.
Performance improvement
One of the regime objectives that has endured to this day is the improvement of municipal services’ performance. Other objectives supporting performance improvement such as the presence of targets and identified best practices were also sought until 2008 and 2009, respectively. In order to determine whether the performance of municipalities improved because of the existence of the performance regime, definitions of performance are needed. How is performance defined by the Ministry and the Committee? It is not. Apart from a passing remark to the effect that performance usually translates as effectiveness and efficiency (MAMSL, 2004: 3), nowhere in the Ministry's brochures and guides is performance defined. The Ministry does not stipulate what constitutes good and bad performance. It does not even suggest what constitutes better or worse performance. The first element coming close to a definition of performance from the Committee is a bullet point in the meeting minutes at the beginning of the formulation phase of the regime: ‘• concern of the Committee (lower costs)’ (TIPM, 2000b: 7). The second element is this formula: ‘performance = amelioration’ (COMAQ, 2000: 12). There has been no mention in Committee meetings of the intention to assess whether the performance regime has resulted in an improvement of performance in municipal services.
The identification of best practices set inspirational examples to be emulated. Initially, it was planned that best practices would be offered by an Institute of Best Practices (TIPM, 2003b: 3). The identification of best practices was discussed by the Committee in 2003 (TIPM, 2003d: 4) and 2004 (TIGM, 2004a: 4). In 2008, one professional association complained that no concrete example of best practices had yet been offered (GTVPIGM, 2008a: 3). The identification of best practices was deleted from the list of official objectives in 2010 (MAMR, 2009, 2010a).
The 2009 evaluative survey informed us about the possibility that municipal services improved in Quebec as a result of the performance regime. Less than 8 percent of survey respondents stated that they have used the indicators for ‘specific performance improvement initiatives (e.g. investments, technical assistance, training, operations improvements)’. The 2009 survey also enquired about behaviors associated with performance improvement: external benchmarking and target setting. As we have seen in Table 6, municipalities pursuing external comparisons and setting targets are the exception. However, there is a mitigating factor in establishing the link between the implementation of a performance regime and an improvement in the performance of municipal services: the use of the performance regime by municipal managers. In the users’ guide to performance indicators released in 2008, it is stated that municipalities should include the indicators in their day-to-day management in order to be able to increase their performance (Gagnon, 2008). In the ‘use’ section above, we saw that the performance indicators are never used by most, and seldom used by the rest. Given the low use of the indicators and the absence of a definition of performance, we cannot verify that the official objective for performance improvement has been met by the regime.
Despite the presence of one performance improvement objective, the performance regime has not been designed to foster improvements in municipal services. No definitions have been offered for performance, or minimal performance, thresholds. Best practices were never featured for emulation. Municipalities did not have to establish targets for themselves. Unlike provincial organizations under the Loi sur l’administration publique, municipalities are not required to have a strategic plan. All in all, the formative approach of the regime in Quebec never came close to becoming summative, as the ‘logic of escalation’ model would predict.
Discussion
What can be expected from an Intelligence Performance regime?
There is much flexibility built into the design of Quebec's municipal regime. The fact that values of indicators are anonymized, coupled with the lack of sanctions by the Ministry, and the presence of influential factors in data transmission tell us that the basic principle behind the regime is to offer background information, not to foster transparency or competition.
The chronicle of the Municipal Management Indicators in 1999–2010 tells the story of a bottom-up regime driven by the Committee, with few demands on municipalities from the Ministry, aside from data collection and transmission. As a bottom-up regime, the Ministry has little leverage on municipalities. As an intelligence regime, the Municipal Management Indicators lack transparency and incentives. Hood's (2007: 101) prediction is summarized by Kasdin (2010: 61): ‘By and large, for performance measures to be effective they need to be more than hortatory: they need to be tied to some incentive.’ A lack of incentive would mean a lack of improvement (Kasdin, 2010: 73). In performance measurement, ‘voluntary’ means it will not be done (Mazouz et al., 2008: 187).
On a consensual basis, the municipal stakeholders faced a tall order in 1999: putting together a system where external benchmarking information would be available for more than a thousand municipalities. The type of regime that was selected was unlikely to embarrass mayors, council members, general managers, and treasurers. The performance regime designed in Quebec has no definitions of performance, and is not referred to as ‘performance’. It has no performance audits, 3 no citizen outreach, no consequences from the Ministry, and little comparison between municipalities.
Different management tools compete for managers’ attention. Some of them, such as the budget and financial statements, are well established, familiar to municipal managers, and focus on processes. Others, such as performance measurement, are unfamiliar to managers of small and very small municipalities, and focus on results. Financial statements are audited. Financial statements are acted upon by other levels of governments. Financial statements are available to citizens. The transmission and the publication of financial statements is timely. In Quebec, as a management tool, budgets and financial statements are used, whereas performance measures, as a management tool in its intelligence form, are not.
Why the ‘logic of escalation’ did not happen
It is to be expected that a complex decade-long government intervention such as a performance regime would go through changes. What is interesting in the present case is the nature of the changes. Faced with an incongruity between objectives and results, the official objectives have been scaled back to match the results. In 2004, a balance of objective types (use, accountability and reporting, and performance improvement) was present. The 2010 objectives are primarily linked to the use of performance information, more specifically symbolic use. In 2010, there was only one performance improvement objective for the regime. How come the inherent limitations in an intelligence regime did not bring about more complex elements leading to targets or ‘league tables’?
Three plausible explanations could be considered. The first is that the regime was not widely used. Not enough managers expressed their doubts about the limited number of indicators, the narrow dimensions they cover, the lack of guidance in analyzing the values of the indicators, the outdated nature of the indicators, the difficulties in making straightforward external comparisons, etc. A second plausible explanation is that Pollitt and his colleagues’ (2010) logic of escalation model that originated from case studies of hospitals in England and in the Netherlands does not apply perfectly to Quebec's case. Two-thirds of municipalities in the province of Quebec have less than 2000 residents. Managers accumulate many responsibilities in these rural towns, and run small organizations. They may devote less time to data analysis than managers in healthcare facilities. Thus, in relation to the first plausible explanation, they might not strive for a more complex management tool. Research in others settings will be needed to ascertain the applicability of the logic of escalation model. The third plausible explanation is that the Committee and the Ministry were pressured by municipal associations and stakeholders, among them elected officials, to keep to a minimum what might be perceived as provincial interference. The bottom-up design of the regime would amplify the pressure felt by the Committee. The data on which this study rests cannot attest to the existence of informal demands for the status quo.
Conclusion
In this article, we presented a dynamic description of a municipal performance regime in the Canadian province of Quebec. The main characteristic of that performance regime is that it was designed to offer background information to managers, but little else. At implementation, the objectives set for that intelligence regime included gains in transparency and improvement that could be expected from more stringent rankings and targets regimes. However, what was hypothesized by Hood's (2007) typology happened: no gain in accountability and performance improvement could be discerned. Additionally, a logic of escalation (Pollitt et al., 2010), where an internal momentum inherent to performance management pushes an intelligence regime toward becoming a rankings or a targets regime, did not occur.
Even if we keep in mind that administrative reforms do not often work as intended (Caiden et al., 1995: 96), and even if we use the four 2010 objectives pursued by Quebec's Municipal Management Indicators in our assessment, we would be hard pressed to proclaim that this regime achieved the goals it had set for itself. In a recent article, Moynihan stated that most performance reforms fail, ‘… if by failure we mean that they do not achieve the wildly unrealistic targets of reformers – often deliberately overstated in order to create momentum for change – in a very short period of time’ (Moynihan, 2009: 8). He added at the same time that the accumulation of reforms has and will change organizational cultures (Moynihan, 2009: 8). How long can it take to work out whether a performance strategy has found widespread adoption? Berman suggested a generation, about 20 years (Berman, 2002: 349). Half-way through the 20-year mark, it is hard to see what kind of tangible benefits can come out of the intelligence variant of the municipal performance regime in Quebec. Nonetheless, Hood (2007: 101) tells us that intelligence regimes have the intended effect of ‘adding knowledge for uses that may not be fully foreseen’.
Footnotes
Acknowledgment
I would like to thank Julie Faucher, COMAQ, for making the Committee meeting minutes available. I would also like to thank the Ministère des Affaires municipales, des Régions et de l’Occupation du territoire (MAMROT) of Quebec for their support with the 2009 managers’ survey. I am also grateful to François Bellavance and Michel Guindon, professors at HEC Montréal, for their guidance and financial support. I am indebted to Alexander Henderson, Gautam Nayer, and Alicia Schatteman for their comments on an earlier version of the article. The views expressed in this article are my own: they do not represent the position of the MAMROT. I also thank the reviewers for their insightful comments.