Abstract
In Canada, in-depth evaluations of federal programs are intended to occur every 5 years. As such, evaluation is a periodic retrospective (lag) indicator examining results achieved versus program objectives. In a Canadian context, stand-alone evaluations have proved challenging to implement, time consuming, and not well adapted to annual management accountability needs. Consequently, there are important benefits from developing parallel ongoing operational performance measurements, complementing periodic evaluations as an integrated system. With links to program evaluations, ongoing performance feedback can include predictive (lead) indicators of progress, through operational linkages to a program’s intended long-term outcomes. The present case study examines program efficiency concerns demonstrating lead indicators as an “early warning system”—targeting problem areas, producing speedier program adjustments (including accountability and efficiency improvements) and also demonstrating potential to increase quality, timeliness, and usefulness of longer term in-depth evaluations.
Introduction
Since public sector performance cannot be assessed against the private sector benchmark of profitability, it falls mainly to governments to conduct periodic in-depth program evaluations to assess the extent to which long-term success has been achieved from their program activities. “Success” in this context is related to effectiveness in achieving the stated objectives of individual, publicly funded programs and services. There is the parallel requirement that the results achieved must be efficient—so as to avoid excessive budgetary costs in the public domain.
The need for such information places enormous stress upon government evaluation activities. In theory, the evaluation function should provide timely, important feedback information produced by individual departments and agencies with respect to their expenditures of public funds. In this context, evaluation forms part of a system of governance based on transparent, results-driven accountability. Further, evaluation findings are intended to provide guidance in improving the effectiveness of existing programs. Evaluations should also have the potential to make an important input to decisions regarding the allocation of scarce resources and in addition should play a role in providing “lessons learned” for developing future program initiatives to respond to government policy developments.
However, as has been clearly underlined (e.g., U.S. Government Accountability Office, 2011; Auditor General of Canada, 2009), the program evaluation function has frequently fallen short of what is required. For example, in a recent publication, the Government Accountability Office notes “Most federal agencies now use performance measures to track progress toward goals, but few seem to regularly conduct in-depth program evaluations to assess their programs’ impact or learn how to improve results” (U.S. Government Accountability Office., 2011, p. 1, 2).
In a Canadian context, periodic evaluations have been emphasized as part of results management accountability policy frameworks. However, over the years, there have nevertheless been chronic problems with respect to lack of availability, coverage, timeliness, quality, and reliability of program results measurement undertaken by individual government organizations.
It is argued in the present article, and illustrated in its case study, that an effective way to develop timely and reliable program results feedback (with the aim of bringing about improvements in value for money from expenditures of public funds) is to combine in-depth periodic program evaluations along with ongoing operational performance measurements that are predictive. The form of analysis is a particular application of predictive analytics undertaken in a government context. Based on experience to date, it seems unlikely that either evaluation or ongoing performance measurement, operating by itself on a stand-alone basis, would be sufficient for the task. Rather it is the combination of the two, operating as a strategically integrated system, that is required.
Evaluation as a Lag Indicator
The program evaluation function across governments is, by design, both periodic and retrospective in nature (i.e., it is a lag indicator). Due to its in-depth focus and intended emphasis on results-driven accountability (requiring highly reliable results measurements), it is both time consuming and costly to implement. For example, in the case of many larger programs, evaluations require a period of up to 18 months to 2 years to complete and involve considerable budgetary resource outlays. In Canada, the federal government’s evaluation policy intends evaluation to operate on a 5-year cycle, involving an arm’s length program results-measurement function located within each individual department or agency. While ongoing operational performance measurement in general has been acknowledged as having a complementary role, it has not been closely integrated with the evaluation function as part of a comprehensive results-measurement process that would include ongoing predictive operational performance indicators. As a result, over the years the Government of Canada’s evaluation function has been encountering limitations that were, to some considerable extent, a consequence of its own design.
Thus, if there were any significant ongoing issues with a program’s effectiveness in achieving its long-term objectives, or efficiency in program delivery, there would be a considerable (and costly) time lag before the situation could be identified and confirmed by means of evaluation and subsequently addressed. Further, in a Canadian context, while periodic evaluations can play a role in the government’s overall accountability to the federal legislature, within individual departments and agencies prevailing mobility levels among members of the executive group (senior management) result in an important dilution of any internal results-driven accountability regime (even with a 5-year evaluation cycle—when it can be achieved).
Challenges of Implementation
Because of logistical complexity of evaluation policy requirements, heavy costs, and difficulties in accessing fully qualified evaluation staff, many government organizations have encountered significant problems over the years in implementing the evaluation function. For example, the Auditor General of Canada notes:
Program evaluation has been practised in the federal government, in one form or another, for close to forty years. The Office of the Auditor General examined program evaluation in 1978, 1983, 1986, 1993, 1996, and 2000 (and now, more recently, in 2009). The history of federal program evaluation from 1970 to 2000 (and subsequently) reveals repeated initiatives at the centre of government to establish and support the function, and often critical observations by the Office of the Auditor General on the success of these efforts. (Auditor General of Canada, 2009, p. 5, 13)
In a Canadian context, even if the evaluation function ultimately proves successful in producing adequate in-depth assessments of government programs every 5 years—without accompanying, ongoing predictive operational performance indicators, this would still provide only a periodic (intermittent) retrospective picture involving extensive lags. In other words, where there are problems with a given program’s effectiveness or efficiency, periodic in-depth 5-year evaluation feedback produced on a largely stand-alone basis is, at best, the equivalent of “closing the stable door long after the horse has run-off” and leaving in its wake a trail of suboptimal operational activities and program design deficiencies that are slow to adjust.
In the context of the extremely serious budgetary challenges at present widely experienced by governments, such a lagged adjustment process is not likely to produce maximum value for money with respect to program expenditures. To the extent possible, managers of individual programs need to be able to demonstrate on an up-to-date basis that maximum levels of program effectiveness and efficiency are being achieved from current expenditures of scarce program resources.
The Case for Ongoing Operational Performance Measurement as a Lead Indicator of Long-Term Program Success
There already exists quite an extensive technical literature (particularly in the United States) with respect to operational performance measurement and its applicability to results-management organizational systems. However, one striking characteristic is that coverage, in general, would benefit from more detailed, concrete examples illustrating such performance indicators as a key element of a comprehensive results-measurement system (i.e., a system that combines (a) short-term operational performance [lead] indicators; with (b) longer term retrospective [lag] indicators from in-depth evaluations). Consequently, there is a need for clear, concrete empirical evidence as to how ongoing performance measurement and periodic program evaluation “fit” together.
An important point of departure on the potential for linkages (Heinrich, 2002) concludes that in the case of U.S. federal job-training programs, results of empirical analysis confirm that the use of stand-alone administrative data on performance management is unlikely, in itself, to produce direct, accurate measures of fully comprehensive program impacts. However, the analysis undertaken also suggests the data can nevertheless generate useful information for public managers about policy levers that can still be meaningful to improve the organizational performance—through linkage with other relevant information sources generated by research and internal reviews, and so on. This can be regarded as a precursor to the notion that ongoing operational performance measurements may be structured as lead indicators of operational progress toward program improvement—to complement retrospective program evaluation (rather than replacing it). Such an approach is particularly appropriate in view of recent progress in predictive analytics as a result of improvements in design and coverage of administrative data systems and improved access to such systems (Davenport and Jarvenpaa, 2008).
In a similar vein, a recent study (Dubnick & Frederickson, 2011, p. 31, Table 2) considers performance measurement in terms of the “measurable surrogates of results processes” (rather than more theoretical “results outcomes”). Again, the next step would be to link such “surrogates” to complement in-depth evaluations, as predictive operational performance indicators.
In the context of the present issues being considered, perhaps the most positive overview/reference in the technical literature (Wholey, Hatry, & Newcomer, 2010) concludes that the focus aiming at measuring program outcomes is becoming more widely established and the prospects for increasing utilization and further refinement of ongoing performance measurement are good. However, it is cautioned that there remain problems that ongoing monitoring systems are still largely output (rather than outcome) oriented. There is a particular need to have linkages to evaluation to clarify “black holes” in program logic and, as well, any additional contextual factors that affect program performance. In this respect, the need for a close linkage between evaluation and ongoing predictive performance measurement, and the benefits from such linkage, are clear.
Once developed, individual predictive performance indicators, again because of their ongoing nature, can also serve as an early warning system for program management and enable a more rapid response to problem areas as they emerge—without the need to wait for an evaluation to be undertaken at some future point (U.S. Government Accountability Office, 2011).
Based on all of the above, it is clear that there are major benefits to be derived from considering ongoing performance information as an important complement to the evaluation function and vice versa. Performance measurement can be strategically designed to include a series of benchmark indicators along with short-term feedback information in areas linked to future, long-term program success. Such predictors can legitimately be regarded as “lead indicators”—if the problems they confirm/identify are left untreated in the short term, there will be a cumulative effect by the time the program is subject to longer term periodic in-depth evaluation.
Under such a results-measurement framework, neither evaluation nor operational performance measurement should be considered independently. Rather, they would function as an integrated whole—with several important benefits:
an early warning system, with annual program updates from lead indicators enabling more rapid program adjustments to problem areas, would improve responsiveness—with accompanying increases in program effectiveness, efficiency, value for money, and resource savings; management accountability systems would become much more robust and effective, based on a more timely feedback and assessment of performance in achieving a program’s objectives. This would greatly strengthen the importance of results-driven accountability within government organizations, bringing still further improvements to management responsiveness; measurement methodologies would be strengthened since results measurement would be subject to ongoing challenge and review, as part of the strengthened accountability process; flowing from the above, not just quality, but also availability, of ongoing predictive performance trend data (as an input to future long-term periodic evaluations) would greatly improve—reducing the onerous time requirements for evaluation processes and enabling the evaluation function to specialize and concentrate more fully on those areas where in-depth analysis is required as to how a program is actually working; and in turn, findings from periodic evaluations can provide in-depth feedback to confirm and, where necessary make further adjustments to improve the usefulness of ongoing operational performance lead indicators. In this context, such key evaluation findings would have a more extended application over future time periods than when simply used on a single, stand-alone basis (Program officials have, on occasion, been dismissive of detailed findings from stand-alone periodic retrospective evaluations even just a few months after they have been developed—on the grounds that coverage is “dated” [e.g., extending back by as much as 2 years, to when the evaluation process commenced], and unlikely to be updated again for several future years until the next periodic evaluation).
A full illustration of how the integrated system would function is outlined in Figure 1. As the figure indicates, the key integration mechanism flows in both directions between (a) ongoing measurements based on predictive operational indicators, on the one hand and (b) periodic retrospective in-depth evaluation findings, on the other. Both the elements strengthen and reinforce each other.
Results-driven accountability: proposed integrated system of effectiveness measurement.
In the case of the former, ongoing operational feedback can provide important up-to-date trend data input on what the lead indicators are showing, focusing on progress in achieving success in key areas related (both directly and indirectly) to program the objectives. Further, these lead indicators also can show how well the organization’s management accountability system is responding by way of operational and/or program design adjustments to issues identified by the early warning system, as part of management’s responsibilities. This can greatly assist the evaluation function in terms of improved data availability, greatly improved data quality, and a much more rapid and strategic periodic program results-measurement function.
Conversely, information feedback flowing from periodic retrospective evaluation findings forms an important input to better inform the early warning system. Because of the in-depth nature of its retrospective view, evaluations provide key contextual analysis within which to better situate and interpret the predictive indicators that are central to the early warning system’s effectiveness. In addition, as noted above, the evaluation function can also provide an important “second opinion” on the need to fine-tune/refine existing lead indicators to maintain the early warning system’s integrity. This would be a new role for the evaluation function to undertake and is central to an organization’s results-management accountability regime.
Case Study Application of Ongoing Performance Measurements as Lead Indicators: The Public Service Commission of Canada
To test the possibility of developing lead indicators using ongoing operational performance data in an evaluation-linked context, the present case study utilized annual operational performance statistics developed by the Public Service Commission of Canada and specifically designed to function as lead indicators of longer term program success.
The Public Service Commission of Canada has oversight responsibilities for the staffing activities of some 80 Canadian federal departments and agencies. These staffing activities accounted for approximately 125,000 staffing actions in 2008–2009, involving permanent, temporary, and student workers—with the principal focus on permanent jobs. The staffing actions include new hires as well as movements of existing workers within the federal government. (Public Service Commission of Canada, Annual Report 2008–2009)
In 2003, legislation was passed, coming into full force in December 2005, that resulted in a major reform of the federal government’s staffing system. The legislation in its preamble sets out the expectations as to the system’s operation, based on the following features (a) a highly delegated arrangement providing individual departments and agencies with authority to staff; (b) to be operated fairly; (c) transparently; (d) with equality of access; and (e) with appropriate representation of employment equity groups—all to be independently safeguarded by the Public Service Commission of Canada.
While the Public Service Commission has formally delegated its authorities to individual departments and agencies for purposes of operational implementation, it is still required to answer annually to the federal legislature on the performance of the system. Taken by itself, a periodic evaluation approach aiming at a 5-year cycle would not meet this requirement; instead reliable annual data were required that would provide early indications of any problems in the operation of this highly delegated system.
As part of the legislative reform, extensive consultations were undertaken with respect to implementing the required changes, not only in terms of operations (outputs) but also in terms of the results to be achieved (outcomes)—including efficiency concerns. Average staffing times (measured in calendar days between date of advertisement and first notification of appointment) were regarded as a particularly important area of such efficiency concerns. For example, in 2007–2008, single-vacancy internal staffing actions for permanent positions required, on average, a period of 132 days to successfully complete the staffing processes involved.
Flowing from the feedback provided by consultations undertaken with key stakeholders, the performance management regime was developed based on the rationale that progress with respect to the following operational performance measures would, in turn, result in desired changes in efficiency in organizations’ staffing processes:
successful delegation of staffing responsibilities to deputy heads of departments and agencies; rigorous human resources (HR) planning for key staffing strategies that are integrated with the organization’s business planning; adequate organizational HR support systems capacity (HR specialists); and effective accountability at the level of individual organizations for the HR function in general and for staffing results produced.
From a management and operational perspective, establishing these linkages means that changes in desired longer term strategic outcomes (in this case efficiency improvements) can be achieved by changes in short-term operational performance. In individual cases, problems in any of the four indicator areas identified were thought likely to detract from the efficiency of the organization’s staffing processes. Analysis of the operational performance assessment was evidence-based in each case and was undertaken by Public Service Commission of Canada staff. See Figure 2 for examples of Public Service Commission assessment criteria.
Examples of criteria to assess key lead indicators (2008).
The predictive nature of evidence-based assessments of each organization’s performance was then tested by means of regression analysis. The regression is focused on the statistical association between (a) variations in average time to staff across individual departments and agencies (involving vacancies for permanent positions filled by workers from within government—the largest vacancy group) and (b) variations in assessments regarding the four performance indicator areas in the case of each organization. For purposes of this case study, the analysis also explored the influence of contextual factors that would normally be developed by a periodic evaluation and thought likely to have an impact on time to staff results. Overall, statistically significant associations linked to increased time to staff impacts constituted an early warning feedback on potential problem areas. Further, due to its ongoing nature, once identified, changes could be tracked and updated annually (in contrast with the much slower 5-year cycle in the case of stand-alone in-depth retrospective program evaluations).
Two models were examined: Model A aimed at examining the role of each organization’s HR planning function, separately from Model B which concentrated on a more comprehensive approach based on the performance and rigor of each organization’s results-management accountability system.
Does the Early Warning System Work?
The regression analysis summarized below points strongly to the conclusion that important variations in the efficiency of staffing processes across departments and agencies are linked to performance differences in certain key operational indicators for each organization. Equally important, other performance indicators were excluded by the regression. Also, the role of contextual variables examined was clarified. Further, this analysis was developed after only 1 year of operation under the new regime. This enabled operational adjustments to be put in place more rapidly and monitored.
Details on the regression specifications and findings are outlined in Table 1 which focuses on single vacancy staffing processes for permanent positions filled from within the government. Single vacancy staffing was selected to facilitate standardized comparisons between individual organizations.
Summary OLS Regression Findings.
Source: Public Service Commission of Canada—Staffing Management and Accountability Framework Data (2007–2008); Regressions—Informetrica Ltd., Ottawa. *p < .10; **p < .05.
Performance in establishing and operationalizing the delegation arrangements across individual departments and agencies was excluded from the regression analysis. There were no widespread divergences with respect to departmental performance in this area. (Basically, the implementation of the delegation processes was assessed in positive terms for the government overall.). The feedback from the early warning indicator system does not point to delegation arrangements as a focus of general concern. Elsewhere, the regression findings are as follows:
Taken by itself, the HR planning function is likely to have only a limited effect in reducing time to staff. While a strong HR planning performance is doubtless a necessary condition for achieving optimal efficiency performance across federal departments and agencies, the regressions indicate that it is not in itself a sufficient condition. (The HR planning variable was tested on a stand-alone basis [Table 1; Regression: Model A] and was found not to be statistically significant.) However, where there is a robust results management accountability system (which includes an HR planning component), there is considerable potential for improvement in staffing efficiency within an organization. When planning forms part of a well-functioning results-based accountability system, considerable potential for improvement exists to reduce the time to staff. Organizations with performance problems in their HR management accountability systems encounter, on average, 23 additional days to complete their staffing processes, compared to those with a stronger performance in this area (Table 1; Regression: Model B). Given an average time to staff of 132 days for single staffing processes for the government as a whole, this represents a significant efficiency shortfall. In addition, inefficient organizations can create wider, compounding efficiency problems (efficiency drag) when using internal recruitment. With in-house recruitment accounting for in excess of 80 percent of organizations’ overall staffing actions for permanent positions filled from within the federal government, initial vacancies frequently trigger further internal replacement staffing within the same organization. Consequently, while there are doubtless benefits accompanying internal recruitment (e.g., related to employee career development; building on existing organizational knowledge, etc.), inefficient internal recruitment processes can nevertheless create compounding negative effects. Such effects can extend the impact of slower time to staff up to several times beyond the initial staffing activity itself (efficiency drag). This is an area that merits future investigation, since extensive compounding effects might well, in aggregate, reduce the overall flexibility of an organization to adjust effectively, when required to adapt to changing circumstances (see also: excessive reliance on mitigating strategies, below). In cost-effectiveness terms, the evidence points to only limited benefits from increased investments in additional “frontline” HR staffing capacity. The regression analysis points to an assessment that increased HR capacity (staffing specialists) can reduce time to staff if such capacity is focused directly on handling staffing activities (see Table 1, including Note on detailed data and adjustments). However, it should be noted that even if it is targeted on frontline HR staffing specialists, the regression coefficient (Regression: ModelB—“Frontline HR Specialists” Variable) suggests that in present conditions an average increase of, say, 10% in frontline specialists would be associated with a reduction in average time to staff of only 4 days (3%) for the government as a whole (Table 1). In terms of comparative returns, it makes more sense to focus on working more efficiently within the existing resources (i.e., improving HR accountability and planning), rather than incurring the incremental costs of increasing budgetary resources (hiring more HR staffing specialists). Mitigating strategies can add to overall costs to fill permanent positions as well as to other downside risks and should not be used as a substitute for rigorous HR planning and accountability systems. The regression analysis indicates that departments and agencies that have a heavier use of temporary (Term) hires tend to be organizations that have slower staffing processes for permanent positions (increased time to staff); Table 1; Model B: Temporary Worker Variable). This suggests that in some instances, temporary workers may be used as a mitigating strategy (i.e., as a temporary stopgap measure pending more permanent solutions). In this context, as a supplement to standard recruitment processes, such strategies would add to longer term overall recruitment costs in meeting the organization’s permanent HR requirements. Overall, Term hirings amounted to some 11,000 hires annually in 2008–2009.
Elsewhere, the role of a further temporary mitigating response option—the use of (internally filled) acting appointments—was also examined in a separate preliminary analysis and similarly points to potential unintended downside effects. As in the case of term hirings, temporary in-house acting appointments across departments and agencies represent an important supplementary adjustment mechanism for government managers. Acting appointments number in excess of 16,000 annually and function as a flexible supplement to the approximately 66,000 regular staffing actions for permanent positions (2008–2009).
Preliminary regression analysis (Informetrica Limited, Ottawa, 2008) indicates that departments and agencies experiencing problems with HR business processes and systems for staffing tend to have excessive reliance on the use of acting appointments. Controlling for the effects from various key contextual factors (differences in organizational size, workloads, HR specialist turnover, organization’s promotion activity, etc.), the indications are that organizations with problems in HR business processes utilize, on average, approximately 150 additional acting appointments annually compared to those with stronger track records in this area. With an annual average activity level (2007–2008) of roughly 320 acting appointments per organization for the departments and agencies examined, this reliance on acting appointments represents a potentially significant issue.
While utilization of acting appointments may generate possible benefits in terms of increased short-term flexibility outcomes and opportunities for staff development, they are also accompanied by an increased risk of suboptimal matches between skills and job requirements (with potential productivity implications). Excessive dependence on acting appointments may also raise morale concerns related to the possibility of perceptions of diluting the merit principle, a cornerstone staffing value to promote confidence in the integrity of Canada’s federal public service.
Thus, in both the mitigation cases examined, the early warning system indicates that there are grounds for concern regarding the utilization of such practices and that excessive reliance on mitigating strategies appears to carry considerable downside risks. In both cases examined, the problem appears to lie in deficiencies in the efficiency of organizations’ HR processes, particularly HR planning and accountability, and it is in these areas that improvements should be sought.
Changes Resulting From the Performance Measurement Findings
Following the development of the regression findings, additional renewed efforts were put in place under the leadership of the Public Service Commission of Canada to establish more rigorous results-management accountability systems across individual departments and agencies combined with a strengthened HR planning regime—with the aim of reducing time to staff and monitoring progress toward this goal. This proceeded as an evidence-based initiative and represented a major undertaking.
In the context of the above, the head of the federal Public Service directed further in-house review work to be undertaken by the individual organizations involved and held deputy heads of departments and agencies directly accountable for the quality of their organizations’ planning and staffing strategies, along with strengthened organizational accountability requirements and monitoring by the Public Service Commission. Flowing from this, subsequent reviews have shown continued improvement.
Further, the reporting process in the context of accountability to the federal legislature highlighted the point (based on the quantitative analysis developed as part of this case study) that operational management changes emphasizing accountability within the existing system can be used to produce required improvements—without further statutory or regulatory change (and accompanying report burden).
To date, improvements in management accountability and planning, combined with the clear linkages demonstrating the potential impact on the improved efficiency of government staffing processes, have been accompanied by positive results in reducing average time to staff. While progress has been uneven, this should not be unexpected during a transition phase aimed at bringing about significant changes in government-wide practices affecting the full range of federal departments and agencies.
Based on the most current (2010) data available, it would appear that departments and agencies have focused their recent efforts on improving multiple staffing processes, with estimated efficiency gains resulting in a decline in average time to staff of nearly 20% between 2008 and 2010. In contrast, overall time to staff performance for single staffing processes has not immediately undergone a similar striking change, continuing with only minor fluctuations from past trends. Based on the wider implications identified by this case study (where internally filled vacancies can trigger further staffing actions), the overall cumulative reductions in time to staff over the period noted above do represent important progress. Nevertheless, while the developments to date are encouraging, there clearly is still significant scope for further improvement.
Conclusions and Assessment
With respect to the role of Public Service Commission of Canada, the case study demonstrates that the use of ongoing short-term operational performance measurement can provide timely, predictive results that meet accountability needs and provide direction to make necessary changes to management practices. In short, the evidence indicates that the early warning system approach works. The success of the approach depended on defining the underlying framework of expectations for success. This framework was widely discussed and accepted beforehand. It was further formalized (and strengthened operationally) as part of individual delegation agreements with departments and agencies signed by deputy heads and the Public Service Commission.
The underlying ongoing performance measurement model clearly emphasizes the importance of sound management practices as a key leading indicator of success. The empirical test of the model confirms its ability to serve as an early warning system that provides for more rapid adjustment than a stand-alone in-depth program evaluation system operating on a 5-year cycle. The early warning system clearly indicates where management should put its effort, and subsequent results showed a striking, measurable improvement.
As demonstrated by the Public Service Commission’s experience, the analysis of ongoing performance measures and the identification of predictive, lead indicators provide the basis to bring about changes in the approach to measuring results achieved. Periodic evaluation also remains an important element within the integrated system, with respect to providing strategic in-depth analysis for those areas that are beyond the scope and capacity of ongoing performance measurement. (For example (a) measuring and assessing the legislation’s long-term impacts on an in-depth basis, and identifying any additional contextual factors that might influence ongoing performance measurement and its feedback; (b) assessing the continued relevance of program rationale and the appropriateness of the lead indicators selected; and (c) examining possible efficiency/effectiveness trade-offs involved—all with a view to making necessary adjustments over time, based on the additional experience acquired.)
Ongoing performance measurement analysis provided an important contribution to the Public Service Commission of Canada’s 5-Year Statutory Review of the legislation. The experience at the Public Service Commission points to success in using predictive ongoing performance measurement indicators as a tool to monitor program performance that results in timely management change and intervention.
The result is a more iterative approach which, through its early warning system, provides sound ongoing information that is timely and serves important accountability and management purposes, backed up by a more strategic in-depth periodic program evaluation function—to provide the basis to adjust the early warning system, as required.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.